Abstract

Modern reverse engineering is moving toward automation, custom tooling, and agent-assisted workflows. These workflows speed up formerly slow and manual analysis tasks, but they quickly run into limits when binaries actively resist analysis through control-flow obfuscation, virtualization, mixed Boolean-Arithmetic (MBA), and other transformations. This training teaches the deobfuscation techniques, validation strategies, and automation patterns needed to make reverse engineering workflows effective on real-world protected targets.

Participants first learn how modern obfuscation techniques complicate reverse engineering, and then gradually build the deobfuscation techniques required to attack them in hands-on sessions. Along the way, they deepen their understanding of program analysis and learn when and how to apply different techniques in practice.

We begin with core obfuscation patterns and practical ways to attack them. Participants learn how to recognize common protection mechanisms, understand how they interfere with analysis and decompilation, and gain traction through careful manual analysis and targeted cleanup.

Early in the course, we introduce automation and agent-assisted reverse engineering as a practical layer on top of manual analysis. Participants learn how to set up analysis environments that make automated workflows easier to run, inspect, and repeat. They then learn how to divide analysis goals into smaller tasks, connect the required tools and scripts to the workflow, preserve useful intermediate artifacts, and validate each step. The goal is to turn manual expertise into repeatable, validated workflows that can be reused on related targets.

From there, the course builds toward more powerful reasoning with SMT-based analysis. Participants learn how to prove properties of code, reason about complex computations, and verify whether MBA-heavy expressions are equivalent to simpler rewrites. SMT then becomes a recurring building block for the automated techniques that follow.

To scale the analysis, the training introduces intermediate representations and compiler-style simplification passes that help normalize protected code and expose higher-level control-flow structures. On top of this, participants use symbolic execution to automate larger parts of deobfuscation, combine symbolic reasoning with SMT solving to attack opaque predicates, and explore feasible execution paths through protected code.

Building on these foundations, the course then returns to virtualization-based obfuscation. Participants learn how to use symbolic execution to identify virtual machine components, reason about instruction handlers, validate recovered semantics against traces, and write custom disassemblers and analysis helpers to reconstruct original program behavior.

The training also covers MBA simplification and program synthesis. Participants simplify code based on observed behavior, use synthesis to recover compact expressions, apply algebraic simplification techniques, and validate the correctness of simplified computations. These techniques are used to attack MBA-heavy expressions and to recover instruction-handler semantics of virtual machines.

By the end of the training, participants will understand how these techniques fit together in practical deobfuscation pipelines. The final part of the course focuses on scaling these workflows across larger protected programs and related obfuscation components. We also examine why automated and agent-assisted analysis fails on hardened targets, including patterns that deliberately mislead tools and agents. Participants learn how to recover from these failures by improving the analysis setup, strengthening validation, and reintroducing human guidance where automation reaches its limits.

Teaching

The training has a strong focus on hands-on exercises. Short lecture segments provide the background needed to understand how a method works, when it is useful, and where its limits are. The exercises then show how to apply these methods to real deobfuscation problems by building small, purpose-built tools and combining individual techniques into practical workflows. The trainer actively supports participants during the exercises, and after each task, we discuss different solutions in class. Participants also receive detailed reference solutions that they can use during and after the course.

In selected optional exercises, participants with access to LLM subscriptions or API keys can also use agent-assisted workflows to analyze protections and to write and validate analysis tools. The focus, however, always remains on the underlying deobfuscation techniques.

While the hands-on sessions primarily focus on x86 assembly, the underlying tools and techniques also transfer to other architectures such as MIPS, PPC, and ARM.

Class Outline

The training roughly follows the outline below:

• Introduction to Code (De)obfuscation

  • Motivation
  • Why protected binaries break standard reverse-engineering workflows
  • Core program-analysis techniques used throughout the course
  • Obfuscation foundations and manual analysis
  • Opaque predicates
  • Control-flow flattening
  • Virtual machines (VMs)
  • VM hardening techniques
  • Mixed Boolean-Arithmetic (MBA)
  • Recognizing protection patterns and developing deobfuscation strategies

• Foundations of Automated and Agent-Assisted Reverse Engineering

  • From manual analysis to repeatable workflows
  • Agents as tool orchestrators
  • Analyst guidance and task decomposition
  • Sandboxed analysis environments
  • Tool access for disassemblers, decompilers, emulators, and scripts
  • Reusable playbooks and validation checkpoints
  • Intermediate artifacts, logs, and context management

• SMT-Based Program Analysis

  • SAT and SMT solvers
  • Encoding program-analysis problems for SMT solvers
  • Proving semantic equivalence
  • Proving properties of code
  • Solving complex program constraints
  • Reasoning about MBA-heavy expressions
  • Validating rewrites and simplifications

• Intermediate Representations and Compiler-Style Simplification

  • Intermediate representations for reverse engineering
  • Static single assignment (SSA)
  • Dead code elimination
  • Constant propagation / folding
  • Normalizing and simplifying obfuscated code
  • Recovering higher-level control-flow structure
  • Agent-assisted development of simplification and analysis passes

• Symbolic Execution Foundations

  • Symbolic and semantic simplification of obfuscated code
  • Scaling symbolic execution workflows
  • Interaction with SMT solvers
  • Breaking arithmetic opaque predicates
  • Exploration of feasible program paths
  • Symbolic execution in iterative deobfuscation workflows

• Symbolic Execution for Attacking Virtualization-Based Obfuscation

  • Automated analysis of virtual machine components
  • Reasoning about instruction handlers
  • Writing disassemblers and analysis helpers based on symbolic execution
  • Trace-based validation of VM execution and handler semantics
  • Agent-assisted refinement of VM analysis tooling
  • Reconstructing original program semantics

• MBA Simplification and Program Synthesis

  • Concept of program synthesis
  • Learning semantics from input/output behavior
  • Obtaining input/output pairs from code
  • Search-based and stochastic synthesis
  • Local synthesis and inference rules
  • Simplification oracles and expression databases
  • Simplifying large expression trees
  • Algebraic approaches to MBA simplification
  • Combining synthesis, algebraic simplification, and SMT validation
  • Simplifying MBA-heavy computations
  • Synthesizing VM instruction-handler semantics

• Scaling Automated and Agent-Assisted Deobfuscation Workflows

  • Identifying and prioritizing obfuscated code regions
  • Deobfuscation pipeline design
  • Combining solvers, symbolic execution, synthesis, and custom tooling
  • Turning manual procedures into reusable automation
  • Scaling across functions, obfuscation components, and related targets
  • Trace-based validation and iterative refinement

• Automation-Resistant and Anti-Agentic Patterns

  • Recognizing automation-resistant and anti-agentic patterns
  • Search-space and validation-cost amplification
  • Tool friction and misleading artifacts
  • Recovery strategies

Requirements and Recommendations

Intended Audience

This training is intended for reverse engineers, malware analysts, vulnerability researchers, offensive security engineers, security consultants, and tooling engineers who want to analyze protected or obfuscated binaries more effectively and build practical deobfuscation workflows. It is also suitable for participants who want to understand how advanced program-analysis techniques, automation, and agent-assisted workflows can be applied to reverse engineering.

The course is not a general AI, LLM, or prompt-engineering class. Its focus remains on software deobfuscation, program analysis, custom tooling, and practical workflows for protected binaries.

Prerequisites

Participants should have basic reverse-engineering skills and be familiar with x86 assembly and Python. No prior experience with AI agents or LLM tooling for reverse engineering is required.

Software Requirements

Participants should have a disassembler of their choice (e.g., IDA, Ghidra, or Binary Ninja) and a working Docker installation. A Docker image with all required tools and course material will be provided.

Optional Agent-Assisted Exercises

External LLM access would be beneficial, but is not strictly required. Participants may use their own LLM subscriptions or API keys for selected optional agent-assisted exercises, but such access is not provided as part of the training.