Abstract

Modern reverse engineering increasingly relies on automation, custom tooling, and agent-assisted workflows. But these approaches quickly run into limits when binaries actively resist analysis through control-flow obfuscation, virtualization, mixed Boolean-Arithmetic (MBA), and other transformations. This training teaches the practical deobfuscation workflows needed to break such protections and to make automated reverse-engineering workflows effective on real-world targets.

Participants first learn how modern obfuscation techniques complicate reverse engineering, and then gradually build the deobfuscation techniques required to attack them in hands-on sessions. Along the way, they deepen their understanding of program analysis and learn when and how to apply different techniques in practice.

We begin with core obfuscation patterns and practical ways to attack them. Participants learn how to recognize common protection mechanisms, understand how they interfere with static analysis and decompilation, and gain traction through careful manual analysis and targeted cleanup. From there, the course builds toward more powerful reasoning with SMT-based analysis to prove properties of code, reason about complex computations, and verify whether MBA-heavy expressions are equivalent to simpler rewrites.

To scale the analysis, the training introduces intermediate representations and compiler-style simplification passes that help normalize protected code and expose higher-level control-flow structures. On top of this, participants use symbolic execution to automate larger parts of deobfuscation, combine symbolic reasoning with SMT solving to attack opaque predicates, and explore feasible execution paths through protected code.

Building on these foundations, the course then returns to virtualization-based obfuscation. Participants learn how to use symbolic execution to identify virtual machine components, reason about instruction handlers, and write custom disassemblers and analysis helpers to reconstruct original program semantics. From there, we cover program synthesis as a way to simplify code based on behavior, recover the semantics of obfuscated expressions and handlers, and verify the correctness of the resulting simplifications.

By the end of the training, participants will understand how manual analysis, SMT solving, intermediate representations, symbolic execution, program synthesis, and custom tooling fit together in end-to-end deobfuscation workflows. The course closes with a module on agent-assisted deobfuscation, where we cover the setup of analysis environments, the orchestration of disassemblers, decompilers, symbolic tooling, and custom scripts, and the partial automation of iterative deobfuscation loops. Participants also learn how to steer these systems with reusable skills or playbooks and validation checks. Finally, we examine the limits of agent-assisted deobfuscation on protected binaries and discuss practical strategies for cases that still require deeper analysis and careful human guidance.

Teaching

The training has a strong focus on hands-on exercises. Short lecture segments provide the background needed to understand when a method is useful, how it works, and where it reaches its limits. The exercises then show how to apply these methods to real deobfuscation problems, build small, purpose-built tools, and combine individual techniques into practical workflows. The trainer actively supports participants during the exercises, and after each task we discuss different solutions in class. Participants also receive detailed reference solutions that they can use during and after the course.

While the hands-on sessions primarily focus on x86 assembly, the underlying tools and techniques also transfer to other architectures such as MIPS, PPC, and ARM.

Class Outline

The training roughly follows the outline below:

• Introduction to Code (De)obfuscation

  • Motivation
  • Why protected binaries break standard reverse-engineering workflows
  • Core program-analysis techniques used throughout the course
  • Obfuscation foundations and manual analysis
  • Opaque predicates
  • Control-flow flattening
  • Mixed Boolean-Arithmetic (MBA)
  • Virtual machines
  • Virtual machine hardening
  • Recognizing protection patterns and gaining analyst traction

• SMT-Based Program Analysis

  • SAT and SMT solvers
  • Encoding program-analysis problems for SMT solvers
  • Proving semantic equivalence
  • Proving properties of code
  • Solving complex program constraints
  • Reasoning about MBA-heavy expressions and validating rewrites

• Intermediate Representations and Compiler-Style Simplification

  • Intermediate representations for reverse engineering
  • Static single assignment (SSA)
  • Dead code elimination
  • Constant propagation / folding
  • Normalizing and simplifying obfuscated code
  • Recovering higher-level control-flow structure

• Symbolic Execution Foundations

  • Symbolic and semantic simplification of obfuscated code
  • Large-scale analysis and automation
  • Interaction with SMT solvers
  • Breaking arithmetic opaque predicates
  • Exploration of feasible program paths

• Symbolic Execution for Attacking Virtualization-based Obfuscation

  • Automated analysis of virtual machine components
  • Reasoning about instruction handlers
  • Writing disassemblers and analysis helpers based on symbolic execution
  • Reconstructing original program semantics

• Program Synthesis

  • Concept of program synthesis
  • Learning code semantics from input/output behavior
  • Obtaining input/output pairs from code
  • Simplifying large expression trees
  • Proving the correctness of simplifications
  • Simplifying MBAs via synthesis
  • Learning the semantics of VM instruction handlers

• Automation and Agent-Assisted Deobfuscation

  • Strategies for moving from manual analysis toward automated deobfuscation pipelines
  • Environment and tool setup for agent-assisted workflows
  • Connecting disassemblers, decompilers, symbolic tooling, and custom scripts
  • Tool orchestration across multi-step deobfuscation workflows
  • Guided tool generation for specific deobfuscation problems
  • Partial automation of iterative transformation and reanalysis loops
  • Reusable playbooks, validation strategies, and practical limits

Requirements and Recommendations

Prerequisites

The participants should have basic reverse engineering skills. Furthermore, they should be familiar with x86 assembly and Python.

Software Requirements

Students should have a disassembler of their choice (e.g., IDA, Ghidra, or Binary Ninja) and a working Docker installation. A Docker image with all required tools and course material will be provided.