TUM Logo

Automated Decompilation of Virtual-Machine-Obfuscated Binaries

Automated Decompilation of Virtual-Machine-Obfuscated Binaries

Supervisor(s): Fabian Kilger
Status: finished
Topic: Others
Author: Leonard Oscar Auer
Submission: 2024-10-01
Type of Thesis: Guided Research

Description

Virtual machine obfuscation is one of the strongest obfuscation techniques often used by malware 
authors to hide their malicious code from reverse engineers: The original code is transformed into 
a custom bytecode using a randomized virtual instruction set. State-of-the-art research attempts to 
automate the analysis and deobfuscation of such samples. We build upon our previous work combining 
analysis using symbolic execution with decompilation using Ghidra and SLEIGH processor specifications 
and expand it to increase the automation rate. Our evaluation results show that using our automation 
and improvements, the proof of concept implementation can correctly recover most samples and is able 
to handle large sample sets in a reasonable amount of time.