TUM Logo

Optimizing Hypervisor Fuzzing through Compiler-Based Symbolic Execution Integration

Optimizing Hypervisor Fuzzing through Compiler-Based Symbolic Execution Integration

Supervisor(s): Manuel Andreas
Status: finished
Topic: Others
Author: Nguyen Dang Khoa
Submission: 2024-09-11
Type of Thesis: Masterthesis

Description

Cloud computing has transformed business operations, largely driven by advances in
virtualization technology. At the core are hypervisors, managing isolation and resource
allocation across multiple environments. Ensuring their security is critical to prevent
breaches, making it essential to implement robust security assessment techniques.
One of the most prevalent techniques for vulnerability discovery in software is
fuzzing, which is proven by the vast number of hypervisor fuzzing tools, as evidenced
by numerous studies and practical applications. Despite advancements in hypervisor
fuzzing tools, current solutions often encounter challenges such as limited branch
exploration, the restriction of testing only specific interfaces, and the need for tailored
inputs, leading to several potential vulnerabilities remaining undiscovered.
In this work, we propose a novel compiler-based symbolic execution approach
specifically designed for hypervisor testing. By adapting SymCC, we enable symbolic
execution for hypervisors by relocating symbolic backends from kernel space to user
space. This approach allows for efficient communication between the symbolic backend
and the hypervisor via hypercalls and memory writes, utilizing patched versions of
KVM and QEMU.
Our evaluation targets the hypercall interface of Xen hypervisors on x86_64 architectures.
Our prototype effectively generated inputs to cover all hypercall functions and
most subcommands, with a few exceptions due to permission checks or pointer constraints.
Although symbolic execution increases runtime by about three times compared
to standard execution, it remains highly effective and only requires hypervisor support
for hypercalls, without needing specialized hardware. Furthermore, our approach can
be easily adapted to test other hypervisors with minimal changes.
These findings demonstrate that our prototype is an efficient symbolic execution tool
for hypervisors, with strong potential for integration into hybrid fuzzing frameworks.