TUM Logo

Improving Performance of Blackbox and Coverage-Guided Fuzzing Using Modified System Calls

Improving Performance of Blackbox and Coverage-Guided Fuzzing Using Modified System Calls

Supervisor(s): Konrad Hohentanner, Florian Jakobsmeier
Status: finished
Topic: Others
Author: Axel Strodel
Submission: 2021-03-15
Type of Thesis: Bachelorthesis
Thesis topic in co-operation with the Fraunhofer Institute for Applied and Integrated Security AISEC, Garching

Description

With many confidential activities, such as shopping or banking,
accessible online and a growing number of people using them, it is
extremely important to identify and fix software bugs as soon as possible.
One way to achieve this is fuzzing, an automated software testing
technique that repeatedly executes a piece of software with randomly
generated inputs attempting to cause unintended behavior in the targeted
program.
The reactions to these inputs are monitored and analyzed, as unexpected
behavior indicates programming errors that could be used by attackers.
In this thesis we explore how modified system calls can be used in
fuzzing and measure how they affect the performance of fuzzing a target.
We present an execution environment that isolates the fuzzing process
and allows us to intercept and modify system calls made by a fuzzed
target both for blackbox fuzzing and coverage-guided fuzzing with
available source code.
For our blackbox approach we use Secure Computing's userspace
notifications and for our coverage-guided method we target library
wrapper functions for system call interception.
Our blackbox approach is able to provide a better isolation of the
target, but imposes an overhead on the fuzzing setup decreasing performance.
On the  contrary, our method for coverage-guided fuzzing manages to
notably increase the number of test cases executed compared to an
unmodified setup, but is only able to improve isolation of the tested
software in a limited form.
A speedup for each run of a target provides a significant benefit for
fuzzing in general, as it reduces the time spent on program executions
by a constant factor.
This leads to more test runs in a shorter time frame, allowing a target
program to be tested more thoroughly in the same time in comparison to
working without modified system calls.
Providing an isolated environment that allows a program to still be
fuzzed without negative impacts, enables more robust fuzzing of
low-level and system critical applications.