Kernel-Assisted Debugging of Linux Applications

On Linux, most---if not all---debuggers use the ptrace debugging API to control their target processes. However, ptrace proves unsatisfactory for many malware analysis and reverse engineering tasks: So-called split-personality malware often adapts its behavior in the presence of a debugger, yet ptrace makes no attempt to hide from a target process. Furthermore, ptrace enforces a strict one-to-many relation meaning that while each tracer can trace many tracees, each tracee can only be controlled by at most one tracer. Simultaneously, the complex API and signal-based communications provide opportunities for erroneous usage. Previous works have identified the newer uprobes tracing API as a candidate for building a replacement for ptrace, but ultimately rejected it due to lack of practical use and documentation. Building upon uprobes, we introduce plutonium-dbg, a Linux kernel module providing debugging facilities independent of the limitations of ptrace alongside a GDB-compatible interface. Our approach aims to mitigate some of the design flaws of ptrace that make it both hard to use and easy to detect by malicious software. We show how plutonium-dbg's design and implementation remove many of the most frequently named issues with ptrace, and that our method improves on traditional ptrace-based debuggers (GDB and LLDB) when evaluated on software samples that attempt to detect the presence of a debugger.