Detecting buffer overflows using testcase synthesis and code instrumentation

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 143-146).The research presented in this thesis aims to improve existing approaches to dynamic buffer overflow detection by developing a system that utilizes code instrumentation and adaptive test case synthesis to find buffer overflows and corresponding failure-inducing inputs automatically. An evaluation of seven modern dynamic buffer overflow detection tools determined that C Range Error Detector (CRED) is capable of providing fine-grained buffer access information necessary for the creation of this system. CRED was also selected because of its ability to provide comprehensive error reports and compile complex programs with reasonable performance overhead. CRED was extended to provide appropriate code instrumentation for the adaptive testing system, which also includes a test case synthesizer that uses data perturbation techniques on legal inputs to produce new test cases, and an analytical module that evaluates the effectiveness of these test cases. Using information provided by code instrumentation in further test case generation creates a feedback loop that enables a focused exploration of the input space and faster buffer overflow detection. Applying the adaptive testing system to jabberd, a Jabber Instant Messaging server, demonstrates its effectiveness in finding buffer overflows and its advantages over existing dynamic testing systems.(cont.) Adaptive test case synthesis using CRED to provide buffer access information for feedback discovered 6 buffer overflows in jabberd using only 53 messages, while dynamic testing using random messages generated from a protocol description found only 4 overflows after sending 10,000 messages.by Michael A. Zhivich.M.Eng

Solvers for Type Recovery and Decompilation of Binaries

Reconstructing the meaning of a program from its binary is known as reverse engineering. Since reverse engineering is ultimately a search for meaning, there is growing interest in inferring a type (a meaning) for the elements of a binary in a consistent way. Currently there is no consensus on how best to achieve this, with the few existing approaches utilising ad-hoc techniques which lack any formal basis. Moreover, previous work does not answer (or even ask) the fundamental question of what it means for recovered types to be correct. This thesis demonstrates how solvers for Satisfiability Modulo Theories (SMT) and Constraint Handling Rules (CHR) can be leveraged to solve the type reconstruction problem. In particular, an approach based on a new SMT theory of rational tree constraints is developed and evaluated. The resulting solver, based on the reification mechanisms of Prolog, is shown to scale well, and leads to a reification driven SMT framework that supports rapid implementation of SMT solvers for different theories in just a few hundred lines of code. The question of how to guarantee semantic relevance for reconstructed types is answered with a new and semantically-founded approach that provides strong guarantees for the reconstructed types. Key to this approach is the derivation of a witness program in a type safe high-level language alongside the reconstructed types. This witness has the same semantics as the binary, is type correct by construction, and it induces a (justifiable) type assignment on the binary. Moreover, the approach, implemented using CHR, yields a type-directed decompiler. Finally, to evaluate the flexibility of reificiation-based SMT solving, the SMT framework is instantiated with theories of general linear inequalities, integer difference problems and octagons. The integer difference solver is shown to perform competitively with state-of-the-art SMT solvers. Two new algorithms for incremental closure of the octagonal domain are presented and proven correct. These are shown to be both conceptually simple, and offer improved performance over existing algorithms. Although not directly related to reverse engineering, these results follow from the work on SMT solver construction