Selective Dynamic Analysis of Virtualized Whole-System Guest Environments

Dynamic binary analysis is a prevalent and indispensable technique in program analysis. While several dynamic binary analysis tools and frameworks have been proposed, all suffer from one or more of: prohibitive performance degradation, a semantic gap between the analysis code and the execution under analysis, architecture/OS specificity, being user-mode only, and lacking flexibility and extendability. This dissertation describes the design of the Dynamic Executable Code Analysis Framework (DECAF), a virtual machine-based, multi-target, whole-system dynamic binary analysis framework. In short, DECAF seeks to address the shortcomings of existing whole-system dynamic analysis tools and extend the state of the art by utilizing a combination of novel techniques to provide rich analysis functionality without crippling amounts of execution overhead. DECAF extends the mature QEMU whole-system emulator, a type-2 hypervisor capable of emulating every instruction that executes within a complete guest system environment. DECAF provides a novel, hardware event-based method of just-in-time virtual machine introspection (VMI) to address the semantic gap problem. It also implements a novel instruction-level taint tracking engine at bitwise level of granularity, ensuring that taint propagation is sound and highly precise throughout the guest environment. A formal analysis of the taint propagation rules is provided to verify that most instructions introduce neither false positives nor false negatives. DECAF’s design also provides a plugin architecture with a simple-to-use, event-driven programming interface that makes it both flexible and extendable for a variety of analysis tasks. The implementation of DECAF consists of 9550 lines of C++ code and 10270 lines of C code. Its performance is evaluated using CPU2006 SPEC benchmarks, which show an average overhead of 605% for system wide tainting and 12% for VMI. Three platformneutral DECAF plugins - Instruction Tracer, Keylogger Detector, and API Tracer - are described and evaluated in this dissertation to demonstrate the ease of use and effectiveness of DECAF in writing cross-platform and system-wide analysis tools. This dissertation also presents the Virtual Device Fuzzer (VDF), a scalable fuzz testing framework for discovering bugs within the virtual devices implemented as part of QEMU. Such bugs could be used by malicious software executing within a guest under analysis by DECAF, so the discovery, reproduction, and diagnosis of such bugs helps to protect DECAF against attack while improving QEMU and any analysis platforms built upon QEMU. VDF uses selective instrumentation to perform targeted fuzz testing, which explores only the branches of execution belonging to virtual devices under analysis. By leveraging record and replay of memory-mapped I/O activity, VDF quickly cycles virtual devices through an arbitrarily large number of states without requiring a guest OS to be booted or present. Once a test case is discovered that triggers a bug, VDF reduces the test case to the minimum number of reads/writes required to trigger the bug and generates source code suitable for reproducing the bug during debugging and analysis. VDF is evaluated by fuzz testing eighteen QEMU virtual devices, generating 1014 crash or hang test cases that reveal bugs in six of the tested devices. Over 80% of the crashes and hangs were discovered within the first day of testing. VDF covered an average of 62.32% of virtual device branches during testing, and the average test case was minimized to a reproduction test case only 18.57% of its original size

A behavioural study in runtime analysis environments and drive-by download attacks

In the information age, the growth in availability of both technology and exploit kits have continuously contributed in a large volume of websites being compromised or set up with malicious intent. The issue of drive-by-download attacks formulate a high percentage (77%) of the known attacks against client systems. These attacks originate from malicious web-servers or compromised web-servers and attack client systems by pushing malware upon interaction. Within the detection and intelligence gathering area of research, high-interaction honeypot approaches have been a longstanding and well-established technology. These are however not without challenges: analysing the entirety of the world wide web using these approaches is unviable due to time and resource intensiveness. Furthermore, the volume of data that is generated as a result of a run-time analysis of the interaction between website and an analysis environment is huge, varied and not well understood. The volume of malicious servers in addition to the large datasets created as a result of run-time analysis are contributing factors in the difficulty of analysing and verifying actual malicious behaviour. The work in this thesis attempts to overcome the difficulties in the analysis process of log files to optimise malicious and anomaly behaviour detection. The main contribution of this work is focused on reducing the volume of data generated from run-time analysis to reduce the impact of noise within behavioural log file datasets. This thesis proposes an alternate approach that uses an expert lead approach to filtering benign behaviour from potentially malicious and unknown behaviour. Expert lead filtering is designed in a risk-averse method that takes into account known benign and expected behaviours before filtering the log file. Moreover, the approach relies upon behavioural investigation as well as potential for 5 system compromisation before filtering out behaviour within dynamic analysis log files. Consequently, this results in a significantly lower volume of data that can be analysed in greater detail. The proposed filtering approach has been implemented and tested in real-world context using a prudent experimental framework. An average of 96.96% reduction in log file size has been achieved which is transferable to behaviour analysis environments. The other contributions of this work include the understanding of observable operating system interactions. Within the study of behaviour analysis environments, it was concluded that run-time analysis environments are sensitive to application and operating system versions. Understanding key changes in operating systems behaviours within Windows is an unexplored area of research yet Windows is currently one of the most popular client operating system. As part of understanding system behaviours for the creation of behavioural filters, this study undertakes a number of experiments to identify the key behaviour differences between operating systems. The results show that there are significant changes in core processes and interactions which can be taken into account in the development of filters for updated systems. Finally, from the analysis of 110,000 potentially malicious websites, typical attacks are explored. These attacks actively exploited the honeypot and offer knowledge on a section of the active web-based attacks faced in the world wide web. Trends and attack vectors are identified and evaluated