Discovering New Vulnerabilities in Computer Systems

Vulnerability research plays a key role in preventing and defending against malicious computer system exploitations. Driven by a multi-billion dollar underground economy, cyber criminals today tirelessly launch malicious exploitations, threatening every aspect of daily computing. to effectively protect computer systems from devastation, it is imperative to discover and mitigate vulnerabilities before they fall into the offensive parties\u27 hands. This dissertation is dedicated to the research and discovery of new design and deployment vulnerabilities in three very different types of computer systems.;The first vulnerability is found in the automatic malicious binary (malware) detection system. Binary analysis, a central piece of technology for malware detection, are divided into two classes, static analysis and dynamic analysis. State-of-the-art detection systems employ both classes of analyses to complement each other\u27s strengths and weaknesses for improved detection results. However, we found that the commonly seen design patterns may suffer from evasion attacks. We demonstrate attacks on the vulnerabilities by designing and implementing a novel binary obfuscation technique.;The second vulnerability is located in the design of server system power management. Technological advancements have improved server system power efficiency and facilitated energy proportional computing. However, the change of power profile makes the power consumption subjected to unaudited influences of remote parties, leaving the server systems vulnerable to energy-targeted malicious exploit. We demonstrate an energy abusing attack on a standalone open Web server, measure the extent of the damage, and present a preliminary defense strategy.;The third vulnerability is discovered in the application of server virtualization technologies. Server virtualization greatly benefits today\u27s data centers and brings pervasive cloud computing a step closer to the general public. However, the practice of physical co-hosting virtual machines with different security privileges risks introducing covert channels that seriously threaten the information security in the cloud. We study the construction of high-bandwidth covert channels via the memory sub-system, and show a practical exploit of cross-virtual-machine covert channels on virtualized x86 platforms

On the Infeasibility of Modeling Polymorphic Shellcode

Polymorphic malcode remains a troubling threat. The ability formal code to automatically transform into semantically equivalent variants frustrates attempts to rapidly construct a single, simple, easily verifiable representation. We present a quantitative analysis of the strengths and limitations of shellcode polymorphism and consider its impact on current intrusion detection practice. We focus on the nature of shellcode decoding routines. The empirical evidence we gather helps show that modeling the class of self-modifying code is likely intractable by known methods, including both statistical constructs and string signatures. In addition, we develop and present measures that provide insight into the capabilities, strengths, and weaknesses of polymorphic engines. In order to explore countermeasures to future polymorphic threats, we show how to improve polymorphic techniques and create a proof-of-concept engine expressing these improvements. Our results indicate that the class of polymorphic behavior is too greatly spread and varied to model effectively. Our analysis also supplies a novel way to understand the limitations of current signature-based techniques. We conclude that modeling normal content is ultimately a more promising defense mechanism than modeling malicious or abnormal content

Automated Virtual Machine Introspection for Host-Based Intrusion Detection

This thesis examines techniques to automate configuration of an intrusion detection system utilizing hardware-assisted virtualization. These techniques are used to detect the version of a running guest operating system, automatically configure version-specific operating system information needed by the introspection library, and to locate and monitor important operating system data structures. This research simplifies introspection library configuration and is a step toward operating system independent introspection. An operating system detection algorithm and Windows virtual machine system service dispatch table monitor are implemented using the Xen hypervisor and a modified version of the XenAccess library. All detection and monitoring is implemented from the Xen management domain. Results of the operating system detection are used to initialize the XenAccess library. Library initialization time and kernel symbol retrieval are compared to the standard library. The algorithm is evaluated using nine versions of the Windows operating system. The system service dispatch table monitor is evaluated using the Agony and ProAgent rootkits. The automation techniques successfully detect the operating system and system service dispatch table hooks for the nine Windows versions tested. The modified XenAccess library exhibits an average initialization speedup of 1.9. Kernel symbol lookup is 10 times faster, on average. The hook detector is able to detect all hooks used by both rookits

Thwarting Advanced Code-reuse Attacks

Code-reuse attacks are the leading mechanism by which attackers infiltrate systems. Various mitigation techniques have been proposed to defend against these attacks, the most prominent one being control-flow integrity (CFI). CFI is a principled approach that restricts all indirect control flows to adhere to a statically determined control-flow graph (CFG). CFI has gained widespread adoption in industry -- such as Microsoft Control Flow Guard and Intel Control-flow Enforcement Technology. However, recent attacks dubbed CFG mimicry attacks, like control flow bending and counterfeit object-oriented programming, have shown that code-reuse attacks are still possible without violating CFI. Furthermore, data-oriented programming (DOP) has generalized non-control data attacks to achieve Turing-complete computation; it accomplishes this by repeatedly corrupting non-control data to execute a sequence of instructions within the legitimate control flow of the program. In this dissertation, we present techniques to mitigate these advanced code-reuse attacks. First, this dissertation presents a novel approach to thwart advanced control flow attacks called ProxyCFI. ProxyCFI replaces all code pointers in a program with a less powerful construct: pointer proxies. Pointer proxies are random identifiers associated with each legitimate control flow edge in the program. Pointer proxy values are defined per-function and are re-randomized at program load time to mitigate their disclosure. To ensure that the approach covers the entire control flow of the program, we have a load-time verifier built-in the program loader that performs reachability analyses of the code and verify that there is no vulnerable control flow transfer. ProxyCFI delivers these protections incurring minimal performance overhead, while stopping a broad range of real-world attacks and achieving a 100% coverage of the RIPE x86-64 attack suite. Second, this dissertation evaluates the effectiveness of previously proposed stack layout randomization techniques against attacks that only utilize relative offset between allocations (e.g., data-oriented programming) and demonstrate that they are ineffective at stopping real-world DOP exploits. We then propose Smokestack, a runtime stack-layout randomization technique that addresses the problems with prior approaches. Smokestack instruments programs to randomize their stack layout at runtime for each invocation of a function. By doing so, Smokestack minimizes the utility of information gained in the probes of chained DOP attacks for later attack stages. Our evaluation on SPEC benchmarks and various real-world applications shows that Smokestack, with a cryptographically secure pseudo random generator, can stop DOP attacks with an average slowdown of 8.7%. Lastly, we present a technique to randomize heap allocations at runtime to prevent attackers from orchestrating advanced control flow attacks as well as DOP attacks through heap-resident variables. To this end, we explored the use of multi-variant execution (MVX) with each variant having uniquely seeded random heap allocators. This capability enables our system to automatically track heap allocation pointers without the need for storing explicit meta-data. We then re-randomize heap allocations to thwart attacks that perform runtime probes to discover allocations. This technique will provide modular heap allocation protection while maintaining compatibility with legacy binaries. In all, this thesis presents novel techniques that carve out a new space in advanced code-reuse attack protections, offering a protection strength as good or better than prior solutions. These techniques provide additional protections for advanced control flow attacks and DOP attacks, while incurring minimal performance overheads.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/155142/1/misiker_1.pd

Binary Program Integrity Models for Defeating Code-Reuse Attacks

During a cyber-attack, an adversary executes offensive maneuvers to target computer systems. Particularly, an attacker often exploits a vulnerability within a program, hijacks control-flow, and executes malicious code. Data Execution Prevention (DEP), a hardware-enforced security feature, prevents an attacker from directly executing the injected malicious code. Therefore, attackers have resorted to code-reuse attacks, wherein carefully chosen fragments of code within existing code sections of a program are sequentially executed to accomplish malicious logic. Code-reuse attacks are ubiquitous and account for majority of the attacks in the wild. On one hand, due to the wide use of closed-source software, binary-level solutions are essential. On the other hand, without access to source-code and debug-information, defending raw binaries is hard. A majority of defenses against code-reuse attacks enforce control-flow integrity , a program property that requires the runtime execution of a program to adhere to a statically determined control-flow graph (CFG) -- a graph that captures the intended flow of control within the program. While defenses against code-reuse attacks have focused on reducing the attack space, due to the lack of high-level semantics in the binary, they lack in precision, which in turn results in smaller yet significant attack space. This dissertation presents program integrity models aimed at narrowing the attack space available to execute code-reuse attacks. First, we take a semantic-recovery approach to restrict the targets of indirect branches in a binary. Then, we further improve the precision by recovering C++-level semantics, and enforce a strict integrity model that improves precision for virtual function calls in the binary. Finally, in order to further reduce the attack space, we take a different perspective on defense against code-reuse attacks, and introduce Stack-Pointer Integrity -- a novel integrity model targeted at ensuring the integrity of stack pointer as opposed to the instruction pointer. Our results show that the semantic-recovery-based approaches can help in significantly reducing the attack space by improving the precision of the underlying CFG. Function-level semantic recovery can eliminate 99.47% of inaccurate targets, whereas recovering virtual callsites and VTables at a C++ level can eliminate 99.99% of inaccurate targets

Security Evaluation of a Banking Fraud Analysis System

open7noThe significant growth of banking frauds, fueled by the underground economy of malware, raised the need for effective detection systems. Therefore, in last the years, banks have upgraded their security to protect transactions from frauds. State-of-the-art solutions detect frauds as deviations from customers’ spending habits. To the best of our knowledge, almost all existing approaches do not provide an in-depth model’s granularity and security analysis against elusive attacks. In this paper, we examine Banksealer, a decision support system for banking fraud analysis, evaluating the influence on the detection performance of the granularity at which the spending habits are modeled and its security against evasive attacks. First, we compare user-centric modeling, which builds a model for each user, with system-centric modeling, which builds a model for the entire system, from the point of view of the detection performance. Then, we assess the robustness of Banksealer against malicious attackers that are aware of the structure of the models in use. To this end, we design and implement a proof-of-concept attack tool that performs mimicry attacks, emulating a sophisticated attacker that cloaks frauds to avoid detection. We experimentally confirm the feasibility of such attacks, their cost and the effort required to an attacker in order to perform them. In addition, we discuss possible countermeasures. We provide a comprehensive evaluation on a large, real-world dataset obtained from one of the largest Italian banks.openCarminati, Michele; Polino, Mario; Continella, Andrea; Lanzi, Andrea; Maggi, Federico; Zanero, StefanoCarminati, Michele; Polino, Mario; Continella, Andrea; Lanzi, Andrea; Maggi, Federico; Zanero, Stefano; Zanero, Stefan

Developing a Qualia-Based Multi-Agent Architecture for Use in Malware Detection

Detecting network intruders and malicious software is a significant problem for network administrators and security experts. New threats are emerging at an increasing rate, and current signature and statistics-based techniques are not keeping pace. Intelligent systems that can adapt to new threats are needed to mitigate these new strains of malware as they are released. This research detects malware based on its qualia, or essence rather than its low-level implementation details. By looking for the underlying concepts that make a piece of software malicious, this research avoids the pitfalls of static solutions that focus on predefined bit sequence signatures or anomaly thresholds. This research develops a novel, hierarchical modeling method to represent a computing system and demonstrates the representation’s effectiveness by modeling the Blaster worm. Using Latent Dirichlet Allocation and Support Vector Machines abstract concepts are automatically generated that can be used in the hierarchical model for malware detection. Finally, the research outlines a novel system that uses multiple levels of individual software agents that sharing contextual relationships and information across different levels of abstraction to make decisions. This qualia-based system provides a framework for developing intelligent classification and decision-making systems for a number of application areas

Formalization and Detection of Host-Based Code Injection Attacks in the Context of Malware

The Host-Based Code Injection Attack (HBCIAs) is a technique that malicious software utilizes in order to avoid detection or steal sensitive information. In a nutshell, this is a local attack where code is injected across process boundaries and executed in the context of a victim process. Malware employs HBCIAs on several operating systems including Windows, Linux, and macOS. This thesis investigates the topic of HBCIAs in the context of malware. First, we conduct basic research on this topic. We formalize HBCIAs in the context of malware and show in several measurements, amongst others, the high prevelance of HBCIA-utilizing malware. Second, we present Bee Master, a platform-independent approach to dynamically detect HBCIAs. This approach applies the honeypot paradigm to operating system processes. Bee Master deploys fake processes as honeypots, which are attacked by malicious software. We show that Bee Master reliably detects HBCIAs on Windows and Linux. Third, we present Quincy, a machine learning-based system to detect HBCIAs in post-mortem memory dumps. It utilizes up to 38 features including memory region sparseness, memory region protection, and the occurence of HBCIA-related strings. We evaluate Quincy with two contemporary detection systems called Malfind and Hollowfind. This evaluation shows that Quincy outperforms them both. It is able to increase the detection performance by more than eight percent