Environment-Sensitive Intrusion Detection

Abstract. We perform host-based intrusion detection by constructing a model from a program’s binary code and then restricting the program’s execution by the model. We improve the effectiveness of such model-based intrusion detection systems by incorporating into the model knowledge of the environment in which the program runs, and by increasing the accuracy of our models with a new dataflow analysis algorithm for context-sensitive recovery of static data. The environment—configuration files, command-line parameters, and environment variables—constrains acceptable process execution. Environment dependencies added to a program model update the model to the current environment at every program execution. Our new static data-flow analysis associates a program’s data flows with specific calling contexts that use the data. We use this analysis to differentiate systemcall arguments flowing from distinct call sites in the program. Using a new average reachability measure suitable for evaluation of call-stackbased program models, we demonstrate that our techniques improve the precision of several test programs ’ models from 76 % to 100%

ACKNOWLEDGMENTS

The title page of this dissertation shows only my name, yet this document represents the culmination of years of research that succeeded only with the support and encouragement of a widespread collection of talented scientists. I am deeply indebted to my excellent co-advisers, Barton P. Miller and Somesh Jha. I think back five or six years and remember the student that I was before entering their tutelage, and I realize just how much I have grown because of their influence. Bart and Somesh embody the term “adviser”. They helped me understand the process of research and why that process is important. Beyond simply helping me complete a paper or finalize a set of experiments, Bart and Somesh taught me how to be a scientist. By their examples, I gained insights about how to teach and advise others. I hope that they never try to collect on my debt; I do not know how I could sufficiently repay them. I thank my reading committee—Bart Miller, Somesh Jha, Dieter van Melkebeek, Mike Swift, and David Page—for taking the time to study my research and to provide useful insights that improved the dissertation. Thank you for being the first five readers of this document. I thank my mother for probably being the sixth

Strengthening Software Self-Checksumming via Self-Modifying Code

Recent research has proposed self-checksumming as a method by which a program can detect any possibly malicious modification to its code. Wurster et al. developed an attack against such programs that renders code modifications undetectable to any self-checksumming routine. The attack replicated pages of program text and altered values in hardware data structures so that data reads and instruction fetches retrieved values from different memory pages. A cornerstone of their attack was its applicability to a variety of commodity hardware: they could alter memory accesses using only a malicious operating system. In this paper, we show that their page-replication attack can be detected by self-checksumming programs with self-modifying code. Our detection is efficient, adding less than 1 microsecond to each checksum computation in our experiments on three processor families, and is robust up to attacks using either costly interpretive emulation or specialized hardware

An Architecture for Generating Semantics-Aware Signatures

Identifying new intrusions and developing effective signatures that detect them is essential for protecting computer networks. We present Nemean, a system for automatic generation of intrusion signatures from honeynet packet traces. Our architecture is distinguished by its emphasis on a modular design framework that encourages independent development and modification of system components and protocol semantics awareness which allows for construction of signatures that greatly reduce false alarms. The building blocks of our architecture include transport and service normalization, intrusion profile clustering and automata learning that generates connection and session aware signatures. We demonstrate the potential of Nemean’s semantics-aware, resilient signatures through a prototype implementation. We use two datasets to evaluate the system: (i) a production dataset for false-alarm evaluation and (ii) a honeynet dataset for measuring detection rates. Signatures generated by Nemean for NetBIOS exploits had a 0% false-positive rate and a 0.04 % false-negative rate

Efficient context-sensitive intrusion detection

Model-based intrusion detection compares a process’s execution against a program model to detect intrusion attempts. Models constructed from static program analysis have historically traded precision for efficiency. We address this problem with our Dyck model, the first efficient statically-constructed context-sensitive model. This model specifies both the correct sequences of system calls that a program can generate and the stack changes occurring at function call sites. Experiments demonstrate that the Dyck model is an order of magnitude more precise than a context-insensitive finite state machine model. With null call squelching, a dynamic technique to bound cost, the Dyck model operates in time similar to the contextinsensitive model. We also present two static analysis techniques designed to counter mimicry and evasion attacks. Our branch analysis identifies between 32 % and 64 % of our test programs’ system call sites as affecting control flow via their return values. Interprocedural argument capture of general values recovers 32 % to 69 % more arguments than previously reported techniques. 1

Detecting manipulated remote call streams

In the Internet, mobile code is ubiquitous and includes such examples as browser plug-ins, Java applets, and document macros. In this paper, we address an important vulnerability in mobile code security that exists in remote execution systems such as Condor, Globus, and SETI@Home. These systems schedule user jobs for execution on remote idle machines. However, they send most of their important system calls back to the local machine for execution. Hence, an evil process on the remote machine can manipulate a user’s job to send destructive system calls back to the local machine. We have developed techniques to remotely detect such manipulation. Before the job is submitted for remote execution, we construct a model of the user’s binary program using static analysis. This binary analysis is applicable to commodity remote execution systems and applications. During remote job execution, the model checks all system calls arriving at the local machine. Execution is only allowed to continue while the model remains valid. We begin with a finite-state machine model that accepts sequences of system calls and then build optimizations into the model to improve its precision and efficiency. We also propose two program transformations, renaming and null call insertion, that have a significant impact on the precision and efficiency. As a desirable side-effect, these techniques also obfuscate the program, thus making it harder for the adversary to reverse engineer the code. We have implemented a simulated remote execution environment to demonstrate how optimizations and transformations of the binary program increase the precision and efficiency. In our test programs, unoptimized models increase run-time by 0.5 % or less. At moderate levels of optimization, run-time increases by less than 13 % with precision gains reaching 74%

An Architecture for Generating Semantics-Aware Signatures

Identifying new intrusions and developing effective signatures that detect them is essential for protecting computer networks. We present Nemean, a system for automatic generation of intrusion signatures from honeynet packet traces. Our architecture is distinguished by its emphasis on a modular design framework that encourages independent development and modification of system components and protocol semantics awareness which allows for construction of signatures that greatly reduce false alarms. The building blocks of our architecture include transport and service normalization, intrusion profile clustering and automata learning that generates connection and session aware signatures. We demonstrate the potential of Nemean’s semantics-aware, resilient signatures through a prototype implementation. We use two datasets to evaluate the system: (i) a production dataset for false-alarm evaluation and (ii) a honeynet dataset for measuring detection rates. Signatures generated by Nemean for NetBIOS exploits had a 0% false-positive rate and a 0.04 % false-negative rate

On Effective Model-Based Intrusion Detection

Model-based intrusion detectors restrict program execution to a previously computed model of expected behavior. We consider two classes of attacks against these systems: bypass attacks that evade detection by avoiding the detection system altogether, and transformational attacks that alter a detected attack into a semantically-equivalent attack that goes undetected. Recent detection approaches are problematic and do not effectively address these threats. We see reductions or outright failures in effectiveness and efficiency when systems (1) monitor execution at the library call interface, (2) provide accuracy via inlining of statically-constructed program models, or (3) use simplistic analysis of indirect function calls. Attacks can defeat library-call monitors by directly executing operating system kernel traps. Inlined models grow exponentially large at the trap interface: models for several test programs are 12,000 to 38,000 times larger at the trap interface than at the library call interface. Naïve indirect call analysis produces models 14 to 177 times larger than models built with in-depth analysis and that are less able to detect attacks. In examining these issues, our aim is to reveal complexities of model-based detection that have not been previously well understood.

ABSTRACT An Auctioning Reputation System Based on Anomaly Detection

Existing reputation systems used by online auction houses do not address the concern of a buyer shopping for commodities—finding a good bargain. These systems do not provide information on the practices adopted by sellers to ensure profitable auctions. These practices may be legitimate, like imposing a minimum starting bid on an auction, or fraudulent, like using colluding bidders to inflate the final price in a practice known as shilling. We develop a reputation system to help buyers identify sellers whose auctions seem price-inflated. Our reputation system is based upon models that characterize sellers according to statistical metrics related to price inflation. We combine the statistical models with anomaly detection techniques to identify the set of suspicious sellers. The output of our reputation system is a set of values for each seller representing the confidence with which the system can say that the auctions of the seller are price-inflated. We evaluate our reputation system on 604 high-volume sellers who posted 37,525 auctions on eBay. Our system automatically pinpoints sellers whose auctions contain potential shill bidders. When we manually analyze these sellers ’ auctions, we find that many winning bids are at about the items ’ market values, thus undercutting a buyer’s ability to find a bargain and demonstrating the effectiveness of our reputation system