Master of Science

thesisSystem administrators use application-level knowledge to identify anomalies in virtual appliances (VAs) and to recover from them. This process can be automated through an anomaly detection and recovery system. In this thesis, we claim that application-level policies defined over kernel-level application state can be effective for automatically detecting and mitigating the effects of malicious software in VAs. By combining user-defined application-level policies, virtual machine introspection (VMI), expert systems, and kernel-based state management techniques for anomaly detection and recovery, we are able to provide a favorable environment for the execution of applications in VAs. We use policies to specify the desired state of the VA based on an administrator's application-level knowledge. By using VMI we are able to generate a snapshot that represents the true internal state of the VA. An expert system evaluates the snapshot and identifies any violations. Potential violations include the execution of an irrelevant application, an unauthorized process, or an unfavorable environment configuration. The expert system also reasons about appropriate recovery strategies for each of the violations detected. The recovery strategy decided by the expert system is carried out by recovery tools so that the VA can be restored to an acceptable state. We evaluate the effectiveness of this approach for anomaly detection and repair by using it to detect and recover from the actions of different types malicious software targeting a web server VA. The system is shown to be effective in guarding the VA against the actions of a kernel-exploit kit, a kernel rootkit, a user-space rootkit, and an application malware. For each of these attacks, the recovery component was able to restore the VA to an acceptable state. Although, the recovery actions carried out did not remove the malicious software, they substantially mitigated the harmful effects of the malicious software

Kernel Code Integrity Protection Based on a Virtualized Memory Architecture

Kernel rootkits pose significant challenges on defensive techniques as they run at the highest privilege level along with the protection systems. Modern architectural approaches such as the NX protection have been used in mitigating attacks, however determined attackers can still bypass these defenses with specifically crafted payloads. In this paper, we propose a virtualized Harvard memory architecture to address the kernel code integrity problem, which virtually separates the code fetch and data access on the kernel code to prevent kernel from code modifications. We have implemented the proposed mechanism in commodity operating system, and the experimental results show that our approach is effective and incurs very low overhead

Autoscopy: Detecting Pattern-Searching Rootkits via Control Flow Tracing

Traditional approaches to rootkit detection assume the execution of code at a privilege level below that of the operating system kernel, with the use of virtual machine technologies to enable the detection system itself to be immune from the virus or rootkit code. In this thesis, we approach the problem of rootkit detection from the standpoint of tracing and instrumentation techniques, which work from within the kernel and also modify the kernel\u27s run-time state to detect aberrant control flows. We wish to investigate the role of emerging tracing frameworks (Kprobes, DTrace etc.) in enforcing operating system security without the reliance on a full-blown virtual machine just for the purposes of such policing. We first build a novel rootkit prototype that uses pattern-searching techniques to hijack hooks embedded in dynamically allocated memory, which we present as a showcase of emerging attack techniques. We then build an intrusion detection system-- autoscopy, atop kprobes, that detects anomalous control flow patterns typically exhibited by rootkits within a running kernel. Furthermore, to validate our approach, we show that we were able to successfully detect 15 existing Linux rootkits. We also conduct performance analyses, which show the overhead of our system to range from 2% to 5% on a wide range of standard benchmarks. Thus by leveraging tracing frameworks within operating systems, we show that it is possible to introduce real-world security in devices where performance and resource constraints are tantamount to security considerations

Practical assessment of Biba integrity for TCG-enabled platforms

Checking the integrity of an application is necessary to determine if the latter will behave as expected. The method defined by the Trusted Computing Group consists in evaluating the fingerprints of the platform hardware and software components required for the proper functioning of the application to be assessed. However, this only ensures that a process was working correctly at load-time but not for its whole life-cycle. Policy-Reduced Integrity Measurement Architecture (PRIMA) addresses this problem by enforcing a security policy that denies information flows from potentially malicious processes to an application target of the evaluation and its dependencies (requirement introduced by CW-Lite, an evolution of the Biba integrity model). Given the difficulty of deploying PRIMA (as platform administrators have to tune their security policies to satisfy the CW-Lite requirements) we propose in this paper Enhanced IMA, an extended version of the Integrity Measurement Architecture (IMA) that, unlike PRIMA, works almost out of the box and just reports information flows instead of enforcing them. In addition, we introduce a model to evaluate the information reported by Enhanced IMA with existing technique

Towards Understanding Systems Through User Interactions

Modern computer systems are complex. Even in the best of conditions, it can be difficult to understand the behavior of the system and identify why certain actions are occurring. Existing systems attempt to provide insight by reviewing the effects of actions on the system and estimating their cause. As computer systems are strongly driven by actions of the user, we propose an approach to identify processes which have interacted with the user and provide data to which system behaviors were caused by the user. We implement three sensors within the graphical user interface capable of extracting the necessary information to identify these processes. We show our instrumentation is effective in characterizing applications with an on-screen presence, and provide data towards the determination of user intentions. We prove that our method for obtaining the information from the user interface can be done in an efficient manner with minimal overheads