Assessing performance overhead of Virtual Machine Introspection and its suitability for malware analysis

Virtual Machine Introspection is the process of introspecting guest VM’s memory and reconstructing the state of the guest operating system. Due to its isolation, stealth and full visibility of the monitored target, VMI lends itself well for security monitoring and malware analysis. The topics covered in this thesis include operating system and hypervisor concepts, the semantic gap issue, VMI techniques and implementations, applying VMI for malware analysis, and analysis of the performance overhead. The behaviour and magnitude of the performance overhead associated with doing virtual machine introspection is analysed with five different empirical test cases. The intention of the tests is to estimate the costs of a single trapped event, determine the feasibility of various monitoring sensors from usability and stealth perspective, and analyse the behaviour of performance overhead. Various VMI-based tools were considered for the measurement, but DRAKVUF was chosen as it is the most advanced tool available. The test cases go as follows. The chosen load is first executed without any monitoring to determine the baseline execution time. Then a DRAKVUF monitoring plugin is turned on and the load is executed again. After both measurements have been made, the difference between the two execution times is the time spent executing monitoring code. The execution overhead is then determined by calculating the difference between the two execution times and dividing it by the baseline execution time. The disc consumption and execution overhead of a sensor, which captures removed files is small enough to be deployed as a monitoring solution. The performance overhead of system call monitoring sensor is dependant on the number of issued system calls. Loads which issue large numbers of system calls cause high performance overhead. The performance overhead of such loads can be limited by monitoring a subset of all system calls

CyberGuarder: a virtualization security assurance architecture for green cloud computing

Cloud Computing, Green Computing, Virtualization, Virtual Security Appliance, Security Isolation

Robust and secure monitoring and attribution of malicious behaviors

Worldwide computer systems continue to execute malicious software that degrades the systemsâ performance and consumes network capacity by generating high volumes of unwanted traffic. Network-based detectors can effectively identify machines participating in the ongoing attacks by monitoring the traffic to and from the systems. But, network detection alone is not enough; it does not improve the operation of the Internet or the health of other machines connected to the network. We must identify malicious code running on infected systems, participating in global attack networks. This dissertation describes a robust and secure approach that identifies malware present on infected systems based on its undesirable use of network. Our approach, using virtualization, attributes malicious traffic to host-level processes responsible for the traffic. The attribution identifies on-host processes, but malware instances often exhibit parasitic behaviors to subvert the execution of benign processes. We then augment the attribution software with a host-level monitor that detects parasitic behaviors occurring at the user- and kernel-level. User-level parasitic attack detection happens via the system-call interface because it is a non-bypassable interface for user-level processes. Due to the unavailability of one such interface inside the kernel for drivers, we create a new driver monitoring interface inside the kernel to detect parasitic attacks occurring through this interface. Our attribution software relies on a guest kernelâ s data to identify on-host processes. To allow secure attribution, we prevent illegal modifications of critical kernel data from kernel-level malware. Together, our contributions produce a unified research outcome --an improved malicious code identification system for user- and kernel-level malware.Ph.D.Committee Chair: Giffin, Jonathon; Committee Member: Ahamad, Mustaque; Committee Member: Blough, Douglas; Committee Member: Lee, Wenke; Committee Member: Traynor, Patric

Hypervisor Integrity Measurement Assistant

An attacker who has gained access to a computer may want to upload or modify configuration files, etc., and run arbitrary programs of his choice. We can severely restrict the power of the attacker by having a white-list of approved file checksums and preventing the kernel from loading loading any file with a bad checksum. The check may be placed in the kernel, but that requires a kernel that is prepared for it. The check may also be placed in a hypervisor which intercepts and prevents the kernel from loading a bad file. We describe the implementation of and give performance results for two systems. In one the checksumming, or integrity measurement, and decision is performed by the hypervisor instead of the OS. In the other only the final integrity decision is done in the hypervisor. By moving the integrity check out from the VM kernel it becomes harder for the intruder to bypass the check. We conclude that it is technically possible to put file integrity control into the hypervisor, both for kernels without and with pre-compiled support for integrity measurement

Secure Virtualization and Multicore Platforms State-of-the-Art report

SVaM

Master of Science

thesisSystem administrators use application-level knowledge to identify anomalies in virtual appliances (VAs) and to recover from them. This process can be automated through an anomaly detection and recovery system. In this thesis, we claim that application-level policies defined over kernel-level application state can be effective for automatically detecting and mitigating the effects of malicious software in VAs. By combining user-defined application-level policies, virtual machine introspection (VMI), expert systems, and kernel-based state management techniques for anomaly detection and recovery, we are able to provide a favorable environment for the execution of applications in VAs. We use policies to specify the desired state of the VA based on an administrator's application-level knowledge. By using VMI we are able to generate a snapshot that represents the true internal state of the VA. An expert system evaluates the snapshot and identifies any violations. Potential violations include the execution of an irrelevant application, an unauthorized process, or an unfavorable environment configuration. The expert system also reasons about appropriate recovery strategies for each of the violations detected. The recovery strategy decided by the expert system is carried out by recovery tools so that the VA can be restored to an acceptable state. We evaluate the effectiveness of this approach for anomaly detection and repair by using it to detect and recover from the actions of different types malicious software targeting a web server VA. The system is shown to be effective in guarding the VA against the actions of a kernel-exploit kit, a kernel rootkit, a user-space rootkit, and an application malware. For each of these attacks, the recovery component was able to restore the VA to an acceptable state. Although, the recovery actions carried out did not remove the malicious software, they substantially mitigated the harmful effects of the malicious software

Analysis and Detection of Heap-based Malwares Using Introspection in a Virtualized Environment

Malware detection and analysis is a major part of computer security. There is an arm race between security experts and malware developers to develop various techniques to secure computer systems and to find ways to circumvent these security methods. In recent years process heap-based attacks have increased significantly. These attacks exploit the system under attack via the heap, typically by using a heap spraying attack. The main drawback with existing techniques is that they either consume too many resources or are complicated to implement. Our work in this thesis focuses on new methods which offloads process heap analysis for guest Virtual Machines (VM) to the privileged domain using Virtual Machine Introspection (VMI) in a Cloud environment. VMI provides us with a seamless, non-intrusive and invisible (to malwares) way of observing the memory and state of VMs without raising red flags for the malwares

Automated Virtual Machine Introspection for Host-Based Intrusion Detection

This thesis examines techniques to automate configuration of an intrusion detection system utilizing hardware-assisted virtualization. These techniques are used to detect the version of a running guest operating system, automatically configure version-specific operating system information needed by the introspection library, and to locate and monitor important operating system data structures. This research simplifies introspection library configuration and is a step toward operating system independent introspection. An operating system detection algorithm and Windows virtual machine system service dispatch table monitor are implemented using the Xen hypervisor and a modified version of the XenAccess library. All detection and monitoring is implemented from the Xen management domain. Results of the operating system detection are used to initialize the XenAccess library. Library initialization time and kernel symbol retrieval are compared to the standard library. The algorithm is evaluated using nine versions of the Windows operating system. The system service dispatch table monitor is evaluated using the Agony and ProAgent rootkits. The automation techniques successfully detect the operating system and system service dispatch table hooks for the nine Windows versions tested. The modified XenAccess library exhibits an average initialization speedup of 1.9. Kernel symbol lookup is 10 times faster, on average. The hook detector is able to detect all hooks used by both rookits