Robust and secure monitoring and attribution of malicious behaviors

Worldwide computer systems continue to execute malicious software that degrades the systemsâ performance and consumes network capacity by generating high volumes of unwanted traffic. Network-based detectors can effectively identify machines participating in the ongoing attacks by monitoring the traffic to and from the systems. But, network detection alone is not enough; it does not improve the operation of the Internet or the health of other machines connected to the network. We must identify malicious code running on infected systems, participating in global attack networks. This dissertation describes a robust and secure approach that identifies malware present on infected systems based on its undesirable use of network. Our approach, using virtualization, attributes malicious traffic to host-level processes responsible for the traffic. The attribution identifies on-host processes, but malware instances often exhibit parasitic behaviors to subvert the execution of benign processes. We then augment the attribution software with a host-level monitor that detects parasitic behaviors occurring at the user- and kernel-level. User-level parasitic attack detection happens via the system-call interface because it is a non-bypassable interface for user-level processes. Due to the unavailability of one such interface inside the kernel for drivers, we create a new driver monitoring interface inside the kernel to detect parasitic attacks occurring through this interface. Our attribution software relies on a guest kernelâ s data to identify on-host processes. To allow secure attribution, we prevent illegal modifications of critical kernel data from kernel-level malware. Together, our contributions produce a unified research outcome --an improved malicious code identification system for user- and kernel-level malware.Ph.D.Committee Chair: Giffin, Jonathon; Committee Member: Ahamad, Mustaque; Committee Member: Blough, Douglas; Committee Member: Lee, Wenke; Committee Member: Traynor, Patric

Detecting Hardware-assisted Hypervisor Rootkits within Nested Virtualized Environments

Virtual machine introspection (VMI) is intended to provide a secure and trusted platform from which forensic information can be gathered about the true behavior of malware within a guest. However, it is possible for malware to escape a guest into the host and for hypervisor rootkits, such as BluePill, to stealthily transition a native OS into a virtualized environment. This research examines the effectiveness of selected detection mechanisms against hardware-assisted virtualization rootkits (HAV-R) within a nested virtualized environment. It presents the design, implementation, analysis, and evaluation of a hypervisor rootkit detection system which exploits both processor and translation lookaside buffer-based mechanisms to detect hypervisor rootkits within a variety of nested virtualized systems. It evaluates the effects of different types of virtualization on hypervisor rootkit detection and explores the effectiveness in-guest HAV-R obfuscation efforts. The results provide convincing evidence that the HAV-Rs are detectable in all SVMI scenarios examined, regardless of HAV-R or virtualization type. Also, that the selected detection techniques are effective at detection of HAV-R within nested virtualized environments, and that the type of virtualization implemented in a VMI system has minimal to no effect on HAV-R detection. Finally, it is determined that in-guest obfuscation does not successfully obfuscate the existence of HAV-R

Secure Virtualization and Multicore Platforms State-of-the-Art report

SVaM

Dynamic Shifting of Virtual Network Topologies for Network Attack Prevention

Computer networks were not designed with security in mind, making research into the subject of network security vital. Virtual Networks are similar to computer networks, except the components of a Virtual Network are in software rather than hardware. With the constant threat of attacks on networks, security is always a big concern, and Virtual Networks are no different. Virtual Networks have many potential attack vectors similar to physical networks, making research into Virtual Network security of great importance. Virtual Networks, since they are composed of virtualized network components, have the ability to dynamically change topologies. In this paper, we explore Virtual Networks and their ability to quickly shift their network topology. We investigate the potential use of this flexibility to protect network resources and defend against malicious activities. To show the ability of reactively shifting a Virtual Network’s topology to se- cure a network, we create a set of four experiments, each with a different dynamic topology shift, or “dynamic defense”. These four groups of experiments are called the Server Protection, Isolated Subnet, Distributed Port Group, and Standard Port Group experiments. The Server Protection experiments involve detecting an attack against a server and shifting the server behind a protected subnet. The other three sets of experiments, called Attacker Prevention experiments, involve detecting a malicious node in the internal network and initiating a dynamic de- fense to move the attacker behind a protected subnet. Each Attacker Prevention experiment utilizes a different dynamic defense to prevent the malicious node from attacking the rest of the Virtual Network. For each experiment, we run 6 different network attacks to validate the effectiveness of the dynamic defenses. The network attacks utilized for each experiment are ICMP Flooding, TCP Syn Flooding, Smurf attack, ARP Spoofing, DNS Spoofing, and NMAP Scanning. Our validation shows that our dynamic defenses, outside of the standard port group, are very effective in stopping each attack, consistently lowering the at- tacks’ success rate significantly. The Standard Port Group was the one dynamic defense that is ineffective, though there are also a couple of experiments that could benefit from being run with more attackers and with different situations to fully understand the effectiveness of the defenses. We believe that, as Virtual Networks become more common and utilized outside of data centers, the ability to dynamically shift topology can be used for network security purposes

Analysis and Detection of Heap-based Malwares Using Introspection in a Virtualized Environment

Malware detection and analysis is a major part of computer security. There is an arm race between security experts and malware developers to develop various techniques to secure computer systems and to find ways to circumvent these security methods. In recent years process heap-based attacks have increased significantly. These attacks exploit the system under attack via the heap, typically by using a heap spraying attack. The main drawback with existing techniques is that they either consume too many resources or are complicated to implement. Our work in this thesis focuses on new methods which offloads process heap analysis for guest Virtual Machines (VM) to the privileged domain using Virtual Machine Introspection (VMI) in a Cloud environment. VMI provides us with a seamless, non-intrusive and invisible (to malwares) way of observing the memory and state of VMs without raising red flags for the malwares

Secure Offloading of Intrusion Detection Systems from VMs with Intel SGX

Virtual machines (VMs) inside clouds need to be monitored using intrusion detection systems (IDS). Since host-based IDS can be easily disabled by intruders, IDS offloading with VM introspection (VMI) is used to securely run IDS outside a target VM. However, offloaded IDS can be still attacked because it runs on top of a vulnerable operating system (OS). Various systems have been proposed to protect offloaded IDS, but no systems provide an appropriate execution environment to IDS. This paper proposes SGmonitor for enabling the secure execution of IDS offloaded from VMs inside clouds using Intel SGX. SGmonitor executes IDS in SGX enclaves and preserves confidentiality and integrity. It provides secure VMI for memory and storage by using encryption and integrity checking. To make the development of offloaded IDS easier, it provides the in-kernel API to in-enclave IDS and enables transparent access to OS data in VMs. We have implemented SGmonitor in Xen with SGX support and showed that the overhead of in-enclave IDS was 31% in compensation for much stronger security.2021 IEEE 14th International Conference on Cloud Computing (CLOUD), September 5-10, 2021, Chicago, IL, US

A survey on security issues and solutions at different layers of Cloud computing

Cloud computing offers scalable on-demand services to consumers with greater flexibility and lesser infrastructure investment. Since Cloud services are delivered using classical network protocols and formats over the Internet, implicit vulnerabilities existing in these protocols as well as threats introduced by newer architectures raise many security and privacy concerns. In this paper, we survey the factors affecting Cloud computing adoption, vulnerabilities and attacks, and identify relevant solution directives to strengthen security and privacy in the Cloud environment