Reducing the Attack Surface of Dynamic Binary Instrumentation Frameworks

Malicious applications pose as one of the most relevant issues in today’s technology scenario, being considered the root of many Internet security threats. In part, this owes the ability of malware developers to promptly respond to the emergence of new security solutions by developing artifacts to detect and avoid them. In this work, we present three countermeasures to mitigate recent mechanisms used by malware to detect analysis environments. Among these techniques, this work focuses on those that enable a malware to detect dynamic binary instrumentation frameworks, thus increasing their attack surface. To ensure the effectiveness of the proposed countermeasures, proofs of concept were developed and tested in a controlled environment with a set of anti-instrumentation techniques. Finally, we evaluated the performance impact of using such countermeasures

Empirical study to fingerprint public malware analysis services

The evolution of malicious software (malware) analysis tools provided controlled, isolated, and virtual environments to analyze malware samples. Several services are found on the Internet that provide to users automatic system to analyze malware samples, as VirusTotal, Jotti, or ClamAV, to name a few. Unfortunately, malware is currently incorporating techniques to recognize execution onto a virtual or sandbox environment. When analysis environment is detected, malware behave as a benign application or even show no activity. In this work, we present an empirical study and characterization of automatic public malware analysis services. In particular, we consider 26 different services. We also show a set of features that allow to easily fingerprint these services as analysis environments. Finally, we propose a method to mitigate fingerprinting

Towards the detection of isolation-aware malware

Malware analysis tools have evolved in the last years providing tightly controlled sandbox and virtualised environments where malware is analysed minimising potential harmful consequences. Unfortunately, malware has advanced in parallel, being currently able to recognise when is running in sandbox or virtual environments and then, behaving as a non-harmful application or even not executing at all. This kind of malware is usually called analysis-aware malware. In this paper, we propose a tool to detect the evasion techniques used by analysis-aware malware within sandbox or virtualised environments. Our tool uses Dynamic Binary Instrumentation to maintain the binary functionality while executing arbitrary code. We evaluate the tool under a set of well-known analysis-aware malware showing its current effectiveness. Finally, we discuss limitations of our proposal and future directions

Detecting Hardware-assisted Hypervisor Rootkits within Nested Virtualized Environments

Virtual machine introspection (VMI) is intended to provide a secure and trusted platform from which forensic information can be gathered about the true behavior of malware within a guest. However, it is possible for malware to escape a guest into the host and for hypervisor rootkits, such as BluePill, to stealthily transition a native OS into a virtualized environment. This research examines the effectiveness of selected detection mechanisms against hardware-assisted virtualization rootkits (HAV-R) within a nested virtualized environment. It presents the design, implementation, analysis, and evaluation of a hypervisor rootkit detection system which exploits both processor and translation lookaside buffer-based mechanisms to detect hypervisor rootkits within a variety of nested virtualized systems. It evaluates the effects of different types of virtualization on hypervisor rootkit detection and explores the effectiveness in-guest HAV-R obfuscation efforts. The results provide convincing evidence that the HAV-Rs are detectable in all SVMI scenarios examined, regardless of HAV-R or virtualization type. Also, that the selected detection techniques are effective at detection of HAV-R within nested virtualized environments, and that the type of virtualization implemented in a VMI system has minimal to no effect on HAV-R detection. Finally, it is determined that in-guest obfuscation does not successfully obfuscate the existence of HAV-R

Classifying malware attacks in IaaS cloud environments

In the last few years, research has been motivated to provide a categorization and classification of security concerns accompanying the growing adaptation of Infrastructure as a Service (IaaS) clouds. Studies have been motivated by the risks, threats and vulnerabilities imposed by the components within the environment and have provided general classifications of related attacks, as well as the respective detection and mitigation mechanisms. Virtual Machine Introspection (VMI) has been proven to be an effective tool for malware detection and analysis in virtualized environments. In this paper, we classify attacks in IaaS cloud that can be investigated using VMI-based mechanisms. This infers a special focus on attacks that directly involve Virtual Machines (VMs) deployed in an IaaS cloud. Our classification methodology takes into consideration the source, target, and direction of the attacks. As each actor in a cloud environment can be both source and target of attacks, the classification provides any cloud actor the necessary knowledge of the different attacks by which it can threaten or be threatened, and consequently deploy adapted VMI-based monitoring architectures. To highlight the relevance of attacks, we provide a statistical analysis of the reported vulnerabilities exploited by the classified attacks and their financial impact on actual business processes

Advanced Techniques to Detect Complex Android Malware

Android is currently the most popular operating system for mobile devices in the world. However, its openness is the main reason for the majority of malware to be targeting Android devices. Various approaches have been developed to detect malware. Unfortunately, new breeds of malware utilize sophisticated techniques to defeat malware detectors. For example, to defeat signature-based detectors, malware authors change the malware’s signatures to avoid detection. As such, a more effective approach to detect malware is by leveraging malware’s behavioral characteristics. However, if a behavior-based detector is based on static analysis, its reported results may contain a large number of false positives. In real-world usage, completing static analysis within a short time budget can also be challenging. Because of the time constraint, analysts adopt approaches based on dynamic analyses to detect malware. However, dynamic analysis is inherently unsound as it only reports analysis results of the executed paths. Besides, recently discovered malware also employs structure-changing obfuscation techniques to evade detection by state-of-the-art systems. Obfuscation allows malware authors to redistribute known malware samples by changing their structures. These factors motivate a need for malware detection systems that are efficient, effective, and resilient when faced with such evasive tactics. In this dissertation, we describe the developments of three malware detection systems to detect complex malware: DroidClassifier, GranDroid, and Obfusifier. DroidClassifier is a systematic framework for classifying network traffic generated by mobile malware. GranDroid is a graph-based malware detection system that combines dynamic analysis, incremental and partial static analysis, and machine learning to provide time-sensitive malicious network behavior detection with high accuracy. Obfusifier is a highly effective machine-learning-based malware detection system that can sustain its effectiveness even when malware authors obfuscate these malicious apps using complex and composite techniques. Our empirical evaluations reveal that DroidClassifier can successfully identify different families of malware with 94.33% accuracy on average. We have also shown GranDroid is quite effective in detecting network-related malware. It achieves 93.0% accuracy, which outperforms other related systems. Lastly, we demonstrate that Obfusifier can achieve 95% precision, recall, and F-measure, collaborating its resilience to complex obfuscation techniques. Adviser: Qiben Yan and Witawas Srisa-a

Nation-State Attackers and their Effects on Computer Security

Nation-state intelligence agencies have long attempted to operate in secret, but recent revelations have drawn the attention of security researchers as well as the general public to their operations. The scale, aggressiveness, and untargeted nature of many of these now public operations were not only alarming, but also baffling as many were thought impossible or at best infeasible at scale. The security community has since made many efforts to protect end-users by identifying, analyzing, and mitigating these now known operations. While much-needed, the security community's response has largely been reactionary to the oracled existence of vulnerabilities and the disclosure of specific operations. Nation-State Attackers, however, are dynamic, forward-thinking, and surprisingly agile adversaries who do not rest on their laurels and are continually advancing their efforts to obtain information. Without the ability to conceptualize their actions, understand their perspective, or account for their presence, the security community's advances will become antiquated and unable to defend against the progress of Nation-State Attackers. In this work, we present and discuss a model of Nation-State Attackers that can be used to represent their attributes, behavior patterns, and world view. We use this representation of Nation-State Attackers to show that real-world threat models do not account for such highly privileged attackers, to identify and support technical explanations of known but ambiguous operations, and to identify and analyze vulnerabilities in current systems that are favorable to Nation-State Attackers.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/143907/1/aaspring_1.pd