Towards the Automated Detection of Unknown Malware on Live Systems

Abstract—In this paper, we propose a new system monitoring framework that can serve as an enabler for automated malware detection on live systems. Our approach takes advantage of the increased availability of hardware assisted virtualization capabilities of modern CPUs, and its basic novelty consists in launching a hypervisor layer on the live system without stopping and restarting it. This hypervisor runs at a higher privilege level than the OS itself, thus, it can be used to observe the behavior of the analyzed system in a transparent manner. For this purpose, we also propose a novel system call tracing method that is designed to be configurable in terms of transparency and granularity. I

Intelligent OS X malware threat detection with code inspection

With the increasing market share of Mac OS X operating system, there is a corresponding increase in the number of malicious programs (malware) designed to exploit vulnerabilities on Mac OS X platforms. However, existing manual and heuristic OS X malware detection techniques are not capable of coping with such a high rate of malware. While machine learning techniques offer promising results in automated detection of Windows and Android malware, there have been limited efforts in extending them to OS X malware detection. In this paper, we propose a supervised machine learning model. The model applies kernel base Support Vector Machine (SVM) and a novel weighting measure based on application library calls to detect OS X malware. For training and evaluating the model, a dataset with a combination of 152 malware and 450 benign were is created. Using common supervised Machine Learning algorithm on the dataset, we obtain over 91% detection accuracy with 3.9% false alarm rate. We also utilize Synthetic Minority Over-sampling Technique (SMOTE) to create three synthetic datasets with different distributions based on the refined version of collected dataset to investigate impact of different sample sizes on accuracy of malware detection. Using SMOTE datasets we could achieve over 96% detection accuracy and false alarm of less than 4%. All malware classification experiments are tested using cross validation technique. Our results reflect that increasing sample size in synthetic datasets has direct positive effect on detection accuracy while increases false alarm rate in compare to the original dataset

Prometheus: Analyzing WebInject-based information stealers

Nowadays Information stealers are reaching high levels of sophistication. The number of families and variants observed increased exponentially in the last years. Furthermore, these trojans are sold on underground markets along with automatic frameworks that include web-based administration panels, builders and customization procedures. From a technical point of view such malware is equipped with a functionality, called WebInject, that exploits API hooking techniques to intercept all sensitive data in a browser context and modify web pages on infected hosts. In this paper we propose Prometheus, an automatic system that is able to analyze trojans that base their attack technique on DOM modifications. Prometheus is able to identify the injection operations performed by malware, and generate signatures based on the injection behavior. Furthermore, it is able to extract the WebInject targets by using memory forensic techniques. We evaluated Prometheus against real-world, online websites and a dataset of distinct variants of financial trojans. In our experiments we show that our approach correctly recognizes known variants of WebInject-based malware and successfully extracts the WebInject targets

Investigations into Decrypting Live Secure Traffic in Virtual Environments

Malicious agents increasingly use encrypted tunnels to communicate with external servers. Communications may contain ransomware keys, stolen banking details, or other confidential information. Rapid discovery of communicated contents through decrypting tunnelled traffic can support effective means of dealing with these malicious activities.Decrypting communications requires knowledge of cryptographic algorithms and artefacts, such as encryption keys and initialisation vectors. Such artefacts may exist in volatile memory when software applications encrypt. Virtualisation technologies can enable the acquisition of virtual machine memory to support the discovery of these cryptographic artefacts.A framework is constructed to investigate the decryption of potentially malicious communications using novel approaches to identify candidate initialisation vectors, and use these to discover candidate keys. The framework focuses on communications that use the Secure Shell and Transport Layer Security protocols in virtualised environments for different operating systems, protocols, encryption algorithms, and software implementations. The framework minimises virtual machine impact, and functions at an elevated level to make detection by virtual machine software difficult.The framework analyses Windows and Linux memory and validates decrypts for both protocols when the Advanced Encryption Standard symmetric block or ChaCha20 symmetric stream algorithms are used for encryption. It also investigates communications originating from malware clients, such as bot and ransomware, that use Windows cryptographic libraries.The framework correctly decrypted tunnelled traffic with near certainty in almost all experiments. The analysis durations ranged from sub-second to less than a minute, demonstrating that decryption of malicious activity before network session completion is possible. This can enable in-line detection of unknown malicious agents, timely discovery of ransomware keys, and knowledge of exfiltrated confidential information

xOSSig: Leveraging OS Diversity to Automatically Extract Malware Code Signatures

We present an automated approach to extract code signatures that serve as the forensic fingerprint of a given malware program. Our high-level idea is to compare the memory contents of a sandbox before and after infection by a malware. To pinpoint the actual memory changes caused by the malware, and ignore all others, we use a novel concept called Cross OS Execution. That is, we execute a malware program on multiple different but compatible operating systems (OSes) to identify its memory commonalities, while neglecting OS-specific noise. The commonalities of the dumps therefore contain patterns whose presence is the consequence of executing the malware, i.e., the forensic fingerprint of the malware. We show that we can use two different versions Windows to accurately extract fingerprints of all 17 popular Windows malware families in our test set. These signatures serve to re-identify malware infections in memory dumps with a TPR of 93% and an FPR of 0.15%

Exploring Host-based Software Defined Networking and its Applications

Network operators need detailed understanding of their networks in order to ensure functionality and to mitigate security risks. Unfortunately, legacy networks are poorly suited to providing this understanding. While the software-defined networking paradigm has the potential to, existing switch-based implementations are unable to scale sufficiently to provide information in a fine-grained. Furthermore, as switches are inherently blind to the inner workings of hosts, significantly hindering an operator\u27s ability to understand the true context behind network traffic. In this work, we explore a host-based software-defined networking implementation. We evaluation our implementation, showing that it is able to scale beyond the capabilities of a switch-based implementation. Furthermore, we discuss various detailed network policies that network operators can write and enforce which are impossible in a switch-based implementation. We also implement and discuss an anti-reconnaissance system that can be deployed without any additional components

MALPITY: Automatic Identification and Exploitation of Tarpit Vulnerabilities in Malware

Law enforcement agencies regularly take down botnets as the ultimate defense against global malware operations. By arresting malware authors, and simultaneously infiltrating or shutting down a botnet’s network infrastructures (such as C2 servers), defenders stop global threats and mitigate pending infections. In this paper, we propose malware tarpits, an orthogonal defense that does not require seizing botnet infrastructures, and at the same time can also be used to slow down malware spreading and infiltrate its monetization techniques. A tarpit is a network service that causes a client to stay busy with a network operation. Our work aims to automatically identify network operations used by malware that will block the malware either forever or for a significant amount of time. We describe how to non-intrusively exploit such tarpit vulnerabilities in malware to slow down or, ideally, even stop malware. Using dynamic malware analysis, we monitor how malware interacts with the POSIX and Winsock socket APIs. From this, we infer network operations that would have blocked when provided certain network inputs. We augment this vulnerability search with an automated generation of tarpits that exploit the identified vulnerabilities. We apply our prototype MALPITY on six popular malware families and discover 12 previously-unknown tarpit vulnerabilities, revealing that all families are susceptible to our defense. We demonstrate how to, e.g., halt Pushdo’s DGA-based C2 communication, hinder SalityP2P peers from receiving commands or updates, and stop Bashlite’s spreading engine

Advanced Threat Intelligence: Interpretation of Anomalous Behavior in Ubiquitous Kernel Processes

Targeted attacks on digital infrastructures are a rising threat against the confidentiality, integrity, and availability of both IT systems and sensitive data. With the emergence of advanced persistent threats (APTs), identifying and understanding such attacks has become an increasingly difficult task. Current signature-based systems are heavily reliant on fixed patterns that struggle with unknown or evasive applications, while behavior-based solutions usually leave most of the interpretative work to a human analyst. This thesis presents a multi-stage system able to detect and classify anomalous behavior within a user session by observing and analyzing ubiquitous kernel processes. Application candidates suitable for monitoring are initially selected through an adapted sentiment mining process using a score based on the log likelihood ratio (LLR). For transparent anomaly detection within a corpus of associated events, the author utilizes star structures, a bipartite representation designed to approximate the edit distance between graphs. Templates describing nominal behavior are generated automatically and are used for the computation of both an anomaly score and a report containing all deviating events. The extracted anomalies are classified using the Random Forest (RF) and Support Vector Machine (SVM) algorithms. Ultimately, the newly labeled patterns are mapped to a dedicated APT attacker–defender model that considers objectives, actions, actors, as well as assets, thereby bridging the gap between attack indicators and detailed threat semantics. This enables both risk assessment and decision support for mitigating targeted attacks. Results show that the prototype system is capable of identifying 99.8% of all star structure anomalies as benign or malicious. In multi-class scenarios that seek to associate each anomaly with a distinct attack pattern belonging to a particular APT stage we achieve a solid accuracy of 95.7%. Furthermore, we demonstrate that 88.3% of observed attacks could be identified by analyzing and classifying a single ubiquitous Windows process for a mere 10 seconds, thereby eliminating the necessity to monitor each and every (unknown) application running on a system. With its semantic take on threat detection and classification, the proposed system offers a formal as well as technical solution to an information security challenge of great significance.The financial support by the Christian Doppler Research Association, the Austrian Federal Ministry for Digital and Economic Affairs, and the National Foundation for Research, Technology and Development is gratefully acknowledged