Using Memory Management to Detect and Extract Illegitimate Code for Malware Analysis

Exploits that successfully attack computers are mostly based on some form of shellcode, i.e., illegitimate code that is injected by the attacker to take control of the system. Detecting and extracting such code is the first step to detailed analysis of malware containing illegitimate code. The amount and sophistication of modern malware calls for automated mechanisms that perform such detection and extraction. In this paper we present a novel generic and fully automatic approach to detect the execution of illegitimate code and extract such code upon detection. The basic idea of the approach is to flag critical memory pages as non-executable and use a modified page fault handler to dump corresponding memory pages. We present an implementation of the approach for the Windows platform called CWXDetector. Evaluations using malicious PDF documents as example show that CWXDetector produces no false positives and has a similarly low false negative rate

Robust and secure monitoring and attribution of malicious behaviors

Worldwide computer systems continue to execute malicious software that degrades the systemsâ performance and consumes network capacity by generating high volumes of unwanted traffic. Network-based detectors can effectively identify machines participating in the ongoing attacks by monitoring the traffic to and from the systems. But, network detection alone is not enough; it does not improve the operation of the Internet or the health of other machines connected to the network. We must identify malicious code running on infected systems, participating in global attack networks. This dissertation describes a robust and secure approach that identifies malware present on infected systems based on its undesirable use of network. Our approach, using virtualization, attributes malicious traffic to host-level processes responsible for the traffic. The attribution identifies on-host processes, but malware instances often exhibit parasitic behaviors to subvert the execution of benign processes. We then augment the attribution software with a host-level monitor that detects parasitic behaviors occurring at the user- and kernel-level. User-level parasitic attack detection happens via the system-call interface because it is a non-bypassable interface for user-level processes. Due to the unavailability of one such interface inside the kernel for drivers, we create a new driver monitoring interface inside the kernel to detect parasitic attacks occurring through this interface. Our attribution software relies on a guest kernelâ s data to identify on-host processes. To allow secure attribution, we prevent illegal modifications of critical kernel data from kernel-level malware. Together, our contributions produce a unified research outcome --an improved malicious code identification system for user- and kernel-level malware.Ph.D.Committee Chair: Giffin, Jonathon; Committee Member: Ahamad, Mustaque; Committee Member: Blough, Douglas; Committee Member: Lee, Wenke; Committee Member: Traynor, Patric

Malicious cryptography techniques for unreversable (malicious or not) binaries

Fighting against computer malware require a mandatory step of reverse engineering. As soon as the code has been disassemblied/decompiled (including a dynamic analysis step), there is a hope to understand what the malware actually does and to implement a detection mean. This also applies to protection of software whenever one wishes to analyze them. In this paper, we show how to amour code in such a way that reserse engineering techniques (static and dymanic) are absolutely impossible by combining malicious cryptography techniques developped in our laboratory and new types of programming (k-ary codes). Suitable encryption algorithms combined with new cryptanalytic approaches to ease the protection of (malicious or not) binaries, enable to provide both total code armouring and large scale polymorphic features at the same time. A simple 400 Kb of executable code enables to produce a binary code and around $2^{140}$ mutated forms natively while going far beyond the old concept of decryptor.Comment: 17 pages, 2 figures, accepted for presentation at H2HC'1

Towards Adversarial Malware Detection: Lessons Learned from PDF-based Attacks

Malware still constitutes a major threat in the cybersecurity landscape, also due to the widespread use of infection vectors such as documents. These infection vectors hide embedded malicious code to the victim users, facilitating the use of social engineering techniques to infect their machines. Research showed that machine-learning algorithms provide effective detection mechanisms against such threats, but the existence of an arms race in adversarial settings has recently challenged such systems. In this work, we focus on malware embedded in PDF files as a representative case of such an arms race. We start by providing a comprehensive taxonomy of the different approaches used to generate PDF malware, and of the corresponding learning-based detection systems. We then categorize threats specifically targeted against learning-based PDF malware detectors, using a well-established framework in the field of adversarial machine learning. This framework allows us to categorize known vulnerabilities of learning-based PDF malware detectors and to identify novel attacks that may threaten such systems, along with the potential defense mechanisms that can mitigate the impact of such threats. We conclude the paper by discussing how such findings highlight promising research directions towards tackling the more general challenge of designing robust malware detectors in adversarial settings

Pre-filters in-transit malware packets detection in the network

Conventional malware detection systems cannot detect most of the new malware in the network without the availability of their signatures. In order to solve this problem, this paper proposes a technique to detect both metamorphic (mutated malware) and general (non-mutated) malware in the network using a combination of known malware sub-signature and machine learning classification. This network-based malware detection is achieved through a middle path for efficient processing of non-malware packets. The proposed technique has been tested and verified using multiple data sets (metamorphic malware, non-mutated malware, and UTM real traffic), this technique can detect most of malware packets in the network-based before they reached the host better than the previous works which detect malware in host-based. Experimental results showed that the proposed technique can speed up the transmission of more than 98% normal packets without sending them to the slow path, and more than 97% of malware packets are detected and dropped in the middle path. Furthermore, more than 75% of metamorphic malware packets in the test dataset could be detected. The proposed technique is 37 times faster than existing technique

Preventing the release of illegitimate applications on mobile markets

The popularity of mobile applications has been growing worldwide over the last few decades. This popularity is attracting more and more authors of malicious applications called malwares. To detect those malwares, mobile markets have implemented analysis methods that suffer from several limitations. Those we have identified and which we propose to solve in the scope of this thesis are mainly two . The first is the inability to cope with a new method of malware publication consisting in anticipating the mobile version of a company that does not yet have one. The second limitation is the difficulty, due to app tracing, encountered by dynamic analysis solutions to be able to scale. To solve the first limitation we designed and implemented a security check system called IMAD (Illegitimate Mobile App Detector), which is based mainly on online search engines and machine learning techniques. To solve the second problem, we introduced a scalable tracing approach, that we call delegated instrumentation. It leverages Android's instrumentation module and mainly relies on ART (Android RunTime) reverse engineering and hacking. The evaluation results show that IMAD can protect companies from anticipation attacks with an acceptable error rate and at a low cost for MMPs. And we demonstrated the effectiveness of the delegated instrumentation with a prototype named ODILE that traces various app types (including benign apps and malwares) on Samsung Galaxy A7 2017. In particular, we show how much ODILE outperforms Frida, the state-of-the-art tool in the domain

Twitter Malware Collection System: An Automated URL Extraction and Examination Platform

As the world becomes more interconnected through various technological services and methods, the threat of malware is increasingly looming overhead. One avenue in particular that is examined in this research is the social networking service Twitter. This research develops the Twitter Malware Collection System (TMCS). This system gathers Uniform Resource Locators (URLs) posted on Twitter and scans them to determine if any are hosting malware. This scanning process is performed by a cluster of Virtual Machines (VMs) running a specified software configuration and the execution prevention system known as ESCAPE, which detects malicious code. When a URL is detected by a TMCS VM instance to be hosting malware, a dump of the web browser is created to determine what kind of malicious activity has taken place and also how this activity was allowed. After collecting over a period of 40 days, and processing a total of 466,237 URLs twice in two different configurations, one consisting of a vulnerable Windows XP SP2 setup and the other consisting of a fully patched and updated Windows Vista setup, a total of 2,989 dumps were created by TMCS based on the results generated by ESCAPE

Applying Machine Learning to Advance Cyber Security: Network Based Intrusion Detection Systems

Many new devices, such as phones and tablets as well as traditional computer systems, rely on wireless connections to the Internet and are susceptible to attacks. Two important types of attacks are the use of malware and exploiting Internet protocol vulnerabilities in devices and network systems. These attacks form a threat on many levels and therefore any approach to dealing with these nefarious attacks will take several methods to counter. In this research, we utilize machine learning to detect and classify malware, visualize, detect and classify worms, as well as detect deauthentication attacks, a form of Denial of Service (DoS). This work also includes two prevention mechanisms for DoS attacks, namely a one- time password (OTP) and through the use of machine learning. Furthermore, we focus on an exploit of the widely used IEEE 802.11 protocol for wireless local area networks (WLANs). The work proposed here presents a threefold approach for intrusion detection to remedy the effects of malware and an Internet protocol exploit employing machine learning as a primary tool. We conclude with a comparison of dimensionality reduction methods to a deep learning classifier to demonstrate the effectiveness of these methods without compromising the accuracy of classification