An analysis of malware evasion techniques against modern AV engines

This research empirically tested the response of antivirus applications to binaries that use virus-like evasion techniques. In order to achieve this, a number of binaries are processed using a number of evasion methods and are then deployed against several antivirus engines. The research also documents the process of setting up an environment for testing antivirus engines, including building the evasion techniques used in the tests. The results of the empirical tests illustrate that an attacker can evade multiple antivirus engines without much effort using well-known evasion techniques. Furthermore, some antivirus engines may respond to the occurrence of an evasion technique instead of the presence of any malicious code. In practical terms, this shows that while antivirus applications are useful for protecting against known threats, their effectiveness against unknown or modified threats is limited

BINARY INSTRUMENTATION AND TRANSFORMATION FOR SOFTWARE SECURITY APPLICATIONS

The capabilities of software analysis and manipulation are crucial to counter software security threats such as malware and vulnerabilities. Binary instrumentation and transformation are the essential techniques to enable software analysis and manipulation. However, existing approaches fail to meet requirements (e.g. flexibility, transparency) specific in software security applications

Buffer Overflow Vulnerability Diagnosis For Commodity Software

Buffer overflow attacks have been a computer security threat in software-based systems andapplications for decades. The existence of buffer overflow vulnerabilities makes the systemsusceptible to Internet worms and denial of service (DDoS) attacks which can cause hugesocial and financial impacts. Due to its importance, buffer overflow problem has been intensively studied. Researchershave proposed different techniques to defend against unknown buffer overflow attacks. Theyhave also investigated various solutions, including automatic signature generation, automatic patch generation, etc., to automatically protect computer systems with known vulnerabilities. The effectiveness and efficiency of the automatic signature generation approaches andthe automatic patch generation approaches are all based on the accurate understanding ofthe vulnerabilities, the buffer overflow vulnerability diagnosis (BOVD). Currently, the results of automatic signature generation and automatic patch generation are far from satisfaction due to the insufficient research results from the automatic BOVD. This thesis defines the automatic buffer overflow vulnerability diagnosis (BOVD) problemand provides solutions towards automatic BOVD for commodity software. It targets oncommodity software when source code and symbol table are not available. The solutionscombine both of the dynamic analysis techniques and static analysis techniques to achievethe goal. Based on the observation that buffer overflow attack happens when the size of the destination buffer is smaller than the total number of writes after the data copy process if the buffer overflow attack happens through a data copy procedure, the diagnosis results return the information of the size of destination buffer, the total number of writes of a data copy procedure and how the user inputs are related with them. They are achieved through bound analysis, loop analysis and input analysis respectively. We demonstrate the effectiveness of this thesis approach using real world vulnerable applications including the buffer overflow vulnerabilities attacked by the record-setting Slammer and Blaster worms. This thesis also does the complete case study for buffer overflow vulnerabilities which may have independent interests to researchers. Our buffer overflow case study results can help other researchers to design more effective defense systems and debugging tools against buffer overflow attacks

Adversarial Robustness of Hybrid Machine Learning Architecture for Malware Classification

The detection heuristic in contemporary machine learning Windows malware classifiers is typically based on the static properties of the sample. In contrast, simultaneous utilization of static and behavioral telemetry is vaguely explored. We propose a hybrid model that employs dynamic malware analysis techniques, contextual information as an executable filesystem path on the system, and static representations used in modern state-of-the-art detectors. It does not require an operating system virtualization platform. Instead, it relies on kernel emulation for dynamic analysis. Our model reports enhanced detection heuristic and identify malicious samples, even if none of the separate models express high confidence in categorizing the file as malevolent. For instance, given the $0.05\%$ false positive rate, individual static, dynamic, and contextual model detection rates are $18.04\%$, $37.20\%$, and $15.66\%$. However, we show that composite processing of all three achieves a detection rate of $96.54\%$, above the cumulative performance of individual components. Moreover, simultaneous use of distinct malware analysis techniques address independent unit weaknesses, minimizing false positives and increasing adversarial robustness. Our experiments show a decrease in contemporary adversarial attack evasion rates from $26.06\%$ to $0.35\%$ when behavioral and contextual representations of sample are employed in detection heuristic

Avaddon ransomware: an in-depth analysis and decryption of infected systems

The commoditization of Malware-as-a-Service (MaaS) allows criminals to obtain financial benefits at a low risk and with little technical background. One such popular product in the underground economy is ransomware. In ransomware attacks, data from infected systems is held hostage (encrypted) until a fee is paid to the criminals. This modus operandi disrupts legitimate businesses, which may become unavailable until the data is restored. A recent blackmailing strategy adopted by criminals is to leak data online from the infected systems if the ransom is not paid. Besides reputational damage, data leakage might produce further economical losses due to fines imposed by data protection laws. Thus, research on prevention and recovery measures to mitigate the impact of such attacks is needed to adapt existing countermeasures to new strains. In this work, we perform an in-depth analysis of Avaddon, a ransomware offered in the underground economy as an affiliate program business. This has infected and leaked data from at least 23 organizations. Additionally, it runs Distributed Denial-of-Service (DDoS) attacks against victims that do not pay the ransom. We first provide an analysis of the criminal business model from the underground economy. Then, we identify and describe its technical capabilities. We provide empirical evidence of links between this variant and a previous family, suggesting that the same group was behind the development and, possibly, the operation of both campaigns. Finally, we describe a method to decrypt files encrypted with Avaddon in real time. We implement and test the decryptor in a tool that can recover the encrypted data from an infected system, thus mitigating the damage caused by the ransomware. The tool is released open-source so it can be incorporated in existing Antivirus engines

Forensic identification and detection of hidden and obfuscated malware

The revolution in online criminal activities and malicious software (malware) has posed a serious challenge in malware forensics. Malicious attacks have become more organized and purposefully directed. With cybercrimes escalating to great heights in quantity as well as in sophistication and stealth, the main challenge is to detect hidden and obfuscated malware. Malware authors use a variety of obfuscation methods and specialized stealth techniques of information hiding to embed malicious code, to infect systems and to thwart any attempt to detect them, specifically with the use of commercially available anti-malware engines. This has led to the situation of zero-day attacks, where malware inflict systems even with existing security measures. The aim of this thesis is to address this situation by proposing a variety of novel digital forensic and data mining techniques to automatically detect hidden and obfuscated malware. Anti-malware engines use signature matching to detect malware where signatures are generated by human experts by disassembling the file and selecting pieces of unique code. Such signature based detection works effectively with known malware but performs poorly with hidden or unknown malware. Code obfuscation techniques, such as packers, polymorphism and metamorphism, are able to fool current detection techniques by modifying the parent code to produce offspring copies resulting in malware that has the same functionality, but with a different structure. These evasion techniques exploit the drawbacks of traditional malware detection methods, which take current malware structure and create a signature for detecting this malware in the future. However, obfuscation techniques aim to reduce vulnerability to any kind of static analysis to the determent of any reverse engineering process. Furthermore, malware can be hidden in file system slack space, inherent in NTFS file system based partitions, resulting in malware detection that even more difficult.Doctor of Philosoph

CACEE: Context Aware Concolic Execution Engine for Malware Analysis

An emerging pattern in malware is the use of public web services for command andcontrol (C&C) infrastructure. This new trend, combined with the short lifespan of malwarein the wild, makes extracting behaviors from malware in an automated fashion a difficultproblem. The Context-Aware Concolic Execution Engine (CACEE) is a tool designed torecreate the original execution context, forcing Windows 32-bit malware to execute theirpayloads as if they were still operational. CACEE monitors the flow of data as the payloadexecutes, and uses this information to synthesize the behaviors the malware exhibits. Threemalware case studies that abuse public web services are analyzed with CACEE, and theresults are compared against manual reverse engineering.M.S

SecureQEMU: Emulation-based Software Protection Providing Encrypted Code Execution and Page Granularity Code Signing

This research presents an original emulation-based software protection scheme providing protection from reverse code engineering (RCE) and software exploitation using encrypted code execution and page-granularity code signing, respectively. Protection mechanisms execute in trusted emulators while remaining out-of-band of untrusted systems being emulated. This protection scheme is called SecureQEMU and is based on a modified version of Quick Emulator (QEMU) [5]. RCE is a process that uncovers the internal workings of a program. It is used during vulnerability and intellectual property (IP) discovery. To protect from RCE program code may have anti-disassembly, anti-debugging, and obfuscation techniques incorporated. These techniques slow the process of RCE, however, once defeated protected code is still comprehensible. Encryption provides static code protection, but encrypted code must be decrypted before execution. SecureQEMUs\u27 scheme overcomes this limitation by keeping code encrypted during execution. Software exploitation is a process that leverages design and implementation errors to cause unintended behavior which may result in security policy violations. Traditional exploitation protection mechanisms provide a blacklist approach to software protection. Specially crafted exploit payloads bypass these protection mechanisms. SecureQEMU provides a whitelist approach to software protection by executing signed code exclusively. Unsigned malicious code (exploits, backdoors, rootkits, etc.) remain unexecuted, therefore, protecting the system. SecureQEMUs\u27 cache mechanisms increase performance by 0.9% to 1.8% relative to QEMU. Emulation overhead for SecureQEMU varies from 1400% to 2100% with respect to native performance. SecureQEMUs\u27 performance increase is negligible with respect to emulation overhead. Dependent on risk management strategy, SecureQEMU\u27s protection benefits may outweigh emulation overhead

Analysis and Defense of Emerging Malware Attacks

The persistent evolution of malware intrusion brings great challenges to current anti-malware industry. First, the traditional signature-based detection and prevention schemes produce outgrown signature databases for each end-host user and user has to install the AV tool and tolerate consuming huge amount of resources for pairwise matching. At the other side of malware analysis, the emerging malware can detect its running environment and determine whether it should infect the host or not. Hence, traditional dynamic malware analysis can no longer find the desired malicious logic if the targeted environment cannot be extracted in advance. Both these two problems uncover that current malware defense schemes are too passive and reactive to fulfill the task. The goal of this research is to develop new analysis and protection schemes for the emerging malware threats. Firstly, this dissertation performs a detailed study on recent targeted malware attacks. Based on the study, we develop a new technique to perform effectively and efficiently targeted malware analysis. Second, this dissertation studies a new trend of massive malware intrusion and proposes a new protection scheme to proactively defend malware attack. Lastly, our focus is new P2P malware. We propose a new scheme, which is named as informed active probing, for large-scale P2P malware analysis and detection. In further, our internet-wide evaluation shows our active probing scheme can successfully detect malicious P2P malware and its corresponding malicious servers