Fast and Accurate Machine Learning-based Malware Detection via RC4 Ciphertext Analysis

Malware is dramatically increasing its viability while hiding its malicious intent and/or behavior by employing ciphers. So far, many efforts have been made to detect malware and prevent it from damaging users by monitoring network packets. However, conventional detection schemes analyzing network packets directly are hardly applicable to detect the advanced malware that encrypts the communication. Cryptoanalysis of each packet flowing over a network might be one feasible solution for the problem. However, the approach is computationally expensive and lacks accuracy, which is consequently not a practical solution. To tackle these problems, in this paper, we propose novel schemes that can accurately detect malware packets encrypted by RC4 without decryption in a timely manner. First, we discovered that a fixed encryption key generates unique statistical patterns on RC4 ciphertexts. Then, we detect malware packets of RC4 ciphertexts efficiently and accurately by utilizing the discovered statistical patterns of RC4 ciphertext given encryption key. Our proposed schemes directly analyze network packets without decrypting ciphertexts. Moreover, our analysis can be effectively executed with only a very small subset of the network packet. To the best of our knowledge, the unique signature has never been discussed in any previous research. Our intensive experimental results with both simulation data and actual malware show that our proposed schemes are extremely fast (23.06±1.52 milliseconds) and highly accurate (100%) on detecting a DarkComet malware with only a network packet of 36 bytes

Limiting DNS covert channels and network validated DNS

Despite the variety and number of network security devices and policies available, sensitive data, such as intellectual property and business data, can still be surreptitiously sent via the Internet to unscrupulous receivers. Furthermore, few security mechanisms address securing or limiting covert channels. This study defines a framework for determining a rule set to minimize covert channel capacity on the DNS protocol specifically. The information and techniques used in this study may be useful in aiding security professionals and developers with enforcing security policies on DNS and other Internet protocols.;This research resulted in the development of a rudimentary tool, referred to as NV-DNS, capable of detecting and effectively limiting the capability of covert channels in DNS communication packets

A Comprehensive Review of Video Steganalysis

Steganography is the art of secret communication and steganalysis is the art of detecting the hidden messages embedded in digital media covers. One of the covers that is gaining interest in the field is video. Presently, the global IP video traffic forms the major part of all consumer Internet traffic. It is also gaining attention in the field of digital forensics and homeland security in which threats of covert communications hold serious consequences. Thus, steganography technicians will prefer video to other types of covers like audio files, still images or texts. Moreover, video steganography will be of more interest because it provides more concealing capacity. Contrariwise, investigation in video steganalysis methods does not seem to follow the momentum even if law enforcement agencies and governments around the world support and encourage investigation in this field. In this paper, we review the most important methods used so far in video steganalysis and sketch the future trends. To the best of our knowledge this is the most comprehensive review of video steganalysis produced so far

Discovering New Vulnerabilities in Computer Systems

Vulnerability research plays a key role in preventing and defending against malicious computer system exploitations. Driven by a multi-billion dollar underground economy, cyber criminals today tirelessly launch malicious exploitations, threatening every aspect of daily computing. to effectively protect computer systems from devastation, it is imperative to discover and mitigate vulnerabilities before they fall into the offensive parties\u27 hands. This dissertation is dedicated to the research and discovery of new design and deployment vulnerabilities in three very different types of computer systems.;The first vulnerability is found in the automatic malicious binary (malware) detection system. Binary analysis, a central piece of technology for malware detection, are divided into two classes, static analysis and dynamic analysis. State-of-the-art detection systems employ both classes of analyses to complement each other\u27s strengths and weaknesses for improved detection results. However, we found that the commonly seen design patterns may suffer from evasion attacks. We demonstrate attacks on the vulnerabilities by designing and implementing a novel binary obfuscation technique.;The second vulnerability is located in the design of server system power management. Technological advancements have improved server system power efficiency and facilitated energy proportional computing. However, the change of power profile makes the power consumption subjected to unaudited influences of remote parties, leaving the server systems vulnerable to energy-targeted malicious exploit. We demonstrate an energy abusing attack on a standalone open Web server, measure the extent of the damage, and present a preliminary defense strategy.;The third vulnerability is discovered in the application of server virtualization technologies. Server virtualization greatly benefits today\u27s data centers and brings pervasive cloud computing a step closer to the general public. However, the practice of physical co-hosting virtual machines with different security privileges risks introducing covert channels that seriously threaten the information security in the cloud. We study the construction of high-bandwidth covert channels via the memory sub-system, and show a practical exploit of cross-virtual-machine covert channels on virtualized x86 platforms

Ensemble Methods for Anomaly Detection

Anomaly detection has many applications in numerous areas such as intrusion detection, fraud detection, and medical diagnosis. Most current techniques are specialized for detecting one type of anomaly, and work well on specific domains and when the data satisfies specific assumptions. We address this problem, proposing ensemble anomaly detection techniques that perform well in many applications, with four major contributions: using bootstrapping to better detect anomalies on multiple subsamples, sequential application of diverse detection algorithms, a novel adaptive sampling and learning algorithm in which the anomalies are iteratively examined, and improving the random forest algorithms for detecting anomalies in streaming data. We design and evaluate multiple ensemble strategies using score normalization, rank aggregation and majority voting, to combine the results from six well-known base algorithms. We propose a bootstrapping algorithm in which anomalies are evaluated from multiple subsets of the data. Results show that our independent ensemble performs better than the base algorithms, and using bootstrapping achieves competitive quality and faster runtime compared with existing works. We develop new sequential ensemble algorithms in which the second algorithm performs anomaly detection based on the first algorithm\u27s outputs; best results are obtained by combining algorithms that are substantially different. We propose a novel adaptive sampling algorithm which uses the score output of the base algorithm to determine the hard-to-detect examples, and iteratively resamples more points from such examples in a complete unsupervised context. On streaming datasets, we analyze the impact of parameters used in random trees, and propose new algorithms that work well with high-dimensional data, improving performance without increasing the number of trees or their heights. We show that further improvements can be obtained with an Evolutionary Algorithm

Use of Cryptography in Malware Obfuscation

Malware authors often use cryptographic tools such as XOR encryption and block ciphers like AES to obfuscate part of the malware to evade detection. Use of cryptography may give the impression that these obfuscation techniques have some provable guarantees of success. In this paper, we take a closer look at the use of cryptographic tools to obfuscate malware. We first find that most techniques are easy to defeat (in principle), since the decryption algorithm and the key is shipped within the program. In order to clearly define an obfuscation technique's potential to evade detection we propose a principled definition of malware obfuscation, and then categorize instances of malware obfuscation that use cryptographic tools into those which evade detection and those which are detectable. We find that schemes that are hard to de-obfuscate necessarily rely on a construct based on environmental keying. We also show that cryptographic notions of obfuscation, e.g., indistinghuishability and virtual black box obfuscation, may not guarantee evasion detection under our model. However, they can be used in conjunction with environmental keying to produce hard to de-obfuscate versions of programs

An examination of the Asus WL-HDD 2.5 as a nepenthes malware collector

The Linksys WRT54g has been used as a host for network forensics tools for instance Snort for a long period of time. Whilst large corporations are already utilising network forensic tools, this paper demonstrates that it is quite feasible for a non-security specialist to track and capture malicious network traffic. This paper introduces the Asus Wireless Hard disk as a replacement for the popular Linksys WRT54g. Firstly, the Linksys router will be introduced detailing some of the research that was undertaken on the device over the years amongst the security community. It then briefly discusses malicious software and the impact this may have for a home user. The paper then outlines the trivial steps in setting up Nepenthes 0.1.7 (a malware collector) for the Asus WL-HDD 2.5 according to the Nepenthes and tests the feasibility of running the malware collector on the selected device. The paper then concludes on discussing the limitations of the device when attempting to execute Nepenthes