Fast and Accurate Machine Learning-based Malware Detection via RC4 Ciphertext Analysis

Malware is dramatically increasing its viability while hiding its malicious intent and/or behavior by employing ciphers. So far, many efforts have been made to detect malware and prevent it from damaging users by monitoring network packets. However, conventional detection schemes analyzing network packets directly are hardly applicable to detect the advanced malware that encrypts the communication. Cryptoanalysis of each packet flowing over a network might be one feasible solution for the problem. However, the approach is computationally expensive and lacks accuracy, which is consequently not a practical solution. To tackle these problems, in this paper, we propose novel schemes that can accurately detect malware packets encrypted by RC4 without decryption in a timely manner. First, we discovered that a fixed encryption key generates unique statistical patterns on RC4 ciphertexts. Then, we detect malware packets of RC4 ciphertexts efficiently and accurately by utilizing the discovered statistical patterns of RC4 ciphertext given encryption key. Our proposed schemes directly analyze network packets without decrypting ciphertexts. Moreover, our analysis can be effectively executed with only a very small subset of the network packet. To the best of our knowledge, the unique signature has never been discussed in any previous research. Our intensive experimental results with both simulation data and actual malware show that our proposed schemes are extremely fast (23.06±1.52 milliseconds) and highly accurate (100%) on detecting a DarkComet malware with only a network packet of 36 bytes

Multi-Stage Detection Technique for DNS-Based Botnets

Domain Name System (DNS) is one of the most widely used protocols in the Internet. The main purpose of the DNS protocol is mapping user-friendly domain names to IP addresses. Unfortunately, many cyber criminals deploy the DNS protocol for malicious purposes, such as botnet communications. In this type of attack, the botmasters tunnel communications between the Command and Control (C&C) servers and the bot-infected machines within DNS request and response. Designing an effective approach for botnet detection has been done previously based on specific botnet types Since botnet communications are characterized by different features, botmasters may evade detection methods by modifying some of these features. This research aims to design and implement a multi-staged detection approach for Domain Generation Algorithm (DGA), Fast Flux Service Network, and Domain Flux-based botnets, as well as encrypted DNS tunneled-based botnets using the BRO Network Security Monitor. This approach is able to detect DNS-based botnet communications by relying on analyzing different techniques used for finding the C&C server, as well as encrypting the malicious traffic

Detecting encrypted botnet traffic

Bot detection methods that rely on deep packet inspection (DPI) can be foiled by encryption. Encryption, however, increases entropy. This paper investigates whether adding high-entropy detectors to an existing bot detection tool that uses DPI can restore some of the bot visibility. We present two high-entropy classifiers, and use one of them to enhance BotHunter. Our results show that while BotHunter misses about 50% of the bots when they employ encryption, our high-entropy classifier restores most of its ability to detect bots, even when they use encryption. © 2013 IEEE

Detecting encrypted botnet traffic

Bot detection methods that rely on deep packet inspection (DPI) can be foiled by encryption. Encryption, however, increases entropy. This paper investigates whether adding high-entropy detectors to an existing bot detection tool that uses DPI can restore some of the bot visibility.We present two high-entropy classifiers, and use one of them to enhance BotHunter. Our results show that while BotHunter misses about 50% of the bots when they employ encryption, our high-entropy classifier restores most of its ability to detect bots, even when they use encryption. © 2013 IEEE