Application of information theory and statistical learning to anomaly detection

In today\u27s highly networked world, computer intrusions and other attacks area constant threat. The detection of such attacks, especially attacks that are new or previously unknown, is important to secure networks and computers. A major focus of current research efforts in this area is on anomaly detection.;In this dissertation, we explore applications of information theory and statistical learning to anomaly detection. Specifically, we look at two difficult detection problems in network and system security, (1) detecting covert channels, and (2) determining if a user is a human or bot. We link both of these problems to entropy, a measure of randomness information content, or complexity, a concept that is central to information theory. The behavior of bots is low in entropy when tasks are rigidly repeated or high in entropy when behavior is pseudo-random. In contrast, human behavior is complex and medium in entropy. Similarly, covert channels either create regularity, resulting in low entropy, or encode extra information, resulting in high entropy. Meanwhile, legitimate traffic is characterized by complex interdependencies and moderate entropy. In addition, we utilize statistical learning algorithms, Bayesian learning, neural networks, and maximum likelihood estimation, in both modeling and detecting of covert channels and bots.;Our results using entropy and statistical learning techniques are excellent. By using entropy to detect covert channels, we detected three different covert timing channels that were not detected by previous detection methods. Then, using entropy and Bayesian learning to detect chat bots, we detected 100% of chat bots with a false positive rate of only 0.05% in over 1400 hours of chat traces. Lastly, using neural networks and the idea of human observational proofs to detect game bots, we detected 99.8% of game bots with no false positives in 95 hours of traces. Our work shows that a combination of entropy measures and statistical learning algorithms is a powerful and highly effective tool for anomaly detection

Employing Entropy in the Detection and Monitoring of Network Covert Channels

The detection of covert channels has quickly become a vital need due to their pervasive nature and the increasing popularity of the Internet. In recent years, new and innovative methods have been proposed to aid in the detection of covert channels. Existing detection schemes are often too specific and are ineffective against new covert channels. In this paper, we expound upon previous work done with timing channels and apply it to detecting covert storage channels. Our approach is based on the assumption that the entropy of covert channels will vary from that of previously observed, legitimate, communications. This change in the entropy of a process provides us with a method for identifying storage channels. Using this assumption we created proof of concept code capable of detecting various covert storage channels. The results of our experiments demonstrate that we can successfully detect existing and unpublished covert storage channels accurately

SnapCatch: Automatic Detection of Covert Timing Channels Using Image Processing and Machine Learning

With the rapid growth of data exfiltration carried out by cyber attacks, Covert Timing Channels (CTC) have become an imminent network security risk that continues to grow in both sophistication and utilization. These types of channels utilize inter-arrival times to steal sensitive data from the targeted networks. CTC detection relies increasingly on machine learning techniques, which utilize statistical-based metrics to separate malicious (covert) traffic flows from the legitimate (overt) ones. However, given the efforts of cyber attacks to evade detection and the growing column of CTC, covert channels detection needs to improve in both performance and precision to detect and prevent CTCs and mitigate the reduction of the quality of service caused by the detection process. In this article, we present an innovative image-based solution for fully automated CTC detection and localization. Our approach is based on the observation that the covert channels generate traffic that can be converted to colored images. Leveraging this observation, our solution is designed to automatically detect and locate the malicious part (i.e., set of packets) within a traffic flow. By locating the covert parts within traffic flows, our approach reduces the drop of the quality of service caused by blocking the entire traffic flows in which covert channels are detected. We first convert traffic flows into colored images, and then we extract image-based features for detection covert traffic. We train a classifier using these features on a large data set of covert and overt traffic. This approach demonstrates a remarkable performance achieving a detection accuracy of 95.83% for cautious CTCs and a covert traffic accuracy of 97.83% for 8 bit covert messages, which is way beyond what the popular statistical-based solutions can achieve

Steganographic Timing Channels

This paper describes steganographic timing channels that use cryptographic primitives to hide the presence of covert channels in the timing of network traffic. We have identified two key properties for steganographic timing channels: (1) the parameters of the scheme should be cryptographically keyed, and (2) the distribution of input timings should be indistinguishable from output timings. These properties are necessary (although we make no claim they are sufficient) for the undetectability of a steganographic timing channel. Without them, the contents of the channel can be read and observed by unauthorized persons, and the presence of the channel is trivially exposed by noticing large changes in timing distributions – a previously proposed methodology for covert channel detection. Our steganographic timing scheme meets the secrecy requirement by employing cryptographic keys, and we achieve a restricted form of input/output distribution parity. Under certain distributions, our schemes conforms to a uniformness property; input timings that are uniformly distributed modulo a timing window are indistinguishable from output timings, measured under the same modulo. We also demonstrate that our scheme is practical under real network conditions, and finally present an empirical study of its covertness using the firstorder entropy metric, as suggested by Gianvecchio and Wang [8], which is currently the best published practical detection heuristic for timing channels