Application of a Layered Hidden Markov Model in the Detection of Network Attacks

Network-based attacks against computer systems are a common and increasing problem. Attackers continue to increase the sophistication and complexity of their attacks with the goal of removing sensitive data or disrupting operations. Attack detection technology works very well for the detection of known attacks using a signature-based intrusion detection system. However, attackers can utilize attacks that are undetectable to those signature-based systems whether they are truly new attacks or modified versions of known attacks. Anomaly-based intrusion detection systems approach the problem of attack detection by detecting when traffic differs from a learned baseline. In the case of this research, the focus was on a relatively new area known as payload anomaly detection. In payload anomaly detection, the system focuses exclusively on the payload of packets and learns the normal contents of those payloads. When a payload\u27s contents differ from the norm, an anomaly is detected and may be a potential attack. A risk with anomaly-based detection mechanisms is they suffer from high false positive rates which reduce their effectiveness. This research built upon previous research in payload anomaly detection by combining multiple techniques of detection in a layered approach. The layers of the system included a high-level navigation layer, a request payload analysis layer, and a request-response analysis layer. The system was tested using the test data provided by some earlier payload anomaly detection systems as well as new data sets. The results of the experiments showed that by combining these layers of detection into a single system, there were higher detection rates and lower false positive rates

Network intrusion detection based on LDA for payload feature selection

Anomaly Intrusion Detection System (IDS) is a statistical based network IDS which can detect attack variants and novel attacks without a priori knowledge. Current anomaly IDSs are inefficient for real-time detection because of their complex computation. This paper proposes a novel approach to reduce the heavy computational cost of an anomaly IDS. Linear Discriminant Analysis (LDA) and difference distance map are used for selection of significant features. This approach is able to transform high-dimensional feature vectors into a low-dimensional domain. The similarity between new incoming packets and a normal profile is determined using Euclidean distance on the simple, low-dimensional feature domain. The final decision will be made according to a pre-calculated threshold to differentiate normal and abnormal network packets. The proposed approach is evaluated using DARPA 1999 IDS dataset. ©2010 IEEE

Unsupervised Learning to Improve Anomaly Detection

An intrusion detection system (IDS) is used to determine when a computer or computer network is under attack. Most contemporary IDSs operate by defining what an intrusion looks like and checking traffic for matching patterns in network traffic. This approach has unavoidable limitations including the inability to detect novel attacks and the maintenance of a rule bank that must grow with every new intrusion discovered. An anomaly detection scheme attempts to define what is normal so that abnormal traffic can be distinguished from it. This thesis explores the ways that an unsupervised technique called clustering can be used to distinguish normal traffic from anomalous traffic. This thesis will also explore an attempt to improve upon existing clustering algorithms to improve anomaly detection by adding in limited amounts of a posteriori knowledge

Securing Enterprise Networks with Statistical Node Behavior Profiling

The substantial proliferation of the Internet has made it the most critical infrastructure in today\u27s world. However, it is still vulnerable to various kinds of attacks/malwares and poses a number of great security challenges. Furthermore, we have also witnessed in the past decade that there is always a fast self-evolution of attacks/malwares (e.g. from worms to botnets) against every success in network security. Network security thereby remains a hot topic in both research and industry and requires both continuous and great attention. In this research, we consider two fundamental areas in network security, malware detection and background traffic modeling, from a new view point of node behavior profiling under enterprise network environments. Our main objectives are to extend and enhance the current research in these two areas. In particular, central to our research is the node behavior profiling approach that groups the behaviors of different nodes by jointly considering time and spatial correlations. We also present an extensive study on botnets, which are believed to be the largest threat to the Internet. To better understand the botnet, we propose a botnet framework and predict a new P2P botnet that is much stronger and stealthier than the current ones. We then propose anomaly malware detection approaches based directly on the insights (statistical characteristics) from the node behavior study and apply them on P2P botnet detection. Further, by considering the worst case attack model where the botmaster knows all the parameter values used in detection, we propose a fast and optimized anomaly detection approach by formulating the detection problem as an optimization problem. In addition, we propose a novel traffic modeling structure using behavior profiles for NIDS evaluations. It is efficient and takes into account the node heterogeneity in traffic modeling. It is also compatible with most current modeling schemes and helpful in generating better realistic background traffic. Last but not least, we evaluate the proposed approaches using real user trace from enterprise networks and achieve encouraging results. Our contributions in this research include: 1) a new node behavior profiling approach to study the normal node behavior; 2) a framework for botnets; 3) a new P2P botnet and performance comparisons with other P2P botnets; 4) two anomaly detection approaches based on node behavior profiles; 4) a fast and optimized anomaly detection approach under the worst case attack model; 5) a new traffic modeling structure and 6) simulations and evaluations of the above approaches under real user data from enterprise networks. To the best of our knowledge, we are the first to propose the botnet framework, consider the worst case attack model and propose corresponding fast and optimized solution in botnet related research. We are also the first to propose efficient solutions in traffic modeling without the assumption of node homogeneity

A Hybrid Classification Framework for Network Intrusion Detection with High Accuracy and Low Latency

Network intrusion detection (NIDS) is a crucial task aimed at safeguarding computer networks against malicious attacks. Traditional NIDS methods can be categorized as either misuse-based or anomaly-based, each having its unique set of limitations. Misuse-based approaches excel in identifying known attacks but fall short when dealing with new or unidentified attack patterns. On the other hand, anomaly-based methods are more adept at identifying novel attacks but tend to produce a substantial number of false positives. To enhance the overall performance of NIDS systems, hybrid classification techniques are employed, leveraging the strengths of both misuse-based and anomaly-based methods. In this research, we present a novel hybrid classification approach for NIDS that excels in both speed and accuracy. Our approach integrates a blend of machine learning algorithms, including decision trees, support vector machines, and deep neural networks. We conducted comprehensive evaluations of our approach using various network intrusion datasets, achieving state-of-the-art results in terms of accuracy and prediction speed

Anomaly Intrusion Detection based on Concept Drift

Nowadays, security on the internet is a vital issue and therefore, intrusion detection is one of the major research problems for networks that defend external attacks. Intrusion detection is a new approach for providing security in existing computers and data networks. An Intrusion Detection System is a software application that monitors the system for malicious activities and unauthorized access to the system. An easy accessibility condition causes computer networks vulnerable against the attack and several threats from attackers. Intrusion Detection System is used to analyze a network of interconnected systems for avoiding uncommon intrusion or chaos. The intrusion detection problem is becoming a challenging task due to the increase in computer networks since the increased connectivity of computer systems gives access to all and makes it easier for hackers to avoid their traces and identification. The goal of intrusion detection is to identify unauthorized use, misuse and abuse of computer systems. This project focuses on algorithms: (i) Concept Drift based ensemble Incremental Learning approach for anomaly intrusion detection, and (ii) Diversity and Transfer-based Ensemble Learning. These are highly ranked anomaly detection models. We study and compare both learning models. The Network Security Laboratory-Knowledge Discovery and Data Mining (NSL-KDD99) dataset have been used for training and to detect the misuse activities

Leveraging siamese networks for one-shot intrusion detection model

The use of supervised Machine Learning (ML) to enhance Intrusion Detection Systems (IDS) has been the subject of significant research. Supervised ML is based upon learning by example, demanding significant volumes of representative instances for effective training and the need to retrain the model for every unseen cyber-attack class. However, retraining the models in-situ renders the network susceptible to attacks owing to the time-window required to acquire a sufficient volume of data. Although anomaly detection systems provide a coarse-grained defence against unseen attacks, these approaches are significantly less accurate and suffer from high false-positive rates. Here, a complementary approach referred to as “One-Shot Learning”, whereby a limited number of examples of a new attack-class is used to identify a new attack-class (out of many) is detailed. The model grants a new cyber-attack classification opportunity for classes that were not seen during training without retraining. A Siamese Network is trained to differentiate between classes based on pairs similarities, rather than features, allowing to identify new and previously unseen attacks. The performance of a pre-trained model to classify new attack-classes based only on one example is evaluated using three mainstream IDS datasets; CICIDS2017, NSL-KDD, and KDD Cup’99. The results confirm the adaptability of the model in classifying unseen attacks and the trade-off between performance and the need for distinctive class representations.</p

PUE attack detection in CWSNs using anomaly detection techniques

Cognitive wireless sensor network (CWSN) is a new paradigm, integrating cognitive features in traditional wireless sensor networks (WSNs) to mitigate important problems such as spectrum occupancy. Security in cognitive wireless sensor networks is an important problem since these kinds of networks manage critical applications and data. The specific constraints of WSN make the problem even more critical, and effective solutions have not yet been implemented. Primary user emulation (PUE) attack is the most studied specific attack deriving from new cognitive features. This work discusses a new approach, based on anomaly behavior detection and collaboration, to detect the primary user emulation attack in CWSN scenarios. Two non-parametric algorithms, suitable for low-resource networks like CWSNs, have been used in this work: the cumulative sum and data clustering algorithms. The comparison is based on some characteristics such as detection delay, learning time, scalability, resources, and scenario dependency. The algorithms have been tested using a cognitive simulator that provides important results in this area. Both algorithms have shown to be valid in order to detect PUE attacks, reaching a detection rate of 99% and less than 1% of false positives using collaboration

PUE attack detection in CWSN using collaboration and learning behavior

Cognitive Wireless Sensor Network (CWSN) is a new paradigm which integrates cognitive features in traditional Wireless Sensor Networks (WSNs) to mitigate important problems such as spectrum occupancy. Security in Cognitive Wireless Sensor Networks is an important problem because these kinds of networks manage critical applications and data. Moreover, the specific constraints of WSN make the problem even more critical. However, effective solutions have not been implemented yet. Among the specific attacks derived from new cognitive features, the one most studied is the Primary User Emulation (PUE) attack. This paper discusses a new approach, based on anomaly behavior detection and collaboration, to detect the PUE attack in CWSN scenarios. A nonparametric CUSUM algorithm, suitable for low resource networks like CWSN, has been used in this work. The algorithm has been tested using a cognitive simulator that brings important results in this area. For example, the result shows that the number of collaborative nodes is the most important parameter in order to improve the PUE attack detection rates. If the 20% of the nodes collaborates, the PUE detection reaches the 98% with less than 1% of false positives