INSOMNIA:Towards Concept-Drift Robustness in Network Intrusion Detection

Despite decades of research in network traffic analysis and incredible advances in artificial intelligence, network intrusion detection systems based on machine learning (ML) have yet to prove their worth. One core obstacle is the existence of concept drift, an issue for all adversary-facing security systems. Additionally, specific challenges set intrusion detection apart from other ML-based security tasks, such as malware detection. In this work, we offer a new perspective on these challenges. We propose INSOMNIA, a semi-supervised intrusion detector which continuously updates the underlying ML model as network traffic characteristics are affected by concept drift. We use active learning to reduce latency in the model updates, label estimation to reduce labeling overhead, and apply explainable AI to better interpret how the model reacts to the shifting distribution. To evaluate INSOMNIA, we extend TESSERACT - a framework originally proposed for performing sound time-aware evaluations of ML-based malware detectors - to the network intrusion domain. Our evaluation shows that accounting for drifting scenarios is vital for effective intrusion detection systems

Cyber Situation Awareness with Active Learning for Intrusion Detection

Intrusion detection has focused primarily on detecting cyberattacks at the event-level. Since there is such a large volume of network data and attacks are minimal, machine learning approaches have focused on improving accuracy and reducing false positives, but this has frequently resulted in overfitting. In addition, the volume of intrusion detection alerts is large and creates fatigue in the human analyst who must review them. This research addresses the problems associated with event-level intrusion detection and the large volumes of intrusion alerts by applying active learning and cyber situation awareness. This paper includes the results of two experiments using the UNSW-NB15 dataset. The first experiment evaluated sampling approaches for querying the oracle, as part of active learning. It then trained a Random Forest classifier using the samples and evaluated its results. The second experiment applied cyber situation awareness by aggregating the detection results of the first experiment and calculating the probability that a computer system was part of a cyberattack. This research showed that moving the perspective of event-level alerts to the probability that a computer system was part of an attack improved the accuracy of detection and reduced the volume of alerts that a human analyst would need to review.Comment: McElwee, S. & Cannady, J. (2019). Cyber situation awareness with active learning for intrusion detection. SoutheastCon 2019. IEEE. Pre-prin

Intrusion Detection in Mobile Ad Hoc Networks Using Transductive Machine Learning Techniques

This thesis presents a research whose objective is to design an intrusion detection model for Mobile Ad hoc NETworks (MANET). MANET is an autonomous system consisting of a group of mobile nodes with no infrastructure support. The MANET environment is particularly vulnerable because of the characteristics of mobile ad hoc networks such as open medium, dynamic topology, distributed cooperation, and constrained capability. Unfortunately, the traditional mechanisms designed for protecting networks are not directly applicable to MANETs without modifications. In the past decades, machine learning methods have been successfully used in several intrusion detection methods because of their ability to discover and detect novel attacks. This research investigates the use of a promising technique from machine learning to designing the most suitable intrusion detection for this challenging network type. The proposed algorithm employs a combined model that uses two different measures (nonconformity metric measures and Local Distance-based Outlier Factor (LDOF)) to improve its detection ability. Moreover, the algorithm can provide a graded confidence that indicates the reliability of the classification. In machine learning algorithm, choosing the most relevant features for each attack is a very important requirement, especially in mobile ad hoc networks where the network topology dynamically changes. Feature selection is undertaken to select the relevant subsets of features to build an efficient prediction model and improve intrusion detection performance by removing irrelevant features. The transductive conformal prediction and outlier detection have been employed for feature selection algorithm. Traditional intrusion detection techniques have had trouble dealing with dynamic environments. In particular, issues such as collects real time attack related audit data and cooperative global detection. Therefore, the researcher is motivated to design a new intrusion detection architecture which involves new detection technique to efficiently detect the abnormalities in the ad hoc networks. The proposed model has distributed and cooperative hierarchical architecture, where nodes communicate with their region gateway node to make decisions. To validate the research, the researcher presents case study using GLOMOSIM simulation platform with AODV ad hoc routing protocols. Various active attacks are implemented. A series of experimental results demonstrate that the proposed intrusion detection model can effectively detect anomalies with low false positive rate, high detection rate and achieve high detection accuracy

A near-autonomous and incremental intrusion detection system through active learning of known and unknown attacks

Intrusion detection is a traditional practice of security experts, however, there are several issues which still need to be tackled. Therefore, in this paper, after highlighting these issues, we present an architecture for a hybrid Intrusion Detection System (IDS) for an adaptive and incremental detection of both known and unknown attacks. The IDS is composed of supervised and unsupervised modules, namely, a Deep Neural Network (DNN) and the K-Nearest Neighbors (KNN) algorithm, respectively. The proposed system is near-autonomous since the intervention of the expert is minimized through the active learning (AL) approach. A query strategy for the labeling process is presented, it aims at teaching the supervised module to detect unknown attacks and improve the detection of the already-known attacks. This teaching is achieved through sliding windows (SW) in an incremental fashion where the DNN is retrained when the data is available over time, thus rendering the IDS adaptive to cope with the evolutionary aspect of the network traffic. A set of experiments was conducted on the CICIDS2017 dataset in order to evaluate the performance of the IDS, promising results were obtained.Comment: 6 pages, 3 figures, 32 references, conferenc

Autoencoders: A Low Cost Anomaly Detection Method for Computer Network Data Streams

Computer networks are vulnerable to cyber attacks that can affect the confidentiality, integrity and availability of mission critical data. Intrusion detection methods can be employed to detect these attacks in real-time. Anomaly detection offers the advantage of detecting unknown attacks in a semi-supervised fashion. This paper aims to answer the question if autoencoders, a type of semisupervised feedforward neural network, can provide a low cost anomaly detector method for computer network data streams. Autoencoder methods were evaluated online with the KDD’99 and UNSW-NB15 data sets, demonstrating that running time and labeling cost are significantly reduced compared to traditional online classification techniques for similar detection performance. Further research would consider the trade-off between single vs stacked networks, multi-label classification, concept drift detection and active learning