From Intrusion Detection to Attacker Attribution: A Comprehensive Survey of Unsupervised Methods

Over the last five years there has been an increase in the frequency and diversity of network attacks. This holds true, as more and more organisations admit compromises on a daily basis. Many misuse and anomaly based Intrusion Detection Systems (IDSs) that rely on either signatures, supervised or statistical methods have been proposed in the literature, but their trustworthiness is debatable. Moreover, as this work uncovers, the current IDSs are based on obsolete attack classes that do not reflect the current attack trends. For these reasons, this paper provides a comprehensive overview of unsupervised and hybrid methods for intrusion detection, discussing their potential in the domain. We also present and highlight the importance of feature engineering techniques that have been proposed for intrusion detection. Furthermore, we discuss that current IDSs should evolve from simple detection to correlation and attribution. We descant how IDS data could be used to reconstruct and correlate attacks to identify attackers, with the use of advanced data analytics techniques. Finally, we argue how the present IDS attack classes can be extended to match the modern attacks and propose three new classes regarding the outgoing network communicatio

Clustering extension of MOVICAB-IDS to distinguish intrusions in flow-based data

Much effort has been devoted to research on intrusion detection (ID) in recent years because intrusion strategies and technologies are constantly and quickly evolving. As an innovative solution based on visualization, MObile VIsualisation Connectionist Agent-Based IDS was previously proposed, conceived as a hybrid-intelligent ID System. It was designed to analyse continuous network data at a packet level and is extended in present paper for the analysis of flow-based traffic data. By incorporating clustering techniques to the original proposal, network flows are investigated trying to identify different types of attacks. The analysed real-life data (the well-known dataset from the University of Twente) come from a honeypot directly connected to the Internet (thus ensuring attack-exposure) and is analysed by means of clustering and neural techniques, individually and in conjunction. Promising results are obtained, proving the validity of the proposed extension for the analysis of network flow dat

Probabilistic Modeling and Inference for Obfuscated Network Attack Sequences

Prevalent computing devices with networking capabilities have become critical network infrastructure for government, industry, academia and every-day life. As their value rises, the motivation driving network attacks on this infrastructure has shifted from the pursuit of notoriety to the pursuit of profit or political gains, leading to network attack on various scales. Facing diverse network attack strategies and overwhelming alters, much work has been devoted to correlate observed malicious events to pre-defined scenarios, attempting to deduce the attack plans based on expert models of how network attacks may transpire. We started the exploration of characterizing network attacks by investigating how temporal and spatial features of attack sequence can be used to describe different types of attack sources in real data set. Attack sequence models were built from real data set to describe different attack strategies. Based on the probabilistic attack sequence model, attack predictions were made to actively predict next possible actions. Experiments through attack predictions have revealed that sophisticated attackers can employ a number of obfuscation techniques to confuse the alert correlation engine or classifier. Unfortunately, most exiting work treats attack obfuscations by developing ad-hoc fixes to specific obfuscation technique. To this end, we developed an attack modeling framework that enables a systematical analysis of obfuscations. The proposed framework represents network attack strategies as general finite order Markov models and integrates it with different attack obfuscation models to form probabilistic graphical model models. A set of algorithms is developed to inference the network attack strategies given the models and the observed sequences, which are likely to be obfuscated. The algorithms enable an efficient analysis of the impact of different obfuscation techniques and attack strategies, by determining the expected classification accuracy of the obfuscated sequences. The algorithms are developed by integrating the recursion concept in dynamic programming and the Monte-Carlo method. The primary contributions of this work include the development of the formal framework and the algorithms to evaluate the impact of attack obfuscations. Several knowledge-driven attack obfuscation models are developed and analyzed to demonstrate the impact of different types of commonly used obfuscation techniques. The framework and algorithms developed in this work can also be applied to other contexts beyond network security. Any behavior sequences that might suffer from noise and require matching to pre-defined models can use this work to recover the most likely original sequence or evaluate quantitatively the expected classification accuracy one can achieve to separate the sequences

A Critical Analysis of Payload Anomaly-Based Intrusion Detection Systems

Examining payload content is an important aspect of network security, particularly in today\u27s volatile computing environment. An Intrusion Detection System (IDS) that simply analyzes packet header information cannot adequately secure a network from malicious attacks. The alternative is to perform deep-packet analysis using n-gram language parsing and neural network technology. Self Organizing Map (SOM), PAYL over Self-Organizing Maps for Intrusion Detection (POSEIDON), Anomalous Payload-based Network Intrusion Detection (PAYL), and Anagram are next-generation unsupervised payload anomaly-based IDSs. This study examines the efficacy of each system using the design-science research methodology. A collection of quantitative data and qualitative features exposes their strengths and weaknesses

New Anomaly Network Intrusion Detection System in Cloud Environment Based on Optimized Back Propagation Neural Network Using Improved Genetic Algorithm

Cloud computing is distributed architecture, providing computing facilities and storage resource as a service over an open environment (Internet), this lead to different matters related to the security and privacy in cloud computing. Thus, defending network accessible Cloud resources and services from various threats and attacks is of great concern. To address this issue, it is essential to create an efficient and effective Network Intrusion System (NIDS) to detect both outsider and insider intruders with high detection precision in the cloud environment. NIDS has become popular as an important component of the network security infrastructure, which detects malicious activities by monitoring network traffic. In this work, we propose to optimize a very popular soft computing tool widely used for intrusion detection namely, Back Propagation Neural Network (BPNN) using an Improved Genetic Algorithm (IGA). Genetic Algorithm (GA) is improved through optimization strategies, namely Parallel Processing and Fitness Value Hashing, which reduce execution time, convergence time and save processing power. Since,  Learning rate and Momentum term are among the most relevant parameters that impact the performance of BPNN classifier, we have employed IGA to find the optimal or near-optimal values of these two parameters which ensure high detection rate, high accuracy and low false alarm rate. The CloudSim simulator 4.0 and DARPA’s KDD cup datasets 1999 are used for simulation. From the detailed performance analysis, it is clear that the proposed system called “ANIDS BPNN-IGA” (Anomaly NIDS based on BPNN and IGA) outperforms several state-of-art methods and it is more suitable for network anomaly detection

Predictive Cyber Situational Awareness and Personalized Blacklisting: A Sequential Rule Mining Approach

Cybersecurity adopts data mining for its ability to extract concealed and indistinct patterns in the data, such as for the needs of alert correlation. Inferring common attack patterns and rules from the alerts helps in understanding the threat landscape for the defenders and allows for the realization of cyber situational awareness, including the projection of ongoing attacks. In this paper, we explore the use of data mining, namely sequential rule mining, in the analysis of intrusion detection alerts. We employed a dataset of 12 million alerts from 34 intrusion detection systems in 3 organizations gathered in an alert sharing platform, and processed it using our analytical framework. We execute the mining of sequential rules that we use to predict security events, which we utilize to create a predictive blacklist. Thus, the recipients of the data from the sharing platform will receive only a small number of alerts of events that are likely to occur instead of a large number of alerts of past events. The predictive blacklist has the size of only 3 % of the raw data, and more than 60 % of its entries are shown to be successful in performing accurate predictions in operational, real-world settings

Holistic Network Defense: Fusing Host and Network Features for Attack Classification

This work presents a hybrid network-host monitoring strategy, which fuses data from both the network and the host to recognize malware infections. This work focuses on three categories: Normal, Scanning, and Infected. The network-host sensor fusion is accomplished by extracting 248 features from network traffic using the Fullstats Network Feature generator and from the host using text mining, looking at the frequency of the 500 most common strings and analyzing them as word vectors. Improvements to detection performance are made by synergistically fusing network features obtained from IP packet flows and host features, obtained from text mining port, processor, logon information among others. In addition, the work compares three different machine learning algorithms and updates the script required to obtain network features. Hybrid method results outperformed host only classification by 31.7% and network only classification by 25%. The new approach also reduces the number of alerts while remaining accurate compared with the commercial IDS SNORT. These results make it such that even the most typical users could understand alert classification messages

Visualization and clustering for SNMP intrusion detection

Accurate intrusion detection is still an open challenge. The present work aims at being one step toward that purpose by studying the combination of clustering and visualization techniques. To do that, the mobile visualization connectionist agent-based intrusion detection system (MOVICAB-IDS), previously proposed as a hybrid intelligent IDS based on visualization techniques, is upgraded by adding automatic response thanks to clustering methods. To check the validity of the proposed clustering extension, it has been applied to the identification of different anomalous situations related to the simple network management network protocol by using real-life data sets. Different ways of applying neural projection and clustering techniques are studied in the present article. Through the experimental validation it is shown that the proposed techniques could be compatible and consequently applied to a continuous network flow for intrusion detectionSpanish Ministry of Economy and Competitiveness with ref: TIN2010-21272-C02-01 (funded by the European Regional Development Fund) and SA405A12-2 from Junta de Castilla y Leon

WK-FNN DESIGN FOR DETECTION OF ANOMALIES IN THE COMPUTER NETWORK TRAFFIC

Anomaly-based intrusion detection systems identify abnormal computer network traffic based on deviations from the derived statistical model that describes the normal network behavior. The basic problem with anomaly detection is deciding what is considered normal. Supervised machine learning can be viewed as binary classification, since models are trained and tested on a data set containing a binary label to detect anomalies. Weighted k-Nearest Neighbor and Feedforward Neural Network are high-precision classifiers for decision-making. However, their decisions sometimes differ. In this paper, we present a WK-FNN hybrid model for the detection of the opposite decisions. It is shown that results can be improved with the xor bitwise operation. The sum of the binary “ones” is used to decide whether additional alerts are activated or not