Network Intrusion Datasets Used in Network Security Education

ABSTRACT There is a gap between the network security graduate and the professional life. In this paper we discussed the different types of network intrusion dataset and then we highlighted the fact that any student can easily create a network intrusion dataset that is representative of the network they are in. Intrusions can be in form of anomaly or network signature; the students cannot grasp all types but they have to have the ability to detect malicious packets within his network. Original Source URL : http://aircconline.com/ijite/V7N3/7318ijite04.pdf For more details : http://airccse.org/journal/ijite/current.htm

Are Public Intrusion Datasets Fit for Purpose: Characterising the State of the Art in Intrusion Event Datasets

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.In recent years cybersecurity attacks have caused major disruption and information loss for online organisations, with high profile incidents in the news. One of the key challenges in advancing the state of the art in intrusion detection is the lack of representative datasets. These datasets typically contain millions of time-ordered events (e.g. network packet traces, flow summaries, log entries); subsequently analysed to identify abnormal behavior and specific attacks [1]. Generating realistic datasets has historically required expensive networked assets, specialised traffic generators, and considerable design preparation. Even with advances in virtualisation it remains challenging to create and maintain a representative environment. Major improvements are needed in the design, quality and availability of datasets, to assist researchers in developing advanced detection techniques. With the emergence of new technology paradigms, such as intelligent transport and autonomous vehicles, it is also likely that new classes of threat will emerge [2]. Given the rate of change in threat behavior [3] datasets become quickly obsolete, and some of the most widely cited datasets date back over two decades. Older datasets have limited value: often heavily filtered and anonymised, with unrealistic event distributions, and opaque design methodology. The relative scarcity of (Intrusion Detection System) IDS datasets is compounded by the lack of a central registry, and inconsistent information on provenance. Researchers may also find it hard to locate datasets or understand their relative merits. In addition, many datasets rely on simulation, originating from academic or government institutions. The publication process itself often creates conflicts, with the need to de-identify sensitive information in order to meet regulations such as General Data Protection Act (GDPR) [4]. Another final issue for researchers is the lack of standardised metrics with which to compare dataset quality. In this paper we attempt to classify the most widely used public intrusion datasets, providing references to archives and associated literature. We illustrate their relative utility and scope, highlighting the threat composition, formats, special features, and associated limitations. We identify best practice in dataset design, and describe potential pitfalls of designing anomaly detection techniques based on data that may be either inappropriate, or compromised due to unrealistic threat coverage. Such contributions as made in this paper is expected to facilitate continuous research and development for effectively combating the constantly evolving cyber threat landscape

A taxonomy of network threats and the effect of current datasets on intrusion detection systems

As the world moves towards being increasingly dependent on computers and automation, building secure applications, systems and networks are some of the main challenges faced in the current decade. The number of threats that individuals and businesses face is rising exponentially due to the increasing complexity of networks and services of modern networks. To alleviate the impact of these threats, researchers have proposed numerous solutions for anomaly detection; however, current tools often fail to adapt to ever-changing architectures, associated threats and zero-day attacks. This manuscript aims to pinpoint research gaps and shortcomings of current datasets, their impact on building Network Intrusion Detection Systems (NIDS) and the growing number of sophisticated threats. To this end, this manuscript provides researchers with two key pieces of information; a survey of prominent datasets, analyzing their use and impact on the development of the past decade’s Intrusion Detection Systems (IDS) and a taxonomy of network threats and associated tools to carry out these attacks. The manuscript highlights that current IDS research covers only 33.3% of our threat taxonomy. Current datasets demonstrate a clear lack of real-network threats, attack representation and include a large number of deprecated threats, which together limit the detection accuracy of current machine learning IDS approaches. The unique combination of the taxonomy and the analysis of the datasets provided in this manuscript aims to improve the creation of datasets and the collection of real-world data. As a result, this will improve the efficiency of the next generation IDS and reflect network threats more accurately within new datasets

A taxonomy of network threats and the effect of current datasets on intrusion detection systems

As the world moves towards being increasingly dependent on computers and automation, building secure applications, systems and networks are some of the main challenges faced in the current decade. The number of threats that individuals and businesses face is rising exponentially due to the increasing complexity of networks and services of modern networks. To alleviate the impact of these threats, researchers have proposed numerous solutions for anomaly detection; however, current tools often fail to adapt to ever-changing architectures, associated threats and zero-day attacks. This manuscript aims to pinpoint research gaps and shortcomings of current datasets, their impact on building Network Intrusion Detection Systems (NIDS) and the growing number of sophisticated threats. To this end, this manuscript provides researchers with two key pieces of information; a survey of prominent datasets, analyzing their use and impact on the development of the past decade's Intrusion Detection Systems (IDS) and a taxonomy of network threats and associated tools to carry out these attacks. The manuscript highlights that current IDS research covers only 33.3% of our threat taxonomy. Current datasets demonstrate a clear lack of real-network threats, attack representation and include a large number of deprecated threats, which together limit the detection accuracy of current machine learning IDS approaches. The unique combination of the taxonomy and the analysis of the datasets provided in this manuscript aims to improve the creation of datasets and the collection of real-world data. As a result, this will improve the efficiency of the next generation IDS and reflect network threats more accurately within new datasets

Generation of a dataset for network intrusion detection in a real 5G environment

Abstract. As 5G technology is widely implemented on a global scale, both the complexity of networks and the amount of data created have exploded. Future mobile networks will incorporate artificial intelligence as a crucial enabler for intelligent wireless communications, closed-loop network optimization, and big data analytics. In these future mobile networks, network security would be of the utmost importance, as many applications expect a higher level of network security from the networking infrastructure. Therefore, conventional procedures in which action is taken following the detection of an attack would be insufficient, and self-adaptive intelligent security systems would be required. This paves the door for AI-based network security strategies in the future. In AI-based security research, the lack of comprehensive, valid datasets is a persistent issue. Publicly accessible data sets are either obsolete or insufficient for 5G security research. In addition, mobile network providers are hesitant to share actual network datasets due to privacy issues. Hence, a genuine data set from a real network is extremely beneficial to AI-based network security research. This study will describe the creation of a genuine dataset containing several attack scenarios implemented on a real 5G network with real mobile users. Since a fully operational 5G network is utilized to generate the data, this dataset is characterized by its close resemblance to real-world situations. In addition, data is collected from multiple base stations and made available as independent datasets for federated learning-based research to build a global model of intelligence for the entire network. The obtained data will be processed to identify the optimal features, and the accuracy of intrusion detection will be validated using several common machine learning and neural network models such as Decision Tree, Random Forest, K-Nearest Neighbor, Support Vector Machines and Multi Layer Perceptron. A detailed analysis of a binary classification to detect malicious and non-malicious flows as well as a multi class classification to detect different attack types is presented

A taxonomy and survey of intrusion detection system design techniques, network threats and datasets

With the world moving towards being increasingly dependent on computers and automation, one of the main challenges in the current decade has been to build secure applications, systems and networks. Alongside these challenges, the number of threats is rising exponentially due to the attack surface increasing through numerous interfaces offered for each service. To alleviate the impact of these threats, researchers have proposed numerous solutions; however, current tools often fail to adapt to ever-changing architectures, associated threats and 0-days. This manuscript aims to provide researchers with a taxonomy and survey of current dataset composition and current Intrusion Detection Systems (IDS) capabilities and assets. These taxonomies and surveys aim to improve both the efficiency of IDS and the creation of datasets to build the next generation IDS as well as to reflect networks threats more accurately in future datasets. To this end, this manuscript also provides a taxonomy and survey or network threats and associated tools. The manuscript highlights that current IDS only cover 25% of our threat taxonomy, while current datasets demonstrate clear lack of real-network threats and attack representation, but rather include a large number of deprecated threats, hence limiting the accuracy of current machine learning IDS. Moreover, the taxonomies are open-sourced to allow public contributions through a Github repository