Unsupervised Intrusion Detection with Cross-Domain Artificial Intelligence Methods

Cybercrime is a major concern for corporations, business owners, governments and citizens, and it continues to grow in spite of increasing investments in security and fraud prevention. The main challenges in this research field are: being able to detect unknown attacks, and reducing the false positive ratio. The aim of this research work was to target both problems by leveraging four artificial intelligence techniques. The first technique is a novel unsupervised learning method based on skip-gram modeling. It was designed, developed and tested against a public dataset with popular intrusion patterns. A high accuracy and a low false positive rate were achieved without prior knowledge of attack patterns. The second technique is a novel unsupervised learning method based on topic modeling. It was applied to three related domains (network attacks, payments fraud, IoT malware traffic). A high accuracy was achieved in the three scenarios, even though the malicious activity significantly differs from one domain to the other. The third technique is a novel unsupervised learning method based on deep autoencoders, with feature selection performed by a supervised method, random forest. Obtained results showed that this technique can outperform other similar techniques. The fourth technique is based on an MLP neural network, and is applied to alert reduction in fraud prevention. This method automates manual reviews previously done by human experts, without significantly impacting accuracy

Network Intrusion Detection Using Autoencode Neural Network

In today's interconnected digital landscape, safeguarding computer networks against unauthorized access and cyber threats is of paramount importance. NIDS play a crucial role in identifying and mitigating potential security breaches. This research paper explores the application of autoencoder neural networks, a subset of deep learning techniques, in the realm of Network Intrusion Detection.Autoencoder neural networks are known for their ability to learn and represent data in a compressed, low-dimensional form. This study investigates their potential in modeling network traffic patterns and identifying anomalous activities. By training autoencoder networks on both normal and malicious network traffic data, we aim to create effective intrusion detection models that can distinguish between benign and malicious network behavior.The paper provides an in-depth analysis of the architecture and training methodologies of autoencoder neural networks for intrusion detection. It also explores various data preprocessing techniques and feature engineering approaches to enhance the model's performance. Additionally, the research evaluates the robustness and scalability of autoencoder-based NIDS in real-world network environments. Furthermore, ethical considerations in network intrusion detection, including privacy concerns and false positive rates, are discussed. It addresses the need for a balanced approach that ensures network security while respecting user privacy and minimizing disruptions. operation. This approach compresses the majority samples & increases the minority sample count in tough samples so that the IDS can achieve greater classification accuracy

Network Intrusion Detection System:A systematic study of Machine Learning and Deep Learning approaches

The rapid advances in the internet and communication fields have resulted in ahuge increase in the network size and the corresponding data. As a result, manynovel attacks are being generated and have posed challenges for network secu-rity to accurately detect intrusions. Furthermore, the presence of the intruderswiththeaimtolaunchvariousattackswithinthenetworkcannotbeignored.Anintrusion detection system (IDS) is one such tool that prevents the network frompossible intrusions by inspecting the network traffic, to ensure its confidential-ity, integrity, and availability. Despite enormous efforts by the researchers, IDSstillfaceschallengesinimprovingdetectionaccuracywhilereducingfalsealarmrates and in detecting novel intrusions. Recently, machine learning (ML) anddeep learning (DL)-based IDS systems are being deployed as potential solutionsto detect intrusions across the network in an efficient manner. This article firstclarifiestheconceptofIDSandthenprovidesthetaxonomybasedonthenotableML and DL techniques adopted in designing network-based IDS (NIDS) sys-tems. A comprehensive review of the recent NIDS-based articles is provided bydiscussing the strengths and limitations of the proposed solutions. Then, recenttrends and advancements of ML and DL-based NIDS are provided in terms ofthe proposed methodology, evaluation metrics, and dataset selection. Using theshortcomings of the proposed methods, we highlighted various research chal-lenges and provided the future scope for the research in improving ML andDL-based NIDS

Models versus Datasets: Reducing Bias through Building a Comprehensive IDS Benchmark

Today, deep learning approaches are widely used to build Intrusion Detection Systems for securing IoT environments. However, the models’ hidden and complex nature raises various concerns, such as trusting the model output and understanding why the model made certain decisions. Researchers generally publish their proposed model’s settings and performance results based on a specific dataset and a classification model but do not report the proposed model’s output and findings. Similarly, many researchers suggest an IDS solution by focusing only on a single benchmark dataset and classifier. Such solutions are prone to generating inaccurate and biased results. This paper overcomes these limitations in previous work by analyzing various benchmark datasets and various individual and hybrid deep learning classifiers towards finding the best IDS solution for IoT that is efficient, lightweight, and comprehensive in detecting network anomalies. We also showed the model’s localized predictions and analyzed the top contributing features impacting the global performance of deep learning models. This paper aims to extract the aggregate knowledge from various datasets and classifiers and analyze the commonalities to avoid any possible bias in results and increase the trust and transparency of deep learning models. We believe this paper’s findings will help future researchers build a comprehensive IDS based on well-performing classifiers and utilize the aggregated knowledge and the minimum set of significantly contributing features

DoS and DDoS mitigation using Variational Autoencoders

DoS and DDoS attacks have been growing in size and number over the last decade and existing solutions to mitigate these attacks are largely inefficient. Compared to other types of malicious cyber attacks, DoS and DDoS attacks are particularly challenging to combat. Because of their ability to mask themselves as legitimate traffic, it has proven difficult to develop methods to detect these types of attacks on a packet or flow level. In this paper, we explore the potential of Variational Autoencoders to serve as a component within an intelligent security solution that differentiates between normal and malicious traffic. The motivation behind resorting to Variational Autoencoders is that unlike normal encoders that would code an input flow as a single point, they encode a flow as a distribution over the latent space which avoids overfitting. Intuitively, this allows a Variational Autoencoder to not only learn latent representations of seen input features, but to generalize in a way that allows for an interpretation of unseen flows and flow features with slight variations. Two methods based on the ability of Variational Autoencoders to learn latent representations from network traffic flows of both benign and malicious traffic, are proposed. The first method resorts to a classifier based on the latent encodings obtained from Variational Autoencoders learned from traffic traces. The second method is an anomaly detection method, where the Variational Autoencoder is used to learn the abstract feature representations of exclusively legitimate traffic. Anomalies are then filtered out by relying on the reconstruction loss of the Variational Autoencoder. In this sense, the construction loss of the autoencoder is fed as input to a classifier that outputs the class of the traffic including benign and malign, and eventually the attack type. Thus, the second approach operates with two separate training processes on two separate data sources: the first training involving only legitimate traffic, and the second training involving all traffic classes. This is different from the first approach which operates only a single training process on the whole traffic dataset. Thus, the autoencoder of the first approach aspires to learn a general feature representation of the flows while the autoencoder of the second approach aims to exclusively learn a representation of the benign traffic. The second approach is thus more susceptible to finding zero day attacks and discovering new attacks as anomalies. Both of the proposed methods have been thoroughly tested on two separate datasets with a similar feature space. The results show that both methods are promising, with the classifier-based method being slightly superior to the anomaly-based one