Intelligent Log Analysis for Anomaly Detection

Computer logs are a rich source of information that can be analyzed to detect various issues. The large volumes of logs limit the effectiveness of manual approaches to log analysis. The earliest automated log analysis tools take a rule-based approach, which can only detect known issues with existing rules. On the other hand, anomaly detection approaches can detect new or unknown issues. This is achieved by looking for unusual behavior different from the norm, often utilizing machine learning (ML) or deep learning (DL) models. In this project, we evaluated various ML and DL techniques used for log anomaly detection. We propose a hybrid neural network (NN) we call CausalConvLSTM for modeling log sequences, which takes advantage of both Convolutional Neural Network and Long Short-Term Memory Network\u27s strengths. Furthermore, we evaluated and proposed a concrete strategy for retraining NN anomaly detection models to maintain a low false-positive rate in a drifting environment

A Deep Learning-Based Framework for Feature Extraction and Classification of Intrusion Detection in Networks

An intrusion detection system, often known as an IDS, is extremely important for preventing attacks on a network, violating network policies, and gaining unauthorized access to a network. The effectiveness of IDS is highly dependent on data preprocessing techniques and classification models used to enhance accuracy and reduce model training and testing time. For the purpose of anomaly identification, researchers have developed several machine learning and deep learning-based algorithms; nonetheless, accurate anomaly detection with low test and train times remains a challenge. Using a hybrid feature selection approach and a deep neural network- (DNN-) based classifier, the authors of this research suggest an enhanced intrusion detection system (IDS). In order to construct a subset of reduced and optimal features that may be used for classification, a hybrid feature selection model that consists of three methods, namely, chi square, ANOVA, and principal component analysis (PCA), is applied. These methods are referred to as “the big three.” On the NSL-KDD dataset, the suggested model receives training and is then evaluated. The proposed method was successful in achieving the following results: a reduction of input data by 40%, an average accuracy of 99.73%, a precision score of 99.75%, an F1 score of 99.72%, and an average training and testing time of 138% and 2.7 seconds, respectively. The findings of the experiments demonstrate that the proposed model is superior to the performance of the other comparison approaches.publishedVersio

Self-Taught Anomaly Detection With Hybrid Unsupervised/Supervised Machine Learning in Optical Networks

This paper proposes a self-taught anomaly detection framework for optical networks. The proposed framework makes use of a hybrid unsupervised and supervised machine learning scheme. First, it employs an unsupervised data clustering module (DCM) to analyze the patterns of monitoring data. The DCM enables a self-learning capability that eliminates the requirement of prior knowledge of abnormal network behaviors and therefore can potentially detect unforeseen anomalies. Second, we introduce a self-taught mechanism that transfers the patterns learned by the DCM to a supervised data regression and classification module (DRCM). The DRCM, whose complexity is mainly related to the scale of the applied supervised learning model, can potentially facilitate more scalable and time-efficient online anomaly detection by avoiding excessively traversing the original dataset. We designed the DCM and DRCM based on the density-based clustering algorithm and the deep neural network structure, respectively. Evaluations with experimental data from two use cases (i.e., single-point detection and end-to-end detection) demonstrate that up to 99% anomaly detection accuracy can be achieved with a false positive rate below 1%

Uncertainty Quantification for Deep Learning in Ultrasonic Crack Characterization

Deep learning for nondestructive evaluation (NDE) has received a lot of attention in recent years for its potential ability to provide human level data analysis. However, little research into quantifying the uncertainty of its predictions has been done. Uncertainty quantification (UQ) is essential for qualifying NDE inspections and building trust in their predictions. Therefore, this article aims to demonstrate how UQ can best be achieved for deep learning in the context of crack sizing for inline pipe inspection. A convolutional neural network architecture is used to size surface breaking defects from plane wave imaging (PWI) images with two modern UQ methods: deep ensembles and Monte Carlo dropout. The network is trained using PWI images of surface breaking defects simulated with a hybrid finite element / ray-based model. Successful UQ is judged by calibration and anomaly detection, which refer to whether in-domain model error is proportional to uncertainty and if out of training domain data is assigned high uncertainty. Calibration is tested using simulated and experimental images of surface breaking cracks, while anomaly detection is tested using experimental side-drilled holes and simulated embedded cracks. Monte Carlo dropout demonstrates poor uncertainty quantification with little separation between in and out-of-distribution data and a weak linear fit ( R=0.84 ) between experimental root-mean-square-error and uncertainty. Deep ensembles improve upon Monte Carlo dropout in both calibration ( R=0.95 ) and anomaly detection. Adding spectral normalization and residual connections to deep ensembles slightly improves calibration ( R=0.98 ) and significantly improves the reliability of assigning high uncertainty to out-of-distribution samples

A Hybrid Classification Framework for Network Intrusion Detection with High Accuracy and Low Latency

Network intrusion detection (NIDS) is a crucial task aimed at safeguarding computer networks against malicious attacks. Traditional NIDS methods can be categorized as either misuse-based or anomaly-based, each having its unique set of limitations. Misuse-based approaches excel in identifying known attacks but fall short when dealing with new or unidentified attack patterns. On the other hand, anomaly-based methods are more adept at identifying novel attacks but tend to produce a substantial number of false positives. To enhance the overall performance of NIDS systems, hybrid classification techniques are employed, leveraging the strengths of both misuse-based and anomaly-based methods. In this research, we present a novel hybrid classification approach for NIDS that excels in both speed and accuracy. Our approach integrates a blend of machine learning algorithms, including decision trees, support vector machines, and deep neural networks. We conducted comprehensive evaluations of our approach using various network intrusion datasets, achieving state-of-the-art results in terms of accuracy and prediction speed

A Dependable Hybrid Machine Learning Model for Network Intrusion Detection

Network intrusion detection systems (NIDSs) play an important role in computer network security. There are several detection mechanisms where anomaly-based automated detection outperforms others significantly. Amid the sophistication and growing number of attacks, dealing with large amounts of data is a recognized issue in the development of anomaly-based NIDS. However, do current models meet the needs of today's networks in terms of required accuracy and dependability? In this research, we propose a new hybrid model that combines machine learning and deep learning to increase detection rates while securing dependability. Our proposed method ensures efficient pre-processing by combining SMOTE for data balancing and XGBoost for feature selection. We compared our developed method to various machine learning and deep learning algorithms to find a more efficient algorithm to implement in the pipeline. Furthermore, we chose the most effective model for network intrusion based on a set of benchmarked performance analysis criteria. Our method produces excellent results when tested on two datasets, KDDCUP'99 and CIC-MalMem-2022, with an accuracy of 99.99% and 100% for KDDCUP'99 and CIC-MalMem-2022, respectively, and no overfitting or Type-1 and Type-2 issues.Comment: Accepted in the Journal of Information Security and Applications (Scopus, Web of Science (SCIE) Journal, Quartile: Q1, Site Score: 7.6, Impact Factor: 4.96) on 7 December 202

Lightweight Deep Learning Framework to Detect Botnets in IoT Sensor Networks by using Hybrid Self-Organizing Map

In recent years, we have witnessed a massive growth of intrusion attacks targeted at the internet of things (IoT) devices. Due to inherent security vulnerabilities, it has become an easy target for hackers to target these devices. Recent studies have been focusing on deploying intrusion detection systems at the edge of the network within these devices to localize threat mitigation to avoid computational expenses. Intrusion detection systems based on machine learning and deep learning algorithm have demonstrated the potential capability to detect zero-day attacks where traditional signature-based detection falls short. The paper aims to propose a lightweight and robust deep learning framework for intrusion detection that has computational potential to be deployed within IoT devices. The research builds upon previous researches showing the demonstrated efficiency of anomaly detection rates of self-organizing map-based intrusion. The paper will contribute to the existing body of knowledge by creating a hybrid self-organizing map (SOM) for the purpose of detecting botnet attacks and analyzing its accuracy compared with a traditional supervised artificial neural network (ANN). The paper also aims to answer questions regarding the computational efficiency of our hybrid self-organizing map by measuring the CPU consumption based on time to train model. The deep learning prototypes will be trained on the NSL-KDD dataset and Detection of IoT botnet Attacks dataset. The study will evaluate the performance of a self-organizing map based k-nearest neighbor prototype with the performance of a supervised artificial neural network based on validation metrics such as confusion matrix, f1, recall, precision, and accuracy score