Using Self-Organizing Maps for Computer Network Intrusion Detection

Anomaly detection in user access patterns using artificial neural networks is a novel way of combating the ever-present concern of computer network intrusion detection for many entities around the world. Anomaly detection is a technique in network security in which a profile is built around a user\u27s normal daily actions. The data collected for these profiles can be as following: file access attempts; failed login attempts; file creations; file access failures; and countless others. This data is collected and used as training data for a neural network. There are many types of neural networks, such as multi-layer feed-forward network; recurrent networks; support vector machines; and others. For our study, we implemented our own self¬ organizing map (SOM), which we found to not be as heavily researched as other neural network approaches. Using the KDD Cup 99 dataset, we compared our own SOM implementation against other neural network implementations and determine the effectiveness of such an approach

Network anomaly detection using artificial neural networks

This paper presents a method of identifying and classifying network anomalies using an artificial neural network for analyzing data gathered via Netflow protocol. Potential anomalies and their properties are described. We propose using a multilayer perceptron, trained with the backpropagation algorithm. We experiment both with datasets acquired from a real ISP monitoring system and with datasets modified to simulate the presence of anomalies; some Netflow records are modified to contain known patterns of several network attacks. We evaluate the viability of the approach by practical experimentation with various anomalies and iteration sizes

NeuDetect: A neural network data mining system for wireless network intrusion detection

This thesis proposes an Intrusion Detection System, NeuDetect, which applies Neural Network technique to wireless network packets captured through hardware sensors for purposes of real time detection of anomalous packets. To address the problem of high false alarm rate confronted by the current wireless intrusion detection systems, this thesis presents a method of applying the artificial neural networks technique to the wireless network intrusion detection system. The proposed system solution approach is to find normal and anomalous patterns on preprocessed wireless packet records by comparing them with training data using Back-propagation algorithm. An anomaly score is assigned to each packet by calculating the difference between the output error and threshold. If the anomaly score is positive then the wireless packet is flagged as anomalous and is negative then the packet is flagged as normal. If the anomaly score is zero or close to zero it will be flagged as an unknown attack and will be sent back to training process for re-evaluation

CHAOTIC SEISMIC SIGNAL MODELING BASED ON NOISE AND EARTHQUAKE ANOMALY DETECTION

Since ancient times, people have tried to predict earthquakes using simple perceptions such as animal behavior. The prediction of the time and strength of an earthquake is of primary concern. In this study chaotic signal modeling is used based on noise and detecting anomalies before an earthquake using artificial neural networks (ANNs). Artificial neural networks are efficient tools for solving complex problems such as prediction and identification. In this study, the effective features of chaotic signal model is obtained considering noise and detection of anomalies five minutes before an earthquake occurrence. Neuro-fuzzy classifier and MLP neural network approaches showed acceptable accuracy of 84.6491% and 82.8947%, respectively. Results demonstrate that the proposed method is an effective seismic signal model based on noise and anomaly detection before an earthquake

Performance evaluation of transfer learning based deep convolutional neural network with limited fused spectro-temporal data for land cover classification

Deep learning (DL) techniques are effective in various applications, such as parameter estimation, image classification, recognition, and anomaly detection. They excel with abundant training data but struggle with limited data. To overcome this, transfer learning is commonly used, leveraging complex learning abilities, saving time, and handling limited labeled data. This study assesses a transfer learning (TL)-based pre-trained “deep convolutional neural network (DCNN)” for classifying land use land cover using a limited and imbalanced dataset of fused spectro-temporal data. It compares the performance of shallow artificial neural networks (ANNs) and deep convolutional neural networks, utilizing multi-spectral sentinel-2 and high-resolution planet scope data. Both machine learning and deep learning algorithms successfully classified the fused data, but the transfer learning-based deep convolutional neural network outperformed the artificial neural network. The evaluation considered a weighted average of F1-score and overall classification accuracy. The transfer learning-based convolutional neural network achieved a weighted average F1-score of 0.92 and a classification accuracy of 0.93, while the artificial neural network achieved a weighted average F1-score of 0.87 and a classification accuracy of 0.89. These results highlight the superior performance of the transfer learned convolutional neural network on a limited and imbalanced dataset compared to the traditional artificial neural network algorithm

Intrusion detection for industrial control systems

Industrial Control Systems (ICS) are rapidly shifting from closed local networks, to remotely accessible networks. This shift has created a need for strong cybersecurity anomaly and intrusion detection for these systems; however, due to the complexity and diversity of ICSs, well defined and reliable anomaly and intrusion detection systems are still being developed. Machine learning approaches for anomaly and intrusion detection on the network level may provide general protection that can be applied to any ICS. This paper explores two machine learning applications for classifying the attack label of the UNSW-NB15 dataset. The UNSW-NB15 is a benchmark dataset that was created off general network communications and includes labels for normal behavior and attack vectors. A baseline was created using K-Nearest Neighbors (kNN) due to its mathematical simplicity. Once the baseline was created a feed forward artificial neural network known as a Multi-Layer Perceptron (MLP), was implemented for comparison due to its ease of reuse for running in a production environment. The experimental results show that both kNN and MLPs are effective approaches for identifying malicious network traffic; although, both still need to be further refined and improved before implementation on a real-world production scale

Artificial intelligence driven anomaly detection for big data systems

The main goal of this thesis is to contribute to the research on automated performance anomaly detection and interference prediction by implementing Artificial Intelligence (AI) solutions for complex distributed systems, especially for Big Data platforms within cloud computing environments. The late detection and manual resolutions of performance anomalies and system interference in Big Data systems may lead to performance violations and financial penalties. Motivated by this issue, we propose AI-based methodologies for anomaly detection and interference prediction tailored to Big Data and containerized batch platforms to better analyze system performance and effectively utilize computing resources within cloud environments. Therefore, new precise and efficient performance management methods are the key to handling performance anomalies and interference impacts to improve the efficiency of data center resources. The first part of this thesis contributes to performance anomaly detection for in-memory Big Data platforms. We examine the performance of Big Data platforms and justify our choice of selecting the in-memory Apache Spark platform. An artificial neural network-driven methodology is proposed to detect and classify performance anomalies for batch workloads based on the RDD characteristics and operating system monitoring metrics. Our method is evaluated against other popular machine learning algorithms (ML), as well as against four different monitoring datasets. The results prove that our proposed method outperforms other ML methods, typically achieving 98–99% F-scores. Moreover, we prove that a random start instant, a random duration, and overlapped anomalies do not significantly impact the performance of our proposed methodology. The second contribution addresses the challenge of anomaly identification within an in-memory streaming Big Data platform by investigating agile hybrid learning techniques. We develop TRACK (neural neTwoRk Anomaly deteCtion in sparK) and TRACK-Plus, two methods to efficiently train a class of machine learning models for performance anomaly detection using a fixed number of experiments. Our model revolves around using artificial neural networks with Bayesian Optimization (BO) to find the optimal training dataset size and configuration parameters to efficiently train the anomaly detection model to achieve high accuracy. The objective is to accelerate the search process for finding the size of the training dataset, optimizing neural network configurations, and improving the performance of anomaly classification. A validation based on several datasets from a real Apache Spark Streaming system is performed, demonstrating that the proposed methodology can efficiently identify performance anomalies, near-optimal configuration parameters, and a near-optimal training dataset size while reducing the number of experiments up to 75% compared with naïve anomaly detection training. The last contribution overcomes the challenges of predicting completion time of containerized batch jobs and proactively avoiding performance interference by introducing an automated prediction solution to estimate interference among colocated batch jobs within the same computing environment. An AI-driven model is implemented to predict the interference among batch jobs before it occurs within system. Our interference detection model can alleviate and estimate the task slowdown affected by the interference. This model assists the system operators in making an accurate decision to optimize job placement. Our model is agnostic to the business logic internal to each job. Instead, it is learned from system performance data by applying artificial neural networks to establish the completion time prediction of batch jobs within the cloud environments. We compare our model with three other baseline models (queueing-theoretic model, operational analysis, and an empirical method) on historical measurements of job completion time and CPU run-queue size (i.e., the number of active threads in the system). The proposed model captures multithreading, operating system scheduling, sleeping time, and job priorities. A validation based on 4500 experiments based on the DaCapo benchmarking suite was carried out, confirming the predictive efficiency and capabilities of the proposed model by achieving up to 10% MAPE compared with the other models.Open Acces

AI Solutions for MDS: Artificial Intelligence Techniques for Misuse Detection and Localisation in Telecommunication Environments

This report considers the application of Articial Intelligence (AI) techniques to the problem of misuse detection and misuse localisation within telecommunications environments. A broad survey of techniques is provided, that covers inter alia rule based systems, model-based systems, case based reasoning, pattern matching, clustering and feature extraction, articial neural networks, genetic algorithms, arti cial immune systems, agent based systems, data mining and a variety of hybrid approaches. The report then considers the central issue of event correlation, that is at the heart of many misuse detection and localisation systems. The notion of being able to infer misuse by the correlation of individual temporally distributed events within a multiple data stream environment is explored, and a range of techniques, covering model based approaches, `programmed' AI and machine learning paradigms. It is found that, in general, correlation is best achieved via rule based approaches, but that these suffer from a number of drawbacks, such as the difculty of developing and maintaining an appropriate knowledge base, and the lack of ability to generalise from known misuses to new unseen misuses. Two distinct approaches are evident. One attempts to encode knowledge of known misuses, typically within rules, and use this to screen events. This approach cannot generally detect misuses for which it has not been programmed, i.e. it is prone to issuing false negatives. The other attempts to `learn' the features of event patterns that constitute normal behaviour, and, by observing patterns that do not match expected behaviour, detect when a misuse has occurred. This approach is prone to issuing false positives, i.e. inferring misuse from innocent patterns of behaviour that the system was not trained to recognise. Contemporary approaches are seen to favour hybridisation, often combining detection or localisation mechanisms for both abnormal and normal behaviour, the former to capture known cases of misuse, the latter to capture unknown cases. In some systems, these mechanisms even work together to update each other to increase detection rates and lower false positive rates. It is concluded that hybridisation offers the most promising future direction, but that a rule or state based component is likely to remain, being the most natural approach to the correlation of complex events. The challenge, then, is to mitigate the weaknesses of canonical programmed systems such that learning, generalisation and adaptation are more readily facilitated