Using Fuzzy Logic Algorithms and Growing Hierarchical Self-Organizing Maps to Define Efficient Security Inspection Strategies in a Container Terminal

Maritime transport is one of the oldest methods of moving various types of goods, and it continues to have an important role in our modern society. More than 20 million containers are transported across the oceans daily. However, this form of transportation is constantly threatened by illegal operations, such as the smuggling of goods or people and merchandise theft. Port security departments must be prepared to face the different threats and challenges that accompany the use of innovative techniques and devices to achieve efficient inspection strategies. Two inspection strategies are presented in this study. The first strategy is based on fuzzy logic (FL), and the second strategy is based on the growing hierarchical self-organizing map (GHSOM) approach. The weight variation and security index (SI) of a container and the readings from certain technologies, such as radio-frequency identification (RFID) and X-ray scanning, are considered as the input data. To minimize the inspection time and considering the costs associated with the security inspections of containers, the results of both inspection strategies are compared and analyzed. The findings indicate there is potential for improving the effectiveness of security inspections by employing both techniques, and the specific relevance in the case of GHSOMs is discussed.Programa Estatal de Investigación, Desarrollo e Innovación Orientada a los Retos de la Sociedad - “Estrategias de diseño microelectronico para IOT en escenarios hostiles” TEC2016-80396-C2-2-

Explainable Intrusion Detection Systems using white box techniques

Artificial Intelligence (AI) has found increasing application in various domains, revolutionizing problem-solving and data analysis. However, in decision-sensitive areas like Intrusion Detection Systems (IDS), trust and reliability are vital, posing challenges for traditional black box AI systems. These black box IDS, while accurate, lack transparency, making it difficult to understand the reasons behind their decisions. This dissertation explores the concept of eXplainable Intrusion Detection Systems (X-IDS), addressing the issue of trust in X-IDS. It explores the limitations of common black box IDS and the complexities of explainability methods, leading to the fundamental question of trusting explanations generated by black box explainer modules. To address these challenges, this dissertation presents the concept of white box explanations, which are innately explainable. While white box algorithms are typically simpler and more interpretable, they often sacrifice accuracy. However, this work utilized white box Competitive Learning (CL), which can achieve competitive accuracy in comparison to black box IDS. We introduce Rule Extraction (RE) as another white box technique that can be applied to explain black box IDS. It involves training decision trees on the inputs, weights, and outputs of black box models, resulting in human-readable rulesets that serve as global model explanations. These white box techniques offer the benefits of accuracy and trustworthiness, which are challenging to achieve simultaneously. This work aims to address gaps in the existing literature, including the need for highly accurate white box IDS, a methodology for understanding explanations, small testing datasets, and comparisons between white box and black box models. To achieve these goals, the study employs CL and eclectic RE algorithms. CL models offer innate explainability and high accuracy in IDS applications, while eclectic RE enhances trustworthiness. The contributions of this dissertation include a novel X-IDS architecture featuring Self-Organizing Map (SOM) models that adhere to DARPA’s guidelines for explainable systems, an extended X-IDS architecture incorporating three CL-based algorithms, and a hybrid X-IDS architecture combining a Deep Neural Network (DNN) predictor with a white box eclectic RE explainer. These architectures create more explainable, trustworthy, and accurate X-IDS systems, paving the way for enhanced AI solutions in decision-sensitive domains

Feature selection by multi-objective optimization: application to network anomaly detection by hierarchical self-organizing maps.

Feature selection is an important and active issue in clustering and classification problems. By choosing an adequate feature subset, a dataset dimensionality reduction is allowed, thus contributing to decreasing the classification computational complexity, and to improving the classifier performance by avoiding redundant or irrelevant features. Although feature selection can be formally defined as an optimisation problem with only one objective, that is, the classification accuracy obtained by using the selected feature subset, in recent years, some multi-objective approaches to this problem have been proposed. These either select features that not only improve the classification accuracy, but also the generalisation capability in case of supervised classifiers, or counterbalance the bias toward lower or higher numbers of features that present some methods used to validate the clustering/classification in case of unsupervised classifiers. The main contribution of this paper is a multi-objective approach for feature selection and its application to an unsupervised clustering procedure based on Growing Hierarchical Self-Organizing Maps (GHSOM) that includes a new method for unit labelling and efficient determination of the winning unit. In the network anomaly detection problem here considered, this multi-objective approach makes it possible not only to differentiate between normal and anomalous traffic but also among different anomalies. The efficiency of our proposals has been evaluated by using the well-known DARPA/NSL-KDD datasets that contain extracted features and labeled attacks from around 2 million connections. The selected feature sets computed in our experiments provide detection rates up to 99.8% with normal traffic and up to 99.6% with anomalous traffic, as well as accuracy values up to 99.12%.This work has been funded by FEDER funds and the Ministerio de Ciencia e Innovación of the Spanish Government under Project No. TIN2012-32039

Pruning GHSOM to create an explainable intrusion detection system

Intrusion Detection Systems (IDS) that provide high detection rates but are black boxes leadto models that make predictions a security analyst cannot understand. Self-Organizing Maps(SOMs) have been used to predict intrusion to a network, while also explaining predictions throughvisualization and identifying significant features. However, they have not been able to compete withthe detection rates of black box models. Growing Hierarchical Self-Organizing Maps (GHSOMs)have been used to obtain high detection rates on the NSL-KDD and CIC-IDS-2017 network trafficdatasets, but they neglect creating explanations or visualizations, which results in another blackbox model.This paper offers a high accuracy, Explainable Artificial Intelligence (XAI) based on GHSOMs.One obstacle to creating a white box hierarchical model is the model growing too large and complexto understand. Another contribution this paper makes is a pruning method used to cut down onthe size of the GHSOM, which provides a model that can provide insights and explanation whilemaintaining a high detection rate

From Intrusion Detection to Attacker Attribution: A Comprehensive Survey of Unsupervised Methods

Over the last five years there has been an increase in the frequency and diversity of network attacks. This holds true, as more and more organisations admit compromises on a daily basis. Many misuse and anomaly based Intrusion Detection Systems (IDSs) that rely on either signatures, supervised or statistical methods have been proposed in the literature, but their trustworthiness is debatable. Moreover, as this work uncovers, the current IDSs are based on obsolete attack classes that do not reflect the current attack trends. For these reasons, this paper provides a comprehensive overview of unsupervised and hybrid methods for intrusion detection, discussing their potential in the domain. We also present and highlight the importance of feature engineering techniques that have been proposed for intrusion detection. Furthermore, we discuss that current IDSs should evolve from simple detection to correlation and attribution. We descant how IDS data could be used to reconstruct and correlate attacks to identify attackers, with the use of advanced data analytics techniques. Finally, we argue how the present IDS attack classes can be extended to match the modern attacks and propose three new classes regarding the outgoing network communicatio

A Novel Threat Intelligence Detection Model Using Neural Networks

A network intrusion detection system (IDS) is commonly recognized as an effective solution for identifying threats and malicious attacks. Due to the rapid emergence of threats and new attack vectors, novel and adaptive approaches must be considered to maintain the effectiveness of IDSs. In this paper, we present a novel Threat Intelligence Detection Model (TIDM) for online intrusion detection. The proposed TIDM focuses on the online processing of massive data flows and is accordingly able to reveal unknown connections, including zero-day attacks. The TIDM consists of three components: an optimized filter (OptiFilter), an adaptive and hybrid classifier, and an alarm component. The main contributions of the OptiFilter component are in its ability to continuously capture data flows and construct unlabeled connection vectors. The second component of the TIDM employs a hybrid model made up of an enhanced growing hierarchical self-organizing map (EGHSOM) and a normal network behavior (NNB) model to jointly identify unknown connections. The proposed TIDM updates the hybrid model continually in real-time. The model’s performance evaluation has been carried out in both offline and online operational modes using a quantitative approach that considers all possible evaluation metrics for the datasets and the hybrid classification method. The achieved results show that the proposed TIDM is able, with promising performance, to process massive data flows in real-time, classify unlabeled connections, reveal the label of unknown connections, and perform online updates successfully

ESTUDIO COMPARATIVO DE TÉCNICAS DE ENTRENAMIENTO Y CLASIFICACIÓN EN SISTEMAS DE DETECCIÓN DE INSTRUSOS (IDS), BASADOS EN ANOMALIAS DE RED.

Maestría en Ingeniería (Énfasis en Redes y Software)The main motivation of this investigation was the implementation of the Draper method applied to intrusion detection systems in different training and classification techniques in order to identify the best intrusion detection model with the objective of improving detection rates of attacks in computer network systems, using a procedure of selection of characteristics and different methods of algorithms of unsupervised trainings, in this case was used the technique INFO.GAIN identifying that the number of optimal characteristics is 15. Consequently, a neural network using a non-supervised learning algorithm (GHSOM, RANDOM FOREST, BAYESIAN NETWORKS, NAIVE BAYES, C4.5, LOGISTIC, PART AND NBTREE) for the purpose of classifying bi-class traffic automatically. obtained the best technique of training and classification using the selection technique In INFO.GAIN with 15 characteristics and cross validation 10 pligues, was the RANDOM FOREST technique.La principal motivación de esta investigación ha sido la implementación del método Draper aplicado a los sistemas de detección de intrusos en distintas técnicas de entrenamiento y clasificación con el propósito de identificar el mejor modelo de detección de intrusiones con el objetivo de mejorar las tasas de detección de ataques en sistemas de redes computacionales, utilizando un procedimiento de selección de características y distintos métodos de algoritmos de entrenamientos no supervisados, en este caso se utilizó la técnica INFO.GAIN identificando que el número de características óptimo es 15. En consecuencia, se entrenó una red neuronal que utilizan un algoritmo de aprendizaje no supervisado (GHSOM, RANDOM FOREST, REDES BAYESIANAS, NAIVE BAYES, C4.5,LOGISTIC, PART Y NBTREE ), con el propósito de clasificar el tráfico bi-clase de forma automática, Como resultado se obtuvo que la mejor técnica de entrenamiento y clasificación utilizando la técnica de selección INFO.GAIN a 15 características y validación cruzada 10 pligues, fue la técnica RANDOM FOREST