Temporally adaptive monitoring procedures with applications in enterprise cyber-security

Due to the perpetual threat of cyber-attacks, enterprises must employ and develop new methods of detection as attack vectors evolve and advance. Enterprise computer networks produce a large volume and variety of data including univariate data streams, time series and network graph streams. Motivated by cyber-security, this thesis develops adaptive monitoring tools for univariate and network graph data streams, however, they are not limited to this domain. In all domains, real data streams present several challenges for monitoring including trend, periodicity and change points. Streams often also have high volume and frequency. To deal with the non-stationarity in the data, the methods applied must be adaptive. Adaptability in the proposed procedures throughout the thesis is introduced using forgetting factors, weighting the data accordingly to recency. Secondly, methods applied must be computationally fast with a small or fixed computation burden and fixed storage requirements for timely processing. Throughout this thesis, sequential or sliding window approaches are employed to achieve this. The first part of the thesis is centred around univariate monitoring procedures. A sequential adaptive parameter estimator is proposed using a Bayesian framework. This procedure is then extended for multiple change point detection, where, unlike existing change point procedures, the proposed method is capable of detecting abrupt changes in the presence of trend. We additionally present a time series model which combines short-term and long-term behaviours of a series for improved anomaly detection. Unlike existing methods which primarily focus on point anomalies detection (extreme outliers), our method is capable of also detecting contextual anomalies, when the data deviates from persistent patterns of the series such as seasonality. Finally, a novel multi-type relational clustering methodology is proposed. As multiple relations exist between the different entities within a network (computers, users and ports), multiple network graphs can be generated. We propose simultaneously clustering over all graphs to produce a single clustering for each entity using Non-Negative Matrix Tri-Factorisation. Through simplifications, the proposed procedure is fast and scalable for large network graphs. Additionally, this methodology is extended for graph streams. This thesis provides an assortment of tools for enterprise network monitoring with a focus on adaptability and scalability making them suitable for intrusion detection and situational awareness.Open Acces

Artificial Immune Systems - Models, algorithms and applications

Copyright © 2010 Academic Research Publishing Agency.This article has been made available through the Brunel Open Access Publishing Fund.Artificial Immune Systems (AIS) are computational paradigms that belong to the computational intelligence family and are inspired by the biological immune system. During the past decade, they have attracted a lot of interest from researchers aiming to develop immune-based models and techniques to solve complex computational or engineering problems. This work presents a survey of existing AIS models and algorithms with a focus on the last five years.This article is available through the Brunel Open Access Publishing Fun

Real-time Intrusion Detection using Multidimensional Sequence-to-Sequence Machine Learning and Adaptive Stream Processing

A network intrusion is any unauthorized activity on a computer network. There are host-based and network-based Intrusion Detection Systems (IDS\u27s), of which there are each signature-based and anomaly-based detection methods. An anomalous network behavior can be defined as an intentional violation of the expected sequence of packets. In a real-time network-based IDS, incoming packets are treated as a stream of data. A stream processor takes any stream of data or events and extracts interesting patterns on the fly. This representation allows applying statistical anomaly detection using sequence prediction algorithms as well as using a stream processor to perform signature-based intrusion detection and sequence extraction from a stream of packets. In this thesis, a Multidimensional Sequence to Multidimensional Sequence (MSeq2MSeq) encoder-decoder model is proposed to predict sequences of packets and an adaptive and functionally auto-scaling stream processor: Wisdom is proposed to process streams of packets. The proposed MSeq2MSeq model trained on legitimate traffic is able to detect Neptune Denial of Service (DoS) attacks, and Port Scan probes with 100% detection rate using the DARPA 1999 dataset. A hybrid algorithm using Particle Swarm Optimization (PSO) and Bisection algorithms was developed to optimize Complex Event Processing (CEP) rules in Wisdom . Adaptive CEP rules optimized by the above algorithm was able to detect FTP Brute Force attack, Slow Header DoS attack, and Port Scan probe with 100% detection rate while processing over 2.5 million events per second. An adaptive and functionally auto-scaling IDS was built using the MSeq2MSeq model and Wisdom stream processor to detect and prevent attacks based on anomalies and signature in real-time. The proposed IDS adapts itself to obtain best results without human intervention and utilizes available system resources in functionally auto-scaling deployment. Results show that the proposed IDS detects FTP Brute Force attack, Slow Header DoS attack, HTTP Unbearable Load King (HULK) DoS attack, SQL Injection attack, Web Brute Force attack, Cross-site scripting attack, Ares Botnet attack, and Port Scan probe with a 100% detection rate in a real-time environment simulated from the CICIDS 2017 dataset