Intrusion alert prioritisation and attack detection using post-correlation analysis

Event Correlation used to be a widely used technique for interpreting alert logs and discovering network attacks. However, due to the scale and complexity of today's networks and attacks, alert logs produced by these modern networks are much larger in volume and difficult to analyse. In this research we show that adding post-correlation methods can be used alongside correlation to significantly improve the analysis of alert logs. We proposed a new framework titled A Comprehensive System for Analysing Intrusion Alerts (ACSAnIA). The post-correlation methods include a new prioritisation metric based on anomaly detection and a novel approach to clustering events using correlation knowledge. One of the key benefits of the framework is that it significantly reduces false-positive alerts and it adds contextual information to true-positive alerts. We evaluated the post-correlation methods of ACSAnIA using data from a 2012 cyber range experiment carried out by industrial partners of the British Telecom Security Practice Team. In one scenario, our results show that false-positives were successfully reduced by 97% and in another scenario, 16%. It also showed that clustering correlated alerts aided in attack detection. The proposed framework is also being developed and integrated into a pre-existing Visual Analytic tool developed by the British Telecom SATURN Research Team for the analysis of cyber security data

Feature selection using information gain for improved structural-based alert correlation

Grouping and clustering alerts for intrusion detection based on the similarity of features is referred to as structurally base alert correlation and can discover a list of attack steps. Previous researchers selected different features and data sources manually based on their knowledge and experience, which lead to the less accurate identification of attack steps and inconsistent performance of clustering accuracy. Furthermore, the existing alert correlation systems deal with a huge amount of data that contains null values, incomplete information, and irrelevant features causing the analysis of the alerts to be tedious, time-consuming and error-prone. Therefore, this paper focuses on selecting accurate and significant features of alerts that are appropriate to represent the attack steps, thus, enhancing the structural-based alert correlation model. A two-tier feature selection method is proposed to obtain the significant features. The first tier aims at ranking the subset of features based on high information gain entropy in decreasing order. The second tier extends additional features with a better discriminative ability than the initially ranked features. Performance analysis results show the significance of the selected features in terms of the clustering accuracy using 2000 DARPA intrusion detection scenario-specific dataset

Discovering Attackers Past Behavior to Generate Online Hyper-Alerts

To support information security, organizations deploy Intrusion Detection Systems (IDS) that monitor information systems and networks, generating alerts for every suspicious behavior. However, the huge amount of alerts that an IDS triggers and their low-level representation make the alerts analysis a challenging task. In this paper, we propose a new approach based on hierarchical clustering that supports intrusion alert analysis in two main steps. First, it correlates historical alerts to identify the most common strategies attackers have used. Then, it associates upcoming alerts in real time according to the strategies discovered in the first step. The experiments were performed using a real dataset from the University of Maryland. The results showed that the proposed approach could properly identify the attack strategy patterns from historical alerts, and organize the upcoming alerts into a smaller amount of meaningful hyper-alerts

Reduction of False Positives in Intrusion Detection Based on Extreme Learning Machine with Situation Awareness

Protecting computer networks from intrusions is more important than ever for our privacy, economy, and national security. Seemingly a month does not pass without news of a major data breach involving sensitive personal identity, financial, medical, trade secret, or national security data. Democratic processes can now be potentially compromised through breaches of electronic voting systems. As ever more devices, including medical machines, automobiles, and control systems for critical infrastructure are increasingly networked, human life is also more at risk from cyber-attacks. Research into Intrusion Detection Systems (IDSs) began several decades ago and IDSs are still a mainstay of computer and network protection and continue to evolve. However, detecting previously unseen, or zero-day, threats is still an elusive goal. Many commercial IDS deployments still use misuse detection based on known threat signatures. Systems utilizing anomaly detection have shown great promise to detect previously unseen threats in academic research. But their success has been limited in large part due to the excessive number of false positives that they produce. This research demonstrates that false positives can be better minimized, while maintaining detection accuracy, by combining Extreme Learning Machine (ELM) and Hidden Markov Models (HMM) as classifiers within the context of a situation awareness framework. This research was performed using the University of New South Wales - Network Based 2015 (UNSW-NB15) data set which is more representative of contemporary cyber-attack and normal network traffic than older data sets typically used in IDS research. It is shown that this approach provides better results than either HMM or ELM alone and with a lower False Positive Rate (FPR) than other comparable approaches that also used the UNSW-NB15 data set

Intrusion Detection for Cyber-Physical Attacks in Cyber-Manufacturing System

In the vision of Cyber-Manufacturing System (CMS) , the physical components such as products, machines, and tools are connected, identifiable and can communicate via the industrial network and the Internet. This integration of connectivity enables manufacturing systems access to computational resources, such as cloud computing, digital twin, and blockchain. The connected manufacturing systems are expected to be more efficient, sustainable and cost-effective. However, the extensive connectivity also increases the vulnerability of physical components. The attack surface of a connected manufacturing environment is greatly enlarged. Machines, products and tools could be targeted by cyber-physical attacks via the network. Among many emerging security concerns, this research focuses on the intrusion detection of cyber-physical attacks. The Intrusion Detection System (IDS) is used to monitor cyber-attacks in the computer security domain. For cyber-physical attacks, however, there is limited work. Currently, the IDS cannot effectively address cyber-physical attacks in manufacturing system: (i) the IDS takes time to reveal true alarms, sometimes over months; (ii) manufacturing production life-cycle is shorter than the detection period, which can cause physical consequences such as defective products and equipment damage; (iii) the increasing complexity of network will also make the detection period even longer. This gap leaves the cyber-physical attacks in manufacturing to cause issues like over-wearing, breakage, defects or any other changes that the original design didn’t intend. A review on the history of cyber-physical attacks, and available detection methods are presented. The detection methods are reviewed in terms of intrusion detection algorithms, and alert correlation methods. The attacks are further broken down into a taxonomy covering four dimensions with over thirty attack scenarios to comprehensively study and simulate cyber-physical attacks. A new intrusion detection and correlation method was proposed to address the cyber-physical attacks in CMS. The detection method incorporates IDS software in cyber domain and machine learning analysis in physical domain. The correlation relies on a new similarity-based cyber-physical alert correlation method. Four experimental case studies were used to validate the proposed method. Each case study focused on different aspects of correlation method performance. The experiments were conducted on a security-oriented manufacturing testbed established for this research at Syracuse University. The results showed the proposed intrusion detection and alert correlation method can effectively disclose unknown attack, known attack and attack interference that causes false alarms. In case study one, the alarm reduction rate reached 99.1%, with improvement of detection accuracy from 49.6% to 100%. The case studies also proved the proposed method can mitigate false alarms, detect attacks on multiple machines, and attacks from the supply chain. This work contributes to the security domain in cyber-physical manufacturing systems, with the focus on intrusion detection. The dataset collected during the experiments has been shared with the research community. The alert correlation methodology also contributes to cyber-physical systems, such as smart grid and connected vehicles, which requires enhanced security protection in today’s connected world

Modélisation formelle des systèmes de détection d'intrusions

L’écosystème de la cybersécurité évolue en permanence en termes du nombre, de la diversité, et de la complexité des attaques. De ce fait, les outils de détection deviennent inefficaces face à certaines attaques. On distingue généralement trois types de systèmes de détection d’intrusions : détection par anomalies, détection par signatures et détection hybride. La détection par anomalies est fondée sur la caractérisation du comportement habituel du système, typiquement de manière statistique. Elle permet de détecter des attaques connues ou inconnues, mais génère aussi un très grand nombre de faux positifs. La détection par signatures permet de détecter des attaques connues en définissant des règles qui décrivent le comportement connu d’un attaquant. Cela demande une bonne connaissance du comportement de l’attaquant. La détection hybride repose sur plusieurs méthodes de détection incluant celles sus-citées. Elle présente l’avantage d’être plus précise pendant la détection. Des outils tels que Snort et Zeek offrent des langages de bas niveau pour l’expression de règles de reconnaissance d’attaques. Le nombre d’attaques potentielles étant très grand, ces bases de règles deviennent rapidement difficiles à gérer et à maintenir. De plus, l’expression de règles avec état dit stateful est particulièrement ardue pour reconnaître une séquence d’événements. Dans cette thèse, nous proposons une approche stateful basée sur les diagrammes d’état-transition algébriques (ASTDs) afin d’identifier des attaques complexes. Les ASTDs permettent de représenter de façon graphique et modulaire une spécification, ce qui facilite la maintenance et la compréhension des règles. Nous étendons la notation ASTD avec de nouvelles fonctionnalités pour représenter des attaques complexes. Ensuite, nous spécifions plusieurs attaques avec la notation étendue et exécutons les spécifications obtenues sur des flots d’événements à l’aide d’un interpréteur pour identifier des attaques. Nous évaluons aussi les performances de l’interpréteur avec des outils industriels tels que Snort et Zeek. Puis, nous réalisons un compilateur afin de générer du code exécutable à partir d’une spécification ASTD, capable d’identifier de façon efficiente les séquences d’événements.Abstract : The cybersecurity ecosystem continuously evolves with the number, the diversity, and the complexity of cyber attacks. Generally, we have three types of Intrusion Detection System (IDS) : anomaly-based detection, signature-based detection, and hybrid detection. Anomaly detection is based on the usual behavior description of the system, typically in a static manner. It enables detecting known or unknown attacks but also generating a large number of false positives. Signature based detection enables detecting known attacks by defining rules that describe known attacker’s behavior. It needs a good knowledge of attacker behavior. Hybrid detection relies on several detection methods including the previous ones. It has the advantage of being more precise during detection. Tools like Snort and Zeek offer low level languages to represent rules for detecting attacks. The number of potential attacks being large, these rule bases become quickly hard to manage and maintain. Moreover, the representation of stateful rules to recognize a sequence of events is particularly arduous. In this thesis, we propose a stateful approach based on algebraic state-transition diagrams (ASTDs) to identify complex attacks. ASTDs allow a graphical and modular representation of a specification, that facilitates maintenance and understanding of rules. We extend the ASTD notation with new features to represent complex attacks. Next, we specify several attacks with the extended notation and run the resulting specifications on event streams using an interpreter to identify attacks. We also evaluate the performance of the interpreter with industrial tools such as Snort and Zeek. Then, we build a compiler in order to generate executable code from an ASTD specification, able to efficiently identify sequences of events

Mining intrusion detection alert logs to minimise false positives & gain attack insight

Utilising Intrusion Detection System (IDS) logs in security event analysis is crucial in the process of assessing, measuring and understanding the security state of a computer network, often defined by its current exposure and resilience to network attacks. Thus, the study of understanding network attacks through event analysis is a fast growing emerging area. In comparison to its first appearance a decade ago, the complexities involved in achieving effective security event analysis have significantly increased. With such increased complexities, advances in security event analytical techniques are required in order to maintain timely mitigation and prediction of network attacks. This thesis focusses on improving the quality of analysing network event logs, particularly intrusion detection logs by exploring alternative analytical methods which overcome some of the complexities involved in security event analysis. This thesis provides four key contributions. Firstly, we explore how the quality of intrusion alert logs can be improved by eliminating the large volume of false positive alerts contained in intrusion detection logs. We investigate probabilistic alert correlation, an alternative to traditional rule based correlation approaches. We hypothesise that probabilistic alert correlation aids in discovering and learning the evolving dependencies between alerts, further revealing attack structures and information which can be vital in eliminating false positives. Our findings showed that the results support our defined hypothesis, aligning consistently with existing literature. In addition, evaluating the model using recent attack datasets (in comparison to outdated datasets used in many research studies) allowed the discovery of a new set of issues relevant to modern security event log analysis which have only been introduced and addressed in few research studies. Secondly, we propose a set of novel prioritisation metrics for the filtering of false positive intrusion alerts using knowledge gained during alert correlation. A combination of heuristic, temporal and anomaly detection measures are used to define metrics which capture characteristics identifiable in common attacks including denial-of-service attacks and worm propagations. The most relevant of the novel metrics, Outmet is based on the well known Local Outlier Factor algorithm. Our findings showed that with a slight trade-off of sensitivity (i.e. true positives performance), outmet reduces false positives significantly. In comparison to prior state-of-the-art, our findings show that it performs more efficiently given a variation of attack scenarios. Thirdly, we extend a well known real-time clustering algorithm, CluStream in order to support the categorisation of attack patterns represented as graph like structures. Our motive behind attack pattern categorisation is to provide automated methods for capturing consistent behavioural patterns across a given class of attacks. To our knowledge, this is a novel approach to intrusion alert analysis. The extension of CluStream resulted is a novel light weight real-time clustering algorithm for graph structures. Our findings are new and complement existing literature. We discovered that in certain case studies, repetitive attack behaviour could be mined. Such a discovery could facilitate the prediction of future attacks. Finally, we acknowledge that due to the intelligence and stealth involved in modern network attacks, automated analytical approaches alone may not suffice in making sense of intrusion detection logs. Thus, we explore visualisation and interactive methods for effective visual analysis which if combined with the automated approaches proposed, would improve the overall results of the analysis. The result of this is a visual analytic framework, integrated and tested in a commercial Cyber Security Event Analysis Software System distributed by British Telecom