AI Solutions for MDS: Artificial Intelligence Techniques for Misuse Detection and Localisation in Telecommunication Environments

This report considers the application of Articial Intelligence (AI) techniques to the problem of misuse detection and misuse localisation within telecommunications environments. A broad survey of techniques is provided, that covers inter alia rule based systems, model-based systems, case based reasoning, pattern matching, clustering and feature extraction, articial neural networks, genetic algorithms, arti cial immune systems, agent based systems, data mining and a variety of hybrid approaches. The report then considers the central issue of event correlation, that is at the heart of many misuse detection and localisation systems. The notion of being able to infer misuse by the correlation of individual temporally distributed events within a multiple data stream environment is explored, and a range of techniques, covering model based approaches, `programmed' AI and machine learning paradigms. It is found that, in general, correlation is best achieved via rule based approaches, but that these suffer from a number of drawbacks, such as the difculty of developing and maintaining an appropriate knowledge base, and the lack of ability to generalise from known misuses to new unseen misuses. Two distinct approaches are evident. One attempts to encode knowledge of known misuses, typically within rules, and use this to screen events. This approach cannot generally detect misuses for which it has not been programmed, i.e. it is prone to issuing false negatives. The other attempts to `learn' the features of event patterns that constitute normal behaviour, and, by observing patterns that do not match expected behaviour, detect when a misuse has occurred. This approach is prone to issuing false positives, i.e. inferring misuse from innocent patterns of behaviour that the system was not trained to recognise. Contemporary approaches are seen to favour hybridisation, often combining detection or localisation mechanisms for both abnormal and normal behaviour, the former to capture known cases of misuse, the latter to capture unknown cases. In some systems, these mechanisms even work together to update each other to increase detection rates and lower false positive rates. It is concluded that hybridisation offers the most promising future direction, but that a rule or state based component is likely to remain, being the most natural approach to the correlation of complex events. The challenge, then, is to mitigate the weaknesses of canonical programmed systems such that learning, generalisation and adaptation are more readily facilitated

Towards Configured Intrusion Detection Systems

This paper studies the challenges in the current intrusion detection system and comparatively analyzes the active and passive response systems. The paper studies the existing IDS and their usefulness in detecting and preventing attacks in any type of network and control traffic with the performance of the system to be improved. The study also evaluates the emerging avenues in Intrusion Detection System and explores the possible future avenues in intrusion detection scheme. It is observed that the detection-based systems have started to gain popularity in the IT security domain. The paper highlights the need to implement an appropriately configured IDS since an optimally configured IDS deters hackers, thus, reducing the need for investigation by security experts for security violations

Unsupervised Intrusion Detection with Cross-Domain Artificial Intelligence Methods

Cybercrime is a major concern for corporations, business owners, governments and citizens, and it continues to grow in spite of increasing investments in security and fraud prevention. The main challenges in this research field are: being able to detect unknown attacks, and reducing the false positive ratio. The aim of this research work was to target both problems by leveraging four artificial intelligence techniques. The first technique is a novel unsupervised learning method based on skip-gram modeling. It was designed, developed and tested against a public dataset with popular intrusion patterns. A high accuracy and a low false positive rate were achieved without prior knowledge of attack patterns. The second technique is a novel unsupervised learning method based on topic modeling. It was applied to three related domains (network attacks, payments fraud, IoT malware traffic). A high accuracy was achieved in the three scenarios, even though the malicious activity significantly differs from one domain to the other. The third technique is a novel unsupervised learning method based on deep autoencoders, with feature selection performed by a supervised method, random forest. Obtained results showed that this technique can outperform other similar techniques. The fourth technique is based on an MLP neural network, and is applied to alert reduction in fraud prevention. This method automates manual reviews previously done by human experts, without significantly impacting accuracy

Impregnable Defence Architecture using Dynamic Correlation-based Graded Intrusion Detection System for Cloud

Data security and privacy are perennial concerns related to cloud migration, whether it is about applications, business or customers. In this paper, novel security architecture for the cloud environment designed with intrusion detection and prevention system (IDPS) components as a graded multi-tier defense framework. It is a defensive formation of collaborative IDPS components with dynamically revolving alert data placed in multiple tiers of virtual local area networks (VLANs). The model has two significant contributions for impregnable protection, one is to reduce alert generation delay by dynamic correlation and the second is to support the supervised learning of malware detection through system call analysis. The defence formation facilitates malware detection with linear support vector machine- stochastic gradient descent (SVM-SGD) statistical algorithm. It requires little computational effort to counter the distributed, co-ordinated attacks efficiently. The framework design, then, takes distributed port scan attack as an example for assessing the efficiency in terms of reduction in alert generation delay, the number of false positives and learning time through comparison with existing techniques is discussed

Flow-oriented anomaly-based detection of denial of service attacks with flow-control-assisted mitigation

Flooding-based distributed denial-of-service (DDoS) attacks present a serious and major threat to the targeted enterprises and hosts. Current protection technologies are still largely inadequate in mitigating such attacks, especially if they are large-scale. In this doctoral dissertation, the Computer Network Management and Control System (CNMCS) is proposed and investigated; it consists of the Flow-based Network Intrusion Detection System (FNIDS), the Flow-based Congestion Control (FCC) System, and the Server Bandwidth Management System (SBMS). These components form a composite defense system intended to protect against DDoS flooding attacks. The system as a whole adopts a flow-oriented and anomaly-based approach to the detection of these attacks, as well as a control-theoretic approach to adjust the flow rate of every link to sustain the high priority flow-rates at their desired level. The results showed that the misclassification rates of FNIDS are low, less than 0.1%, for the investigated DDOS attacks, while the fine-grained service differentiation and resource isolation provided within the FCC comprise a novel and powerful built-in protection mechanism that helps mitigate DDoS attacks

On Holistic Multi-Step Cyberattack Detection via a Graph-based Correlation Approach

While digitization of distribution grids through information and communications technology brings numerous benefits, it also increases the grid's vulnerability to serious cyber attacks. Unlike conventional systems, attacks on many industrial control systems such as power grids often occur in multiple stages, with the attacker taking several steps at once to achieve its goal. Detection mechanisms with situational awareness are needed to detect orchestrated attack steps as part of a coherent attack campaign. To provide a foundation for detection and prevention of such attacks, this paper addresses the detection of multi-stage cyber attacks with the aid of a graph-based cyber intelligence database and alert correlation approach. Specifically, we propose an approach to detect multi-stage attacks by leveraging heterogeneous data to form a knowledge base and employ a model-based correlation approach on the generated alerts to identify multi-stage cyber attack sequences taking place in the network. We investigate the detection quality of the proposed approach by using a case study of a multi-stage cyber attack campaign in a future-orientated power grid pilot.Comment: IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) 202

A lightweight intrusion alert fusion system

In this paper, we present some practical experience on implementing an alert fusion mechanism from our project. After investigation on most of the existing alert fusion systems, we found the current body of work alternatively weighed down in the mire of insecure design or rarely deployed because of their complexity. As confirmed by our experimental analysis, unsuitable mechanisms could easily be submerged by an abundance of useless alerts. Even with the use of methods that achieve a high fusion rate and low false positives, attack is also possible. To find the solution, we carried out analysis on a series of alerts generated by well-known datasets as well as realistic alerts from the Australian Honey-Pot. One important finding is that one alert has more than an 85% chance of being fused in the following 5 alerts. Of particular importance is our design of a novel lightweight Cache-based Alert Fusion Scheme, called CAFS. CAFS has the capacity to not only reduce the quantity of useless alerts generated by IDS (Intrusion Detection System), but also enhance the accuracy of alerts, therefore greatly reducing the cost of fusion processing. We also present reasonable and practical specifications for the target-oriented fusion policy that provides a quality guarantee on alert fusion, and as a result seamlessly satisfies the process of successive correlation. Our experimental results showed that the CAFS easily attained the desired level of survivable, inescapable alert fusion design. Furthermore, as a lightweight scheme, CAFS can easily be deployed and excel in a large amount of alert fusions, which go towards improving the usability of system resources. To the best of our knowledge, our work is a novel exploration in addressing these problems from a survivable, inescapable and deployable point of view