Unsupervised Intrusion Detection with Cross-Domain Artificial Intelligence Methods

Cybercrime is a major concern for corporations, business owners, governments and citizens, and it continues to grow in spite of increasing investments in security and fraud prevention. The main challenges in this research field are: being able to detect unknown attacks, and reducing the false positive ratio. The aim of this research work was to target both problems by leveraging four artificial intelligence techniques. The first technique is a novel unsupervised learning method based on skip-gram modeling. It was designed, developed and tested against a public dataset with popular intrusion patterns. A high accuracy and a low false positive rate were achieved without prior knowledge of attack patterns. The second technique is a novel unsupervised learning method based on topic modeling. It was applied to three related domains (network attacks, payments fraud, IoT malware traffic). A high accuracy was achieved in the three scenarios, even though the malicious activity significantly differs from one domain to the other. The third technique is a novel unsupervised learning method based on deep autoencoders, with feature selection performed by a supervised method, random forest. Obtained results showed that this technique can outperform other similar techniques. The fourth technique is based on an MLP neural network, and is applied to alert reduction in fraud prevention. This method automates manual reviews previously done by human experts, without significantly impacting accuracy

Anomaly detection via high-dimensional data analysis on web access data.

Suen, Ho Yan.Thesis (M.Phil.)--Chinese University of Hong Kong, 2009.Includes bibliographical references (leaves 99-104).Abstract also in Chinese.Abstract --- p.iAcknowledgement --- p.ivChapter 1 --- Introduction --- p.1Chapter 1.1 --- Motivation --- p.1Chapter 1.2 --- Organization --- p.4Chapter 2 --- Literature Review --- p.6Chapter 2.1 --- Related Works --- p.6Chapter 2.2 --- Background Study --- p.7Chapter 2.2.1 --- World Wide Web --- p.7Chapter 2.2.2 --- Distributed Denial of Service Attack --- p.11Chapter 2.2.3 --- Tools for Dimension Reduction --- p.13Chapter 2.2.4 --- Tools for Anomaly Detection --- p.20Chapter 2.2.5 --- Receiver operating characteristics (ROC) Analysis --- p.22Chapter 3 --- System Design --- p.25Chapter 3.1 --- Methodology --- p.25Chapter 3.2 --- System Overview --- p.27Chapter 3.3 --- Reference Profile Construction --- p.31Chapter 3.4 --- Real-time Anomaly Detection and Response --- p.32Chapter 3.5 --- Chapter Summary --- p.34Chapter 4 --- Reference Profile Construction --- p.35Chapter 4.1 --- Web Access Logs Collection --- p.35Chapter 4.2 --- Data Preparation --- p.37Chapter 4.3 --- Feature Extraction and Embedding Engine (FEE Engine) --- p.40Chapter 4.3.1 --- Sub-Sequence Extraction --- p.42Chapter 4.3.2 --- Hash Function on Sub-sequences (optional) --- p.45Chapter 4.3.3 --- Feature Vector Construction --- p.46Chapter 4.3.4 --- Diffusion Wavelets Embedding --- p.47Chapter 4.3.5 --- Numerical Example of Feature Set Reduction --- p.49Chapter 4.3.6 --- Reference Profile and Further Use of FEE Engine --- p.50Chapter 4.4 --- Chapter Summary --- p.50Chapter 5 --- Real-time Anomaly Detection and Response --- p.52Chapter 5.1 --- Session Filtering and Data Preparation --- p.54Chapter 5.2 --- Feature Extraction and Embedding --- p.54Chapter 5.3 --- Distance-based Outlier Scores Calculation --- p.55Chapter 5.4 --- Anomaly Detection and Response --- p.56Chapter 5.4.1 --- Length-Based Anomaly Detection Modules --- p.56Chapter 5.4.2 --- Characteristics of Anomaly Detection Modules --- p.59Chapter 5.4.3 --- Dynamic Threshold Adaptation --- p.60Chapter 5.5 --- Chapter Summary --- p.63Chapter 6 --- Experimental Results --- p.65Chapter 6.1 --- Experiment Datasets --- p.65Chapter 6.1.1 --- Normal Web Access Logs --- p.66Chapter 6.1.2 --- Attack Data Generation --- p.68Chapter 6.2 --- ROC Curve Construction --- p.70Chapter 6.3 --- System Parameters Selection --- p.71Chapter 6.4 --- Performance of Anomaly Detection --- p.82Chapter 6.4.1 --- Performance Analysis --- p.85Chapter 6.4.2 --- Performance in defending DDoS attacks --- p.87Chapter 6.5 --- Computation Requirement --- p.91Chapter 6.6 --- Chapter Summary --- p.95Chapter 7 --- Conclusion and Future Work --- p.96Bibliography --- p.9