Applications of Machine Learning to Threat Intelligence, Intrusion Detection and Malware

Artificial Intelligence (AI) and Machine Learning (ML) are emerging technologies with applications to many fields. This paper is a survey of use cases of ML for threat intelligence, intrusion detection, and malware analysis and detection. Threat intelligence, especially attack attribution, can benefit from the use of ML classification. False positives from rule-based intrusion detection systems can be reduced with the use of ML models. Malware analysis and classification can be made easier by developing ML frameworks to distill similarities between the malicious programs. Adversarial machine learning will also be discussed, because while ML can be used to solve problems or reduce analyst workload, it also introduces new attack surfaces

SQL Injection Detection Using Machine Learning Techniques and Multiple Data Sources

SQL Injection continues to be one of the most damaging security exploits in terms of personal information exposure as well as monetary loss. Injection attacks are the number one vulnerability in the most recent OWASP Top 10 report, and the number of these attacks continues to increase. Traditional defense strategies often involve static, signature-based IDS (Intrusion Detection System) rules which are mostly effective only against previously observed attacks but not unknown, or zero-day, attacks. Much current research involves the use of machine learning techniques, which are able to detect unknown attacks, but depending on the algorithm can be costly in terms of performance. In addition, most current intrusion detection strategies involve collection of traffic coming into the web application either from a network device or from the web application host, while other strategies collect data from the database server logs. In this project, we are collecting traffic from two points: the web application host, and a Datiphy appliance node located between the webapp host and the associated MySQL database server. In our analysis of these two datasets, and another dataset that is correlated between the two, we have been able to demonstrate that accuracy obtained with the correlated dataset using algorithms such as rule-based and decision tree are nearly the same as those with a neural network algorithm, but with greatly improved performance

TSE-IDS: A Two-Stage Classifier Ensemble for Intelligent Anomaly-based Intrusion Detection System

Intrusion detection systems (IDS) play a pivotal role in computer security by discovering and repealing malicious activities in computer networks. Anomaly-based IDS, in particular, rely on classification models trained using historical data to discover such malicious activities. In this paper, an improved IDS based on hybrid feature selection and two-level classifier ensembles is proposed. An hybrid feature selection technique comprising three methods, i.e. particle swarm optimization, ant colony algorithm, and genetic algorithm, is utilized to reduce the feature size of the training datasets (NSL-KDD and UNSW-NB15 are considered in this paper). Features are selected based on the classification performance of a reduced error pruning tree (REPT) classifier. Then, a two-level classifier ensembles based on two meta learners, i.e., rotation forest and bagging, is proposed. On the NSL-KDD dataset, the proposed classifier shows 85.8% accuracy, 86.8% sensitivity, and 88.0% detection rate, which remarkably outperform other classification techniques recently proposed in the literature. Results regarding the UNSW-NB15 dataset also improve the ones achieved by several state of the art techniques. Finally, to verify the results, a two-step statistical significance test is conducted. This is not usually considered by IDS research thus far and, therefore, adds value to the experimental results achieved by the proposed classifier

Outlier detection techniques for wireless sensor networks: A survey

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree

Detection of Denial of Service Attacks against Domain Name System Using Neural Networks

Along with the explosive growth of the Internet, the demand for efficient and secure Internet Infrastructure has been increasing. For the entire chain of Internet connectivity the Domain Name System (DNS) provides name to address mapping services. Hackers exploit this fact to damage different parts of Internet. In order to prevent this system from different types of attacks, we need to prepare a classification of possible security threats against DNS. This dissertation focuses on Denial of Service (DoS) attacks as the major security issue during last years, and gives an overview of techniques used to discover and analyze them. The process of detection and classification of DoS against DNS has been presented in two phases in our model. The proposed system architecture consists of a statistical pre-processor and a machine learning engine. The first step in our work was to generate the DNS traffic in normal and attack situations for using as the input of our intrusion detection system (IDS). With the prior knowledge of DoS attacks against DNS, we used a network simulator to model DNS traffic with high variability. Therefore, the difficulty of creating different scenarios of attacks in a real environment has been decreased. The pre-processor, processes the collected data statistically and derives the final variable values. These parameters are the inputs of the detector engine. In the current research for our machine learning engine, we aimed to find the optimum machine learning algorithm to be used as an IDS. The performance of our system was measured in terms of detection rate, accuracy, and false alarm rate. The results indicated that the three layered back propagation neural network with a 3-7-3 structure provides a detection rate of 99.55% for direct DoS attacks and 97.82% for amplification DoS attacks. It can give us 99% accuracy and an acceptable false alarm rate of 0.28% comparing to other types of classifiers

Development of a Reference Design for Intrusion Detection Using Neural Networks for a Smart Inverter

The purpose of this thesis is to develop a reference design for a base level implementation of an intrusion detection module using artificial neural networks that is deployed onto an inverter and runs on live data for cybersecurity purposes, leveraging the latest deep learning algorithms and tools. Cybersecurity in the smart grid industry focuses on maintaining optimal standards of security in the system and a key component of this is being able to detect cyberattacks. Although researchers and engineers aim to design such devices with embedded security, attacks can and do still occur. The foundation for eventually mitigating these attacks and achieving more robust security is to identify them reliably. Thus, a high-fidelity intrusion detection system (IDS) capable of identifying a variety of attacks must be implemented. This thesis provides an implementation of a behavior-based intrusion detection system that uses a recurrent artificial neural network deployed on hardware to detect cyberattacks in real time. Leveraging the growing power of artificial intelligence, the strength of this approach is that given enough data, it is capable of learning to identify highly complex patterns in the data that may even go undetected by humans. By intelligently identifying malicious activity at the fundamental behavior level, the IDS remains robust against new methods of attack. This work details the process of collecting and simulating data, selecting the particular algorithm, training the neural network, deploying the neural network onto hardware, and then being able to easily update the deployed model with a newly trained one. The full system is designed with a focus on modularity, such that it can be easily adapted to perform well on different use cases, different hardware, and fulfill changing requirements. The neural network behavior-based IDS is found to be a very powerful method capable of learning highly complex patterns and identifying intrusion from different types of attacks using a single unified algorithm, achieving up to 98% detection accuracy in distinguishing between normal and anomalous behavior. Due to the ubiquitous nature of this approach, the pipeline developed here can be applied in the future to build in more and more sophisticated detection abilities depending on the desired use case. The intrusion detection module is implemented in an ARM processor that exists at the communication layer of the inverter. There are four main components described in this thesis that explain the process of deploying an artificial neural network intrusion detection algorithm onto the inverter: 1) monitoring and collecting data through a front-end web based graphical user interface that interacts with a Digital Signal Processor that is connected to power-electronics, 2) simulating various malicious datasets based on attack vectors that violate the Confidentiality-Integrity-Availability security model, 3) training and testing the neural network to ensure that it successfully identifies normal behavior and malicious behavior with a high degree of accuracy, and lastly 4) deploying the machine learning algorithm onto the hardware and having it successfully classify the behavior as normal or malicious with the data feeding into the model running in real time. The results from the experimental setup will be analyzed, a conclusion will be made based upon the work, and lastly discussions of future work and optimizations will be discussed

Cyber-attack detection in network traffic using machine learning

Rapid shifting by government sectors and companies to provide their services and products over the internet, has immensely increased internet usage by individuals. Through extranets to network services or corporate networks used for personal purposes, computer hackers can lead to financial losses and manpower/time consumption. Therefore, it is vital to take all necessary measures to minimize losses by detecting attacks preemptively. Due to learning algorithms in cyberspace security challenges, deep learning-based cyber defense has lately become a hot topic. Penetration testing, malware categorization and identification, spam filtering, and spoofing detection are just a few of the key concerns in cyber defense that were tackled using Machine Learning (ML) approaches (Somme, 2020). Result, effective adaptive approaches, such as machine learning approaches could result in increased response times, reduced probability of false alerts, as well as cheaper computing and communication expenses. Our primary point is to demonstrate that the problem of detecting malware is distinct from other technologies, making it far more difficult for the access control group to properly use machine learning