Real-time big data processing for anomaly detection : a survey

The advent of connected devices and omnipresence of Internet have paved way for intruders to attack networks, which leads to cyber-attack, financial loss, information theft in healthcare, and cyber war. Hence, network security analytics has become an important area of concern and has gained intensive attention among researchers, off late, specifically in the domain of anomaly detection in network, which is considered crucial for network security. However, preliminary investigations have revealed that the existing approaches to detect anomalies in network are not effective enough, particularly to detect them in real time. The reason for the inefficacy of current approaches is mainly due the amassment of massive volumes of data though the connected devices. Therefore, it is crucial to propose a framework that effectively handles real time big data processing and detect anomalies in networks. In this regard, this paper attempts to address the issue of detecting anomalies in real time. Respectively, this paper has surveyed the state-of-the-art real-time big data processing technologies related to anomaly detection and the vital characteristics of associated machine learning algorithms. This paper begins with the explanation of essential contexts and taxonomy of real-time big data processing, anomalous detection, and machine learning algorithms, followed by the review of big data processing technologies. Finally, the identified research challenges of real-time big data processing in anomaly detection are discussed. © 2018 Elsevier Lt

Mobile Anomaly Detection Based on Improved Self-Organizing Maps

Anomaly detection has always been the focus of researchers and especially, the developments of mobile devices raise new challenges of anomaly detection. For example, mobile devices can keep connection with Internet and they are rarely turned off even at night. This means mobile devices can attack nodes or be attacked at night without being perceived by users and they have different characteristics from Internet behaviors. The introduction of data mining has made leaps forward in this field. Self-organizing maps, one of famous clustering algorithms, are affected by initial weight vectors and the clustering result is unstable. The optimal method of selecting initial clustering centers is transplanted from K-means to SOM. To evaluate the performance of improved SOM, we utilize diverse datasets and KDD Cup99 dataset to compare it with traditional one. The experimental results show that improved SOM can get higher accuracy rate for universal datasets. As for KDD Cup99 dataset, it achieves higher recall rate and precision rate

Addressing High False Positive Rates of DDoS Attack Detection Methods

Distributed denial of service (DDoS) attack detection methods based on the clustering method are ineffective in detecting attacks correctly. Service interruptions caused by DDoS attacks impose concerns for IT leaders and their organizations, leading to financial damages. Grounded in the cross industry standard process for data mining framework, the purpose of this ex post facto study was to examine whether adding the filter and wrapper methods prior to the clustering method is effective in terms of lowering false positive rates of DDoS attack detection methods. The population of this study was 225,745 network traffic data records of the CICIDS2017 network traffic dataset. The 10-fold cross validation method was applied to identify effective DDoS attack detection methods. The results of the 10-fold cross validation method showed that in some instances, addition of the filter and wrapper methods prior to the clustering method was effective in terms of lowering false positive rates of DDoS attack detection methods; in some instances, it was not. A recommendation to IT leaders is to deploy the effective DDoS attack detection method that produced the lowest false positive rate of 0.013 in detecting attacks outside of demilitarized zones to identify attacks directly from the Internet. Implications for positive social change is potentially in enabling organizations to protect their systems and provide uninterrupted services to their communities with reduced financial damages