A monitoring and threat detection system using stream processing as a virtual function for big data

The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast realtime threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.A detecção tardia de ameaças de segurança causa um significante aumento no risco de danos irreparáveis, impossibilitando qualquer tentativa de defesa. Como consequência, a detecção rápida de ameaças em tempo real é essencial para a administração de segurança. Além disso, A tecnologia de virtualização de funções de rede (Network Function Virtualization - NFV) oferece novas oportunidades para soluções de segurança eficazes e de baixo custo. Propomos um sistema de detecção de ameaças rápido e eficiente, baseado em algoritmos de processamento de fluxo e de aprendizado de máquina. As principais contribuições deste trabalho são: i) um novo sistema de monitoramento e detecção de ameaças baseado no processamento de fluxo; ii) dois conjuntos de dados, o primeiro ´e um conjunto de dados sintético de segurança contendo tráfego suspeito e malicioso, e o segundo corresponde a uma semana de tráfego real de um operador de telecomunicações no Rio de Janeiro, Brasil; iii) um algoritmo de pré-processamento de dados composto por um algoritmo de normalização e um algoritmo para seleção rápida de características com base na correlação entre variáveis; iv) uma função de rede virtualizada em uma plataforma de código aberto para fornecer um serviço de detecção de ameaças em tempo real; v) posicionamento quase perfeito de sensores através de uma heurística proposta para posicionamento estratégico de sensores na infraestrutura de rede, com um número mínimo de sensores; e, finalmente, vi) um algoritmo guloso que aloca sob demanda uma sequencia de funções de rede virtual

Real-time big data processing for anomaly detection : a survey

The advent of connected devices and omnipresence of Internet have paved way for intruders to attack networks, which leads to cyber-attack, financial loss, information theft in healthcare, and cyber war. Hence, network security analytics has become an important area of concern and has gained intensive attention among researchers, off late, specifically in the domain of anomaly detection in network, which is considered crucial for network security. However, preliminary investigations have revealed that the existing approaches to detect anomalies in network are not effective enough, particularly to detect them in real time. The reason for the inefficacy of current approaches is mainly due the amassment of massive volumes of data though the connected devices. Therefore, it is crucial to propose a framework that effectively handles real time big data processing and detect anomalies in networks. In this regard, this paper attempts to address the issue of detecting anomalies in real time. Respectively, this paper has surveyed the state-of-the-art real-time big data processing technologies related to anomaly detection and the vital characteristics of associated machine learning algorithms. This paper begins with the explanation of essential contexts and taxonomy of real-time big data processing, anomalous detection, and machine learning algorithms, followed by the review of big data processing technologies. Finally, the identified research challenges of real-time big data processing in anomaly detection are discussed. © 2018 Elsevier Lt

A Novel Approach for Detection of DoS / DDoS Attack in Network Environment using Ensemble Machine Learning Model

One of the most  serious threat to network security is Denial of service (DOS) attacks. Internet and computer networks are now important parts of our businesses and daily lives. Malicious actions have become more common as our reliance on computers and communication networks has grown. Network threats are a big problem in the way people communicate today. To make sure that the networks work well and that users' information is safe, the network data must be watched and analysed to find malicious activities and attacks. Flooding may be the simplest DDoS assault. Computer networks and services are vulnerable to DoS and DDoS attacks. These assaults flood target systems with malicious traffic, making them unreachable to genuine users. The work aims to enhance the resilience of network infrastructures against these attacks and ensure uninterrupted service delivery. This research develops and evaluates enhanced DoS/DDoS detection methods. DoS attacks usually stop or slow down legal computer or network use. Denial-of-service (DoS) attacks prevent genuine users from accessing and using information systems and resources. The OSI model's layers make up the computer network. Different types of DDoS strikes target different layers. The Network Layer can be broken by using ICMP Floods or Smurf Attacks. The Transport layer can be attacked using UDP Floods, TCP Connection Exhaustion, and SYN Floods. HTTP-encrypted attacks can be used to get through to the application layer. DoS/DDoS attacks are malicious attacks. Protect network data from harm. Computer network services are increasingly threatened by DoS/DDoS attacks. Machine learning may detect prior DoS/DDoS attacks. DoS/DDoS attacks proliferate online and via social media. Network security is IT's top priority. DoS and DDoS assaults include ICMP, UDP, and the more prevalent TCP flood attacks. These strikes must be identified and stopped immediately. In this work, a stacking ensemble method is suggested for detecting DoS/DDoS attacks so that our networked data doesn't get any worse. This paper used a method called "Ensemble of classifiers," in which each class uses a different way to learn. In proposed  methodology Experiment#1 , I used the Home Wifi Network Traffic Collected and generated own Dataset named it as MywifiNetwork.csv, whereas in proposed methodology Experiment#2, I used the kaggle repository “NSL-KDD benchmark dataset” to perform experiments in order to find detection accuracy of dos attack detection using python language in jupyter notebook. The system detects attack-type or legitimate-type of network traffic during detection ML classification methods are used to compare how well the suggested system works. The results show that when the ensembled stacking learning model is used, 99% of the time it is able to find the problem. In proposed methodology two Experiments are implemented for comparing detection accuracy with the existing techniques. Compared to other measuring methods, we get a big step forward in finding attacks. So, our model gives a lot of faith in securing these networks. This paper will analyse the behaviour of network traffics

DDoS Attack Detection in WSN using Modified Invasive Weed Optimization with Extreme Learning Machine

Wireless sensor networks (WSN) are the wide-spread methodology for its distribution of the vast amount of devoted sensor nodes (SNs) that is employed for sensing the atmosphere and gather information. The gathered information was transmitted to the sink nodes via intermediate nodes. Meanwhile, the SN data are prone to the internet, and they are vulnerable to diverse security risks, involving distributed denial of service (DDoS) outbreaks that might interrupt network operation and compromises data integrity. In recent times, developed machine learning (ML) approaches can be applied for the discovery of DDoS attacks and accomplish security in WSN. To achieve this, this study presents a modified invasive weed optimization with extreme learning machine (MIWO-ELM) model for DDoS outbreak recognition in the WSN atmosphere. In the presented MIWO-ELM technique, an initial stage of data pre-processing is conducted. The ELM model can be applied for precise DDoS attack detection and classification process. At last, the MIWO method can be exploited for the parameter tuning of the ELM model which leads to improved performance of the classification. The experimental analysis of the MIWO-ELM method takes place using WSN dataset. The comprehensive simulation outputs show the remarkable performance of the MIWO-ELM method compared to other recent approaches

Near real-time network analysis for the identification of malicious activity

The evolution of technology and the increasing connectivity between devices lead to an increased risk of cyberattacks. Reliable protection systems, such as Intrusion Detection System (IDS) and Intrusion Prevention System (IPS), are essential to try to prevent, detect and counter most of the attacks. However, the increased creativity and type of attacks raise the need for more resources and processing power for the protection systems which, in turn, requires horizontal scalability to keep up with the massive companies’ network infrastructure and with the complexity of attacks. Technologies like machine learning, show promising results and can be of added value in the detection and prevention of attacks in near real-time. But good algorithms and tools are not enough. They require reliable and solid datasets to be able to effectively train the protection systems. The development of a good dataset requires horizontal-scalable, robust, modular and faulttolerant systems so that the analysis may be done in near real-time. This work describes an architecture design for horizontal-scaling capture, storage and analyses, able to collect packets from multiple sources and analyse them in a parallel fashion. The system depends on multiple modular nodes with specific roles to support different algorithms and tools.A evolução da tecnologia e o aumento da conectividade entre dispositivos, levam a um aumento do risco de ciberataques. Os sistemas de deteção de intrusão são essenciais para tentar prevenir, detetar e conter a maioria dos ataques. No entanto, o aumento da criatividade e do tipo de ataques aumenta a necessidade dos sistemas de proteção possuírem cada vez mais recursos e poder computacional. Por sua vez, requerem escalabilidade horizontal para acompanhar a massiva infraestrutura de rede das empresas e a complexidade dos ataques. Tecnologias como machine learning apresentam resultados promissores e podem ser de grande valor na deteção e prevenção de ataques em tempo útil. No entanto, a utilização dos algoritmos e ferramentas requer sempre um conjunto de dados sólidos e confiáveis para treinar os sistemas de proteção de maneira eficaz. A implementação de um bom conjunto de dados requer sistemas horizontalmente escaláveis, robustos, modulares e tolerantes a falhas para que a análise seja rápida e rigorosa. Este trabalho descreve a arquitetura de um sistema de captura, armazenamento e análise, capaz de capturar pacotes de múltiplas fontes e analisá-los de forma paralela. O sistema depende de vários nós modulares com funções específicas para oferecer suporte a diferentes algoritmos e ferramentas

Detection of DDoS Attacks in OpenStack-based Private Cloud Using Apache Spark, Journal of Telecommunications and Information Technology, 2020, nr 4

Security is a critical concern for cloud service providers. Distributed denial of service (DDoS) attacks are the most frequent of all cloud security threats, and the consequences of damage caused by DDoS are very serious. Thus, the design of an efficient DDoS detection system plays an important role in monitoring suspicious activity in the cloud. Real-time detection mechanisms operating in cloud environments and relying on machine learning algorithms and distributed processing are an important research issue. In this work, we propose a real-time detection of DDoS attacks using machine learning classifiers on a distributed processing platform. We evaluate the DDoS detection mechanism in an OpenStack-based cloud testbed using the Apache Spark framework. We compare the classification performance using benchmark and real-time cloud datasets. Results of the experiments reveal that the random forest method offers better classifier accuracy. Furthermore, we demonstrate the effectiveness of the proposed distributed approach in terms of training and detection tim

Advanced threat hunting over software-defined networks in smart cities

The emergence of Software-Defined Networking (SDN) has brought along a wave of new technologies and developments in the field of networking with hopes of dealing with network resources more efficiently and providing a foundation of programmability. SDN allows for both flexibility and adaptability by separating the control and data planes in a network environment by virtualizing network hardware. Threat hunting is a technique that allows for the detection of advanced network threats through forensic analysis. We present an advanced threat hunting model by combining the SDN infrastructure with threat hunting techniques and machine learning models aiming to intelligently handle advanced network threats such as lateral movement. We found that our approach outperforms current threat hunting models in vital areas such as the detection to mitigation time. Our results show that we are able to detect advanced threats with 93.4% accuracy and begin mitigation within 10 seconds of detection

Network anomalies detection via event analysis and correlation by a smart system

The multidisciplinary of contemporary societies compel us to look at Information Technology (IT) systems as one of the most significant grants that we can remember. However, its increase implies a mandatory security force for users, a force in the form of effective and robust tools to combat cybercrime to which users, individual or collective, are ex-posed almost daily. Monitoring and detection of this kind of problem must be ensured in real-time, allowing companies to intervene fruitfully, quickly and in unison. The proposed framework is based on an organic symbiosis between credible, affordable, and effective open-source tools for data analysis, relying on Security Information and Event Management (SIEM), Big Data and Machine Learning (ML) techniques commonly applied for the development of real-time monitoring systems. Dissecting this framework, it is composed of a system based on SIEM methodology that provides monitoring of data in real-time and simultaneously saves the information, to assist forensic investigation teams. Secondly, the application of the Big Data concept is effective in manipulating and organising the flow of data. Lastly, the use of ML techniques that help create mechanisms to detect possible attacks or anomalies on the network. This framework is intended to provide a real-time analysis application in the institution ISCTE – Instituto Universitário de Lisboa (Iscte), offering a more complete, efficient, and secure monitoring of the data from the different devices comprising the network.A multidisciplinaridade das sociedades contemporâneas obriga-nos a perspetivar os sistemas informáticos como uma das maiores dádivas de que há memória. Todavia o seu incremento implica uma mandatária força de segurança para utilizadores, força essa em forma de ferramentas eficazes e robustas no combate ao cibercrime a que os utilizadores, individuais ou coletivos, são sujeitos quase diariamente. A monitorização e deteção deste tipo de problemas tem de ser assegurada em tempo real, permitindo assim, às empresas intervenções frutuosas, rápidas e em uníssono. A framework proposta é alicerçada numa simbiose orgânica entre ferramentas open source credíveis, acessíveis pecuniariamente e eficazes na monitorização de dados, recorrendo a um sistema baseado em técnicas de Security Information and Event Management (SIEM), Big Data e Machine Learning (ML) comumente aplicadas para a criação de sistemas de monitorização em tempo real. Dissecando esta framework, é composta pela metodologia SIEM que possibilita a monitorização de dados em tempo real e em simultâneo guardar a informação, com o objetivo de auxiliar as equipas de investigação forense. Em segundo lugar, a aplicação do conceito Big Data eficaz na manipulação e organização do fluxo dos dados. Por último, o uso de técnicas de ML que ajudam a criação de mecanismos de deteção de possíveis ataques ou anomalias na rede. Esta framework tem como objetivo uma aplicação de análise em tempo real na instituição ISCTE – Instituto Universitário de Lisboa (Iscte), apresentando uma monitorização mais completa, eficiente e segura dos dados dos diversos dispositivos presentes na mesma