Data-Driven Anomaly Detection in Industrial Networks

Since the conception of the first Programmable Logic Controllers (PLCs) in the 1960s, Industrial Control Systems (ICSs) have evolved vastly. From the primitive isolated setups, ICSs have become increasingly interconnected, slowly forming the complex networked environments, collectively known as Industrial Networks (INs), that we know today. Since ICSs are responsible for a wide range of physical processes, including those belonging to Critical Infrastructures (CIs), securing INs is vital for the well-being of modern societies. Out of the many research advances on the field, Anomaly Detection Systems (ADSs) play a prominent role. These systems monitor IN and/or ICS behavior to detect abnormal events, known or unknown. However, as the complexity of INs has increased, monitoring them in the search of anomalous trends has effectively become a Big Data problem. In other words, IN data has become too complex to process it by traditional means, due to its large scale, diversity and generation speeds. Nevertheless, ADSs designed for INs have not evolved at the same pace, and recent proposals are not designed to handle this data complexity, as they do not scale well or do not leverage the majority of the data types created in INs. This thesis aims to fill that gap, by presenting two main contributions: (i) a visual flow monitoring system and (ii) a multivariate ADS that is able to tackle data heterogeneity and to scale efficiently. For the flow monitor, we propose a system that, based on current flow data, builds security visualizations depicting network behavior while highlighting anomalies. For the multivariate ADS, we analyze the performance of Multivariate Statistical Process Control (MSPC) for detecting and diagnosing anomalies, and later we present a Big Data, MSPCinspired ADS that monitors field and network data to detect anomalies. The approaches are experimentally validated by building INs in test environments and analyzing the data created by them. Based on this necessity for conducting IN security research in a rigorous and reproducible environment, we also propose the design of a testbed that serves this purpose

A monitoring and threat detection system using stream processing as a virtual function for big data

The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast realtime threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.A detecção tardia de ameaças de segurança causa um significante aumento no risco de danos irreparáveis, impossibilitando qualquer tentativa de defesa. Como consequência, a detecção rápida de ameaças em tempo real é essencial para a administração de segurança. Além disso, A tecnologia de virtualização de funções de rede (Network Function Virtualization - NFV) oferece novas oportunidades para soluções de segurança eficazes e de baixo custo. Propomos um sistema de detecção de ameaças rápido e eficiente, baseado em algoritmos de processamento de fluxo e de aprendizado de máquina. As principais contribuições deste trabalho são: i) um novo sistema de monitoramento e detecção de ameaças baseado no processamento de fluxo; ii) dois conjuntos de dados, o primeiro ´e um conjunto de dados sintético de segurança contendo tráfego suspeito e malicioso, e o segundo corresponde a uma semana de tráfego real de um operador de telecomunicações no Rio de Janeiro, Brasil; iii) um algoritmo de pré-processamento de dados composto por um algoritmo de normalização e um algoritmo para seleção rápida de características com base na correlação entre variáveis; iv) uma função de rede virtualizada em uma plataforma de código aberto para fornecer um serviço de detecção de ameaças em tempo real; v) posicionamento quase perfeito de sensores através de uma heurística proposta para posicionamento estratégico de sensores na infraestrutura de rede, com um número mínimo de sensores; e, finalmente, vi) um algoritmo guloso que aloca sob demanda uma sequencia de funções de rede virtual

Near real-time network analysis for the identification of malicious activity

The evolution of technology and the increasing connectivity between devices lead to an increased risk of cyberattacks. Reliable protection systems, such as Intrusion Detection System (IDS) and Intrusion Prevention System (IPS), are essential to try to prevent, detect and counter most of the attacks. However, the increased creativity and type of attacks raise the need for more resources and processing power for the protection systems which, in turn, requires horizontal scalability to keep up with the massive companies’ network infrastructure and with the complexity of attacks. Technologies like machine learning, show promising results and can be of added value in the detection and prevention of attacks in near real-time. But good algorithms and tools are not enough. They require reliable and solid datasets to be able to effectively train the protection systems. The development of a good dataset requires horizontal-scalable, robust, modular and faulttolerant systems so that the analysis may be done in near real-time. This work describes an architecture design for horizontal-scaling capture, storage and analyses, able to collect packets from multiple sources and analyse them in a parallel fashion. The system depends on multiple modular nodes with specific roles to support different algorithms and tools.A evolução da tecnologia e o aumento da conectividade entre dispositivos, levam a um aumento do risco de ciberataques. Os sistemas de deteção de intrusão são essenciais para tentar prevenir, detetar e conter a maioria dos ataques. No entanto, o aumento da criatividade e do tipo de ataques aumenta a necessidade dos sistemas de proteção possuírem cada vez mais recursos e poder computacional. Por sua vez, requerem escalabilidade horizontal para acompanhar a massiva infraestrutura de rede das empresas e a complexidade dos ataques. Tecnologias como machine learning apresentam resultados promissores e podem ser de grande valor na deteção e prevenção de ataques em tempo útil. No entanto, a utilização dos algoritmos e ferramentas requer sempre um conjunto de dados sólidos e confiáveis para treinar os sistemas de proteção de maneira eficaz. A implementação de um bom conjunto de dados requer sistemas horizontalmente escaláveis, robustos, modulares e tolerantes a falhas para que a análise seja rápida e rigorosa. Este trabalho descreve a arquitetura de um sistema de captura, armazenamento e análise, capaz de capturar pacotes de múltiplas fontes e analisá-los de forma paralela. O sistema depende de vários nós modulares com funções específicas para oferecer suporte a diferentes algoritmos e ferramentas

Big Data Analytics for Flow-based Anomaly Detection in High-Speed Networks

The Cisco VNI Complete Forecast Highlights clearly states that the Internet traffic is growing in three different directions, Volume, Velocity, and Variety, bringing computer network into the big data era. At the same time, sophisticated network attacks are growing exponentially. Such growth making the existing signature-based security tools, like firewall and traditional intrusion detection systems, ineffective against new kind of attacks or variations of known attacks. In this dissertation, we propose an unsupervised method for network anomaly detection. This method is able to detect unknown and new malicious activities in high-speed network traffic. Our method uses an innovative detection algorithm able to identify the hosts responsible for anomalous flows by using a new statistical feature related to traffic flow. This feature is defined as the ratio between the number of flows generated by a host and the number of flows it receives. We evaluate our method with real backbone traffic traces from the Measurement and Analysis on the WIDE Internet (MAWI) archive. Furthermore, we compare the results of our method with MAWILab archive, a database that assists researchers to evaluate their traffic anomaly detection methods. The results point out that our method achieves an average positive prediction rate (i.e. Precision) of 90\% outperforming the four MAWILab detection methods in terms of false negative rate. We deploy three cluster configurations to evaluate the horizontal and vertical scalability performance of the proposed architecture and our method shows outstanding performance in terms of response time