Big Data Classification: Problems and Challenges in Network Intrusion Prediction with Machine Learning

This paper focuses on the specific problem of Big Data classification of network intrusion traffic. It discusses the system challenges presented by the Big Data problems associated with network intrusion prediction. The prediction of a possible intrusion attack in a network requires continuous collection of traffic data and learning of their characteristics on the fly. The continuous collection of traffic data by the network leads to Big Data problems that are caused by the volume, variety and velocity properties of Big Data. The learning of the network characteristics requires machine learning techniques that capture global knowledge of the traffic patterns. The Big Data properties will lead to significant system challenges to implement machine learning frameworks. This paper discusses the problems and challenges in handling Big Data classification using geometric representation-learning techniques and the modern Big Data networking technologies. In particular this paper discusses the issues related to combining supervised learning techniques, representation-learning techniques, machine lifelong learning techniques and Big Data technologies (e.g. Hadoop, Hive and Cloud) for solving network traffic classification problems

Graph-based feature enrichment for online intrusion detection in virtual networks

The increasing number of connected devices to provide the required ubiquitousness of Internet of Things paves the way for distributed network attacks at an unprecedented scale. Graph theory, strengthened by machine learning techniques, improves an automatic discovery of group behavior patterns of network threats often omitted by traditional security systems. Furthermore, Network Function Virtualization is an emergent technology that accelerates the provisioning of on-demand security function chains tailored to an application. Therefore, repeatable compliance tests and performance comparison of such function chains are mandatory. The contributions of this dissertation are divided in two parts. First, we propose an intrusion detection system for online threat detection enriched by a graph-learning analysis. We develop a feature enrichment algorithm that infers metrics from a graph analysis. By using different machine learning techniques, we evaluated our algorithm for three network traffic datasets. We show that the proposed graph-based enrichment improves the threat detection accuracy up to 15.7% and significantly reduces the false positives rate. Second, we aim to evaluate intrusion detection systems deployed as virtual network functions. Therefore, we propose and develop SFCPerf, a framework for an automatic performance evaluation of service function chaining. To demonstrate SFCPerf functionality, we design and implement a prototype of a security service function chain, composed of our intrusion detection system and a firewall. We show the results of a SFCPerf experiment that evaluates the chain prototype on top of the open platform for network function virtualization (OPNFV).O crescente número de dispositivos IoT conectados contribui para a ocorrência de ataques distribuídos de negação de serviço a uma escala sem precedentes. A Teoria de Grafos, reforçada por técnicas de aprendizado de máquina, melhora a descoberta automática de padrões de comportamento de grupos de ameaças de rede, muitas vezes omitidas pelos sistemas tradicionais de segurança. Nesse sentido, a virtualização da função de rede é uma tecnologia emergente que pode acelerar o provisionamento de cadeias de funções de segurança sob demanda para uma aplicação. Portanto, a repetição de testes de conformidade e a comparação de desempenho de tais cadeias de funções são obrigatórios. As contribuições desta dissertação são separadas em duas partes. Primeiro, é proposto um sistema de detecção de intrusão que utiliza um enriquecimento baseado em grafos para aprimorar a detecção de ameaças online. Um algoritmo de enriquecimento de características é desenvolvido e avaliado através de diferentes técnicas de aprendizado de máquina. Os resultados mostram que o enriquecimento baseado em grafos melhora a acurácia da detecção de ameaças até 15,7 % e reduz significativamente o número de falsos positivos. Em seguida, para avaliar sistemas de detecção de intrusões implantados como funções virtuais de rede, este trabalho propõe e desenvolve o SFCPerf, um framework para avaliação automática de desempenho do encadeamento de funções de rede. Para demonstrar a funcionalidade do SFCPerf, ´e implementado e avaliado um protótipo de uma cadeia de funções de rede de segurança, composta por um sistema de detecção de intrusão (IDS) e um firewall sobre a plataforma aberta para virtualização de função de rede (OPNFV)

Symmetry-Adapted Machine Learning for Information Security

Symmetry-adapted machine learning has shown encouraging ability to mitigate the security risks in information and communication technology (ICT) systems. It is a subset of artificial intelligence (AI) that relies on the principles of processing future events by learning past events or historical data. The autonomous nature of symmetry-adapted machine learning supports effective data processing and analysis for security detection in ICT systems without the interference of human authorities. Many industries are developing machine-learning-adapted solutions to support security for smart hardware, distributed computing, and the cloud. In our Special Issue book, we focus on the deployment of symmetry-adapted machine learning for information security in various application areas. This security approach can support effective methods to handle the dynamic nature of security attacks by extraction and analysis of data to identify hidden patterns of data. The main topics of this Issue include malware classification, an intrusion detection system, image watermarking, color image watermarking, battlefield target aggregation behavior recognition model, IP camera, Internet of Things (IoT) security, service function chain, indoor positioning system, and crypto-analysis

Intrusion Detection for Cyber-Physical Attacks in Cyber-Manufacturing System

In the vision of Cyber-Manufacturing System (CMS) , the physical components such as products, machines, and tools are connected, identifiable and can communicate via the industrial network and the Internet. This integration of connectivity enables manufacturing systems access to computational resources, such as cloud computing, digital twin, and blockchain. The connected manufacturing systems are expected to be more efficient, sustainable and cost-effective. However, the extensive connectivity also increases the vulnerability of physical components. The attack surface of a connected manufacturing environment is greatly enlarged. Machines, products and tools could be targeted by cyber-physical attacks via the network. Among many emerging security concerns, this research focuses on the intrusion detection of cyber-physical attacks. The Intrusion Detection System (IDS) is used to monitor cyber-attacks in the computer security domain. For cyber-physical attacks, however, there is limited work. Currently, the IDS cannot effectively address cyber-physical attacks in manufacturing system: (i) the IDS takes time to reveal true alarms, sometimes over months; (ii) manufacturing production life-cycle is shorter than the detection period, which can cause physical consequences such as defective products and equipment damage; (iii) the increasing complexity of network will also make the detection period even longer. This gap leaves the cyber-physical attacks in manufacturing to cause issues like over-wearing, breakage, defects or any other changes that the original design didn’t intend. A review on the history of cyber-physical attacks, and available detection methods are presented. The detection methods are reviewed in terms of intrusion detection algorithms, and alert correlation methods. The attacks are further broken down into a taxonomy covering four dimensions with over thirty attack scenarios to comprehensively study and simulate cyber-physical attacks. A new intrusion detection and correlation method was proposed to address the cyber-physical attacks in CMS. The detection method incorporates IDS software in cyber domain and machine learning analysis in physical domain. The correlation relies on a new similarity-based cyber-physical alert correlation method. Four experimental case studies were used to validate the proposed method. Each case study focused on different aspects of correlation method performance. The experiments were conducted on a security-oriented manufacturing testbed established for this research at Syracuse University. The results showed the proposed intrusion detection and alert correlation method can effectively disclose unknown attack, known attack and attack interference that causes false alarms. In case study one, the alarm reduction rate reached 99.1%, with improvement of detection accuracy from 49.6% to 100%. The case studies also proved the proposed method can mitigate false alarms, detect attacks on multiple machines, and attacks from the supply chain. This work contributes to the security domain in cyber-physical manufacturing systems, with the focus on intrusion detection. The dataset collected during the experiments has been shared with the research community. The alert correlation methodology also contributes to cyber-physical systems, such as smart grid and connected vehicles, which requires enhanced security protection in today’s connected world

Intrusion Detection System based on time related features and Machine Learning

The analysis of the behavior of network communications over time allows the extraction of statistical features capable of characterizing the network traffic flows. These features can be used to create an Intrusion Detection System (IDS) that can automatically classify network traffic. But introducing an IDS into a network changes the latency of its communications. From a different viewpoint it is possible to analyze the latencies of a network to try to identifying the presence or absence of the IDS. The proposed method can be used to extract a set of phisical or time related features that characterize the communication behavior of an Internet of Things (IoT) infrastructure. For example the number of packets sent every 5 minutes. Then these features can help identify anomalies or cyber attacks. For example a jamming of the radio channel. This method does not necessarily take into account the content of the network packet and therefore can also be used on encrypted connections where is impossible to carry out a Deep Packet Inspection (DPI) analysis

Complex Event Processing(CEP) for Intrusion Detection

Σε αυτή την εργασία ασχολούμαστε με τη χρήση των τεχνολογιών ανάλυσης δεδομένων για τη μελέτη της συμπεριφοράς των δικτύων IoT [3]. Οι συσκευές IoT βρίσκονται παντού γύρω μας και δεν πρόκειται να ξεπεραστούν σύντομα, οπως είναι τα έξυπνα βραχιόλια υγειας , έξυπνες συσκευές που συνδέονται με οχήματα και έξυπνα ενεργειακοί πάροχοι. Αλλά τι γίνεται με την ασφάλεια; Αυτά τα συστήματα είναι σε θέση να συγκεντρώνουν και να μοιράζονται τεράστιες ποσότητες ευαίσθητων δεδομένων του χρήστη. Οι καταναλωτές είναι συνεχώς εκτεθειμένοι σε επιθέσεις και φυσικές εισβολές επειδή χρησιμοποιουν ένα ευρύ φάσμα των διαθέσιμων συσκευών IoT, όπως κεντρικές συσκευές ελέγχου για αισθητήρες οικιακού αυτοματισμού. Όπως μπορούμε να φανταστούμε αυτές οι συσκευές είναι εγγενώς ανασφαλής (και οι χρήστες τους συχνά αγνοούν τις επικείμενες απειλές), και αποτελούν εύκολη λεία για τους επιτιθέμενους. Παράλληλα, οι συσκευές IoT μπορούν να χαρακτηριστούν ως χαμηλού κόστους, δηλαδή συσκευές με περιορισμένη επεξεργαστική ισχύ, μπαταρία και μνήμη. Αυτό σημαίνει ότι οι λύσεις που αφορούν την ασφάλεια των έξυπνων συσκευών, καθώς και τα προσωπικά δεδομένα των χρηστών αποτελουν πρόκληση. Η προτεινόμενη προσέγγιση προσφέρει μια εφαρμογή που λύνει το πρόβλημα των εισβολών ασφαλείας με τη χρήση δεδομένων που δημιουργούνται από συσκευές IoT που σχετίζονται με τις ιδιότητες του δικτύου τους με σκοπό τον εντοπισμό μη φυσιολογικών συμπεριφορών και ενημερώνει τον χρήστη μέσω ειδοποιήσεων. Στην περίπτωσή μας κάθε συσκευή που συμμετέχει σε ένα δίκτυο IoT αντιμετωπίζεται ως μια συσκευή αισθητήρα που μετράει τα χαρακτηριστικα του δικτύου, χρησιμοποιώντας ένα πρωτόκολλο διαχείρισης δικτύου (SNMP). Οι μετρήσεις αυτές παρέχονται ως είσοδος σε Σύνθετη Επεξεργασία Γεγονότων (CEP) που ονομάζεται Esper [1]. Οι αισθητήρες του CEP εντοπίζουν και να αναλύουν τα δεδομένα του αισθητήρα σε πραγματικό χρόνο με βάση τα κατώτατα όρια που σχετίζονται με τη φυσιολογική συμπεριφορά. Μια τέτοια διαφορετική συμπεριφορά μπορεί να είναι μια σαφής ένδειξη της εμφάνισης συμβάντος (π.χ. επίθεση). Οι μετρήσεις των συσκευών μπορούν να συνδυαστούν ώστε να μπορούμε να ανιχνεύσουμε διαφόρες επιθέσεις ασφάλειας με μεγαλύτερη σιγουριά. Οι εκτιμήσεις του προγράμματος CEP βασίζεται σε στατιστικούς προγνωστικούς παράγοντες, συμπεριλαμβανομένων των μεθόδων μηχανικής μάθησης όπως ο αλγόριθμος ARΤ. Σας παρουσιάζουμε μια σειρά πειραμάτων για τις προτεινόμενες μεθοδολογίες που δείχνουν την απόδοσή τουςIn this thesis we deal with the usage of data analysis technologies to study the behavior of IoT [3] networks. IoT devices are everywhere, and they’re not going away any time soon, including wearable health, connected vehicles and smart grids. But what about security? These systems are able to gather and share huge quantities of sensitive user data. Consumers are constantly exposed to attacks and physical intrusions due to the use of a wide range of available IoT devices, such central control devices for home automation sensors. As we can imagine these devices are inherently insecure (and their users are often unaware of any impending threats), they’re easy prey for hackers. In parallel IoT devices can be characterized as low cost, i.e. devices with limited processing power, battery and memory. This means that device-centric solutions for incorporating security and privacy components will be a challenge as well. The proposed approach offers an application solution to the problem of security intrusions (anomaly-based detection) by using streams generated by IoT devices relevant to their network properties in order to detect abnormal behavior and notify the user via an alert. In our case, each device participating in a IoT network is handled as a sensor device that generates streams of network measurements by using Simple Network Management Protocol (SNMP) [1]. These measurements are provided as input to Complex Event Processing (CEP) [4] framework, i.e. Esper [2]. CEP listeners detect and analyze the sensor streams in real time based on thresholds related to the normal behavior. Such abnormal statistical behavior can be a clear indication of an event occurrence (e.g., intrusion). Typical measurements of the devices can be combined in order to more accurately observe the outbreak of various security incidents. The estimations of CEP engine will be based on statistical predictors including machine learning methods like ART [5]. We present a number of experiments for the proposed methodologies that show their performance

A monitoring and threat detection system using stream processing as a virtual function for big data

The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast realtime threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.A detecção tardia de ameaças de segurança causa um significante aumento no risco de danos irreparáveis, impossibilitando qualquer tentativa de defesa. Como consequência, a detecção rápida de ameaças em tempo real é essencial para a administração de segurança. Além disso, A tecnologia de virtualização de funções de rede (Network Function Virtualization - NFV) oferece novas oportunidades para soluções de segurança eficazes e de baixo custo. Propomos um sistema de detecção de ameaças rápido e eficiente, baseado em algoritmos de processamento de fluxo e de aprendizado de máquina. As principais contribuições deste trabalho são: i) um novo sistema de monitoramento e detecção de ameaças baseado no processamento de fluxo; ii) dois conjuntos de dados, o primeiro ´e um conjunto de dados sintético de segurança contendo tráfego suspeito e malicioso, e o segundo corresponde a uma semana de tráfego real de um operador de telecomunicações no Rio de Janeiro, Brasil; iii) um algoritmo de pré-processamento de dados composto por um algoritmo de normalização e um algoritmo para seleção rápida de características com base na correlação entre variáveis; iv) uma função de rede virtualizada em uma plataforma de código aberto para fornecer um serviço de detecção de ameaças em tempo real; v) posicionamento quase perfeito de sensores através de uma heurística proposta para posicionamento estratégico de sensores na infraestrutura de rede, com um número mínimo de sensores; e, finalmente, vi) um algoritmo guloso que aloca sob demanda uma sequencia de funções de rede virtual