Semantics based analysis of botnet activity from heterogeneous data sources

International audienceThe diversity in network devices, protocols, data sources and probes impose different challenges to uniformly measure and analyse network traffic. Analysing a network means considering distinctive reporting approaches and diverse methods to represent data, measure times or identify nodes. In this work, we tackle these challenges by relying on semantics, taking advantage of the ontologies' ability to map high-level network concepts to concrete information sources of different nature. In particular, we propose a simple architecture to map network concepts to data stored in relational databases. Based on this architecture, we implement a tool that looks for malicious bot activity, studying, from a unique point of view, DNS traffic from PCAP sources, and TCP connections from IPFIX reports. This approach is able to enhance current DNS based botnet detection methods, taking into account additional heterogeneous analysis elements

Graph Mining for Cybersecurity: A Survey

The explosive growth of cyber attacks nowadays, such as malware, spam, and intrusions, caused severe consequences on society. Securing cyberspace has become an utmost concern for organizations and governments. Traditional Machine Learning (ML) based methods are extensively used in detecting cyber threats, but they hardly model the correlations between real-world cyber entities. In recent years, with the proliferation of graph mining techniques, many researchers investigated these techniques for capturing correlations between cyber entities and achieving high performance. It is imperative to summarize existing graph-based cybersecurity solutions to provide a guide for future studies. Therefore, as a key contribution of this paper, we provide a comprehensive review of graph mining for cybersecurity, including an overview of cybersecurity tasks, the typical graph mining techniques, and the general process of applying them to cybersecurity, as well as various solutions for different cybersecurity tasks. For each task, we probe into relevant methods and highlight the graph types, graph approaches, and task levels in their modeling. Furthermore, we collect open datasets and toolkits for graph-based cybersecurity. Finally, we outlook the potential directions of this field for future research

A Cyber Threat Intelligence Sharing Scheme based on Federated Learning for Network Intrusion Detection

The uses of Machine Learning (ML) in detection of network attacks have been effective when designed and evaluated in a single organisation. However, it has been very challenging to design an ML-based detection system by utilising heterogeneous network data samples originating from several sources. This is mainly due to privacy concerns and the lack of a universal format of datasets. In this paper, we propose a collaborative federated learning scheme to address these issues. The proposed framework allows multiple organisations to join forces in the design, training, and evaluation of a robust ML-based network intrusion detection system. The threat intelligence scheme utilises two critical aspects for its application; the availability of network data traffic in a common format to allow for the extraction of meaningful patterns across data sources. Secondly, the adoption of a federated learning mechanism to avoid the necessity of sharing sensitive users' information between organisations. As a result, each organisation benefits from other organisations cyber threat intelligence while maintaining the privacy of its data internally. The model is trained locally and only the updated weights are shared with the remaining participants in the federated averaging process. The framework has been designed and evaluated in this paper by using two key datasets in a NetFlow format known as NF-UNSW-NB15-v2 and NF-BoT-IoT-v2. Two other common scenarios are considered in the evaluation process; a centralised training method where the local data samples are shared with other organisations and a localised training method where no threat intelligence is shared. The results demonstrate the efficiency and effectiveness of the proposed framework by designing a universal ML model effectively classifying benign and intrusive traffic originating from multiple organisations without the need for local data exchange

Agent‐based modeling of malware dynamics in heterogeneous environments

The increasing convergence of power‐law networks such as social networking and peer‐to‐peer applications, web‐delivered applications, and mobile platforms makes today's users highly vulnerable to entirely new generations of malware that exploit vulnerabilities in web applications and mobile platforms for new infections, while using the power‐law connectivity for finding new victims. The traditional epidemic models based on assumptions of homogeneity, average‐degree distributions, and perfect‐mixing are inadequate to model this type of malware propagation. In this paper, we study four aspects crucial to modeling malware propagation: application‐level interactions among users of such networks , local network structure , user mobility , and network coordination of malware such as botnets . Since closed‐form solutions of malware propagation considering these aspects are difficult to obtain, we describe an open‐source, flexible agent‐based emulation framework that can be used by malware researchers for studying today's complex malware. The framework, called Agent‐Based Malware Modeling (AMM), allows different applications, network structure, network coordination, and user mobility in either a geographic or a logical domain to study various infection and propagation scenarios. In addition to traditional worms and viruses, the framework also allows modeling network coordination of malware such as botnets. The majority of the parameters used in the framework can be derived from real‐life network traces collected from a network, and therefore, represent realistic malware propagation and infection scenarios. As representative examples, we examine two well‐known malware spreading mechanisms: (i) a malicious virus such as Cabir spreading among the subscribers of a cellular network using Bluetooth and (ii) a hybrid worm that exploit email and file‐sharing to infect users of a social network. In both cases, we identify the parameters most important to the spread of the epidemic based upon our extensive simulation results. Copyright © 2011 John Wiley & Sons, Ltd. This paper presents a novel agent‐based framework for realistic modeling of malware propagation in heterogeneous networks, applications and platforms. The majority of the parameters used in the framework can be derived from real‐life network traces collected from a network, and therefore, represent realistic malware propagation and infection scenarios for the given network. Two well‐known malware spreading mechanisms in traditional as well as mobile environments were studied using extensive simulations within the framework and the most important spreading parameters were identified.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/101832/1/sec298.pd

Multi-agent-based DDoS detection on big data systems

The Hadoop framework has become the most deployed platform for processing Big Data. Despite its advantages, Hadoop s infrastructure is still deployed within the secured network perimeter because the framework lacks adequate inherent security mechanisms against various security threats. However, this approach is not sufficient for providing adequate security layer against attacks such as Distributed Denial of Service. Furthermore, current work to secure Hadoop s infrastructure against DDoS attacks is unable to provide a distributed node-level detection mechanism. This thesis presents a software agent-based framework that allows distributed, real-time intelligent monitoring and detection of DDoS attack at Hadoop s node-level. The agent s cognitive system is ingrained with cumulative sum statistical technique to analyse network utilisation and average server load and detect attacks from these measurements. The framework is a multi-agent architecture with transducer agents that interface with each Hadoop node to provide real-time detection mechanism. Moreover, the agents contextualise their beliefs by training themselves with the contextual information of each node and monitor the activities of the node to differentiate between normal and anomalous behaviours. In the experiments, the framework was exposed to TCP SYN and UDP flooding attacks during a legitimate MapReduce job on the Hadoop testbed. The experimental results were evaluated regarding performance metrics such as false-positive ratio, false-negative ratio and response time to attack. The results show that UDP and TCP SYN flooding attacks can be detected and confirmed on multiple nodes in nineteen seconds with 5.56% false-positive ration, 7.70% false-negative ratio and 91.5% success rate of detection. The results represent an improvement compare to the state-of the-ar

Clustered Federated Learning Architecture for Network Anomaly Detection in Large Scale Heterogeneous IoT Networks

There is a growing trend of cyberattacks against Internet of Things (IoT) devices; moreover, the sophistication and motivation of those attacks is increasing. The vast scale of IoT, diverse hardware and software, and being typically placed in uncontrolled environments make traditional IT security mechanisms such as signature-based intrusion detection and prevention systems challenging to integrate. They also struggle to cope with the rapidly evolving IoT threat landscape due to long delays between the analysis and publication of the detection rules. Machine learning methods have shown faster response to emerging threats; however, model training architectures like cloud or edge computing face multiple drawbacks in IoT settings, including network overhead and data isolation arising from the large scale and heterogeneity that characterizes these networks. This work presents an architecture for training unsupervised models for network intrusion detection in large, distributed IoT and Industrial IoT (IIoT) deployments. We leverage Federated Learning (FL) to collaboratively train between peers and reduce isolation and network overhead problems. We build upon it to include an unsupervised device clustering algorithm fully integrated into the FL pipeline to address the heterogeneity issues that arise in FL settings. The architecture is implemented and evaluated using a testbed that includes various emulated IoT/IIoT devices and attackers interacting in a complex network topology comprising 100 emulated devices, 30 switches and 10 routers. The anomaly detection models are evaluated on real attacks performed by the testbed's threat actors, including the entire Mirai malware lifecycle, an additional botnet based on the Merlin command and control server and other red-teaming tools performing scanning activities and multiple attacks targeting the emulated devices