29,978 research outputs found
Distributed Network Anomaly Detection on an Event Processing Framework
Network Intrusion Detection Systems (NIDS) are an integral part of modern data centres to ensure high availability and compliance with Service Level Agreements (SLAs). Currently, NIDS are deployed on high-performance, high-cost middleboxes that are responsible for monitoring a limited section of the network. The fast increasing size and aggregate throughput of modern data centre networks have come to challenge the current approach to anomaly detection to satisfy the fast growing compute demand. In this paper, we propose a novel approach to distributed intrusion detection systems based on the architecture of recently proposed event processing frameworks. We have designed and implemented a prototype system using Apache Storm to show the benefits of the proposed approach as well as the architectural differences with traditional systems. Our system distributes modules across the available devices within the network fabric and uses a centralised controller for orchestration, management and correlation. Following the Software Defined Networking (SDN) paradigm, the controller maintains a complete view of the network but distributes the processing logic for quick event processing while performing complex event correlation centrally. We have evaluated the proposed system using publicly available data centre traces and demonstrated that the system can scale with the network topology while providing high performance and minimal impact on packet latency
A consensus based network intrusion detection system
Network intrusion detection is the process of identifying malicious behaviors
that target a network and its resources. Current systems implementing intrusion
detection processes observe traffic at several data collecting points in the
network but analysis is often centralized or partly centralized. These systems
are not scalable and suffer from the single point of failure, i.e. attackers
only need to target the central node to compromise the whole system. This paper
proposes an anomaly-based fully distributed network intrusion detection system
where analysis is run at each data collecting point using a naive Bayes
classifier. Probability values computed by each classifier are shared among
nodes using an iterative average consensus protocol. The final analysis is
performed redundantly and in parallel at the level of each data collecting
point, thus avoiding the single point of failure issue. We run simulations
focusing on DDoS attacks with several network configurations, comparing the
accuracy of our fully distributed system with a hierarchical one. We also
analyze communication costs and convergence speed during consensus phases.Comment: Presented at THE 5TH INTERNATIONAL CONFERENCE ON IT CONVERGENCE AND
SECURITY 2015 IN KUALA LUMPUR, MALAYSI
StreamLearner: Distributed Incremental Machine Learning on Event Streams: Grand Challenge
Today, massive amounts of streaming data from smart devices need to be
analyzed automatically to realize the Internet of Things. The Complex Event
Processing (CEP) paradigm promises low-latency pattern detection on event
streams. However, CEP systems need to be extended with Machine Learning (ML)
capabilities such as online training and inference in order to be able to
detect fuzzy patterns (e.g., outliers) and to improve pattern recognition
accuracy during runtime using incremental model training. In this paper, we
propose a distributed CEP system denoted as StreamLearner for ML-enabled
complex event detection. The proposed programming model and data-parallel
system architecture enable a wide range of real-world applications and allow
for dynamically scaling up and out system resources for low-latency,
high-throughput event processing. We show that the DEBS Grand Challenge 2017
case study (i.e., anomaly detection in smart factories) integrates seamlessly
into the StreamLearner API. Our experiments verify scalability and high event
throughput of StreamLearner.Comment: Christian Mayer, Ruben Mayer, and Majd Abdo. 2017. StreamLearner:
Distributed Incremental Machine Learning on Event Streams: Grand Challenge.
In Proceedings of the 11th ACM International Conference on Distributed and
Event-based Systems (DEBS '17), 298-30
- …