1,717 research outputs found
Network Intrusion Detection System using Spark's Scalable Machine Learning Library
In this paper, considering that the serious network security situation we are facing and the problem of an increasing amount of data generated by the network, we proposed an Intrusion Detection System based on Spark's scalable machine learning library,In this paper we are showing that performance of Intrusion Detection system using sparks machine learning library is high in compare to hadoop. Fro IDS we will use K-Means algorithm
Distributed Network Anomaly Detection on an Event Processing Framework
Network Intrusion Detection Systems (NIDS) are an integral part of modern data centres to ensure high availability and compliance with Service Level Agreements (SLAs). Currently, NIDS are deployed on high-performance, high-cost middleboxes that are responsible for monitoring a limited section of the network. The fast increasing size and aggregate throughput of modern data centre networks have come to challenge the current approach to anomaly detection to satisfy the fast growing compute demand. In this paper, we propose a novel approach to distributed intrusion detection systems based on the architecture of recently proposed event processing frameworks. We have designed and implemented a prototype system using Apache Storm to show the benefits of the proposed approach as well as the architectural differences with traditional systems. Our system distributes modules across the available devices within the network fabric and uses a centralised controller for orchestration, management and correlation. Following the Software Defined Networking (SDN) paradigm, the controller maintains a complete view of the network but distributes the processing logic for quick event processing while performing complex event correlation centrally. We have evaluated the proposed system using publicly available data centre traces and demonstrated that the system can scale with the network topology while providing high performance and minimal impact on packet latency
BigFCM: Fast, Precise and Scalable FCM on Hadoop
Clustering plays an important role in mining big data both as a modeling
technique and a preprocessing step in many data mining process implementations.
Fuzzy clustering provides more flexibility than non-fuzzy methods by allowing
each data record to belong to more than one cluster to some degree. However, a
serious challenge in fuzzy clustering is the lack of scalability. Massive
datasets in emerging fields such as geosciences, biology and networking do
require parallel and distributed computations with high performance to solve
real-world problems. Although some clustering methods are already improved to
execute on big data platforms, but their execution time is highly increased for
large datasets. In this paper, a scalable Fuzzy C-Means (FCM) clustering named
BigFCM is proposed and designed for the Hadoop distributed data platform. Based
on the map-reduce programming model, it exploits several mechanisms including
an efficient caching design to achieve several orders of magnitude reduction in
execution time. Extensive evaluation over multi-gigabyte datasets shows that
BigFCM is scalable while it preserves the quality of clustering
On the Efficacy of Live DDoS Detection with Hadoop
Distributed Denial of Service flooding attacks are one of the biggest
challenges to the availability of online services today. These DDoS attacks
overwhelm the victim with huge volume of traffic and render it incapable of
performing normal communication or crashes it completely. If there are delays
in detecting the flooding attacks, nothing much can be done except to manually
disconnect the victim and fix the problem. With the rapid increase of DDoS
volume and frequency, the current DDoS detection technologies are challenged to
deal with huge attack volume in reasonable and affordable response time.
In this paper, we propose HADEC, a Hadoop based Live DDoS Detection framework
to tackle efficient analysis of flooding attacks by harnessing MapReduce and
HDFS. We implemented a counter-based DDoS detection algorithm for four major
flooding attacks (TCP-SYN, HTTP GET, UDP and ICMP) in MapReduce, consisting of
map and reduce functions. We deployed a testbed to evaluate the performance of
HADEC framework for live DDoS detection. Based on the experiments we showed
that HADEC is capable of processing and detecting DDoS attacks in affordable
time
- …