Distributed Network Anomaly Detection on an Event Processing Framework

Network Intrusion Detection Systems (NIDS) are an integral part of modern data centres to ensure high availability and compliance with Service Level Agreements (SLAs). Currently, NIDS are deployed on high-performance, high-cost middleboxes that are responsible for monitoring a limited section of the network. The fast increasing size and aggregate throughput of modern data centre networks have come to challenge the current approach to anomaly detection to satisfy the fast growing compute demand. In this paper, we propose a novel approach to distributed intrusion detection systems based on the architecture of recently proposed event processing frameworks. We have designed and implemented a prototype system using Apache Storm to show the benefits of the proposed approach as well as the architectural differences with traditional systems. Our system distributes modules across the available devices within the network fabric and uses a centralised controller for orchestration, management and correlation. Following the Software Defined Networking (SDN) paradigm, the controller maintains a complete view of the network but distributes the processing logic for quick event processing while performing complex event correlation centrally. We have evaluated the proposed system using publicly available data centre traces and demonstrated that the system can scale with the network topology while providing high performance and minimal impact on packet latency

P4CEP: Towards In-Network Complex Event Processing

In-network computing using programmable networking hardware is a strong trend in networking that promises to reduce latency and consumption of server resources through offloading to network elements (programmable switches and smart NICs). In particular, the data plane programming language P4 together with powerful P4 networking hardware has spawned projects offloading services into the network, e.g., consensus services or caching services. In this paper, we present a novel case for in-network computing, namely, Complex Event Processing (CEP). CEP processes streams of basic events, e.g., stemming from networked sensors, into meaningful complex events. Traditionally, CEP processing has been performed on servers or overlay networks. However, we argue in this paper that CEP is a good candidate for in-network computing along the communication path avoiding detouring streams to distant servers to minimize communication latency while also exploiting processing capabilities of novel networking hardware. We show that it is feasible to express CEP operations in P4 and also present a tool to compile CEP operations, formulated in our P4CEP rule specification language, to P4 code. Moreover, we identify challenges and problems that we have encountered to show future research directions for implementing full-fledged in-network CEP systems.Comment: 6 pages. Author's versio

A Survey on IT-Techniques for a Dynamic Emergency Management in Large Infrastructures

This deliverable is a survey on the IT techniques that are relevant to the three use cases of the project EMILI. It describes the state-of-the-art in four complementary IT areas: Data cleansing, supervisory control and data acquisition, wireless sensor networks and complex event processing. Even though the deliverable’s authors have tried to avoid a too technical language and have tried to explain every concept referred to, the deliverable might seem rather technical to readers so far little familiar with the techniques it describes

Event detection, tracking, and visualization in Twitter: a mention-anomaly-based approach

The ever-growing number of people using Twitter makes it a valuable source of timely information. However, detecting events in Twitter is a difficult task, because tweets that report interesting events are overwhelmed by a large volume of tweets on unrelated topics. Existing methods focus on the textual content of tweets and ignore the social aspect of Twitter. In this paper we propose MABED (i.e. mention-anomaly-based event detection), a novel statistical method that relies solely on tweets and leverages the creation frequency of dynamic links (i.e. mentions) that users insert in tweets to detect significant events and estimate the magnitude of their impact over the crowd. MABED also differs from the literature in that it dynamically estimates the period of time during which each event is discussed, rather than assuming a predefined fixed duration for all events. The experiments we conducted on both English and French Twitter data show that the mention-anomaly-based approach leads to more accurate event detection and improved robustness in presence of noisy Twitter content. Qualitatively speaking, we find that MABED helps with the interpretation of detected events by providing clear textual descriptions and precise temporal descriptions. We also show how MABED can help understanding users' interest. Furthermore, we describe three visualizations designed to favor an efficient exploration of the detected events.Comment: 17 page

Search for gravitational-wave bursts in LIGO data from the fourth science run

The fourth science run of the LIGO and GEO 600 gravitational-wave detectors, carried out in early 2005, collected data with significantly lower noise than previous science runs. We report on a search for short-duration gravitational-wave bursts with arbitrary waveform in the 64-1600 Hz frequency range appearing in all three LIGO interferometers. Signal consistency tests, data quality cuts, and auxiliary-channel vetoes are applied to reduce the rate of spurious triggers. No gravitational-wave signals are detected in 15.5 days of live observation time; we set a frequentist upper limit of 0.15 per day (at 90% confidence level) on the rate of bursts with large enough amplitudes to be detected reliably. The amplitude sensitivity of the search, characterized using Monte Carlo simulations, is several times better than that of previous searches. We also provide rough estimates of the distances at which representative supernova and binary black hole merger signals could be detected with 50% efficiency by this analysis.Comment: Corrected amplitude sensitivities (7% change on average); 30 pages, submitted to Classical and Quantum Gravit

Exploring the Time Domain With Synoptic Sky Surveys

Synoptic sky surveys are becoming the largest data generators in astronomy, and they are opening a new research frontier, that touches essentially every field of astronomy. Opening of the time domain to a systematic exploration will strengthen our understanding of a number of interesting known phenomena, and may lead to the discoveries of as yet unknown ones. We describe some lessons learned over the past decade, and offer some ideas that may guide strategic considerations in planning and execution of the future synoptic sky surveys.Comment: Invited talk, to appear in proc. IAU SYmp. 285, "New Horizons in Time Domain Astronomy", eds. E. Griffin et al., Cambridge Univ. Press (2012). Latex file, 6 pages, style files include