spChains: A Declarative Framework for Data Stream Processing in Pervasive Applications

Pervasive applications rely on increasingly complex streams of sensor data continuously captured from the physical world. Such data is crucial to enable applications to ``understand'' the current context and to infer the right actions to perform, be they fully automatic or involving some user decisions. However, the continuous nature of such streams, the relatively high throughput at which data is generated and the number of sensors usually deployed in the environment, make direct data handling practically unfeasible. Data not only needs to be cleaned, but it must also be filtered and aggregated to relieve higher level algorithms from near real-time handling of such massive data flows. We propose here a stream-processing framework (spChains), based upon state-of-the-art stream processing engines, which enables declarative and modular composition of stream processing chains built atop of a set of extensible stream processing blocks. While stream processing blocks are delivered as a standard, yet extensible, library of application-independent processing elements, chains can be defined by the pervasive application engineering team. We demonstrate the flexibility and effectiveness of the spChains framework on two real-world applications in the energy management and in the industrial plant management domains, by evaluating them on a prototype implementation based on the Esper stream processo

Knowledge-infused and Consistent Complex Event Processing over Real-time and Persistent Streams

Emerging applications in Internet of Things (IoT) and Cyber-Physical Systems (CPS) present novel challenges to Big Data platforms for performing online analytics. Ubiquitous sensors from IoT deployments are able to generate data streams at high velocity, that include information from a variety of domains, and accumulate to large volumes on disk. Complex Event Processing (CEP) is recognized as an important real-time computing paradigm for analyzing continuous data streams. However, existing work on CEP is largely limited to relational query processing, exposing two distinctive gaps for query specification and execution: (1) infusing the relational query model with higher level knowledge semantics, and (2) seamless query evaluation across temporal spaces that span past, present and future events. These allow accessible analytics over data streams having properties from different disciplines, and help span the velocity (real-time) and volume (persistent) dimensions. In this article, we introduce a Knowledge-infused CEP (X-CEP) framework that provides domain-aware knowledge query constructs along with temporal operators that allow end-to-end queries to span across real-time and persistent streams. We translate this query model to efficient query execution over online and offline data streams, proposing several optimizations to mitigate the overheads introduced by evaluating semantic predicates and in accessing high-volume historic data streams. The proposed X-CEP query model and execution approaches are implemented in our prototype semantic CEP engine, SCEPter. We validate our query model using domain-aware CEP queries from a real-world Smart Power Grid application, and experimentally analyze the benefits of our optimizations for executing these queries, using event streams from a campus-microgrid IoT deployment.Comment: 34 pages, 16 figures, accepted in Future Generation Computer Systems, October 27, 201

A Survey on IT-Techniques for a Dynamic Emergency Management in Large Infrastructures

This deliverable is a survey on the IT techniques that are relevant to the three use cases of the project EMILI. It describes the state-of-the-art in four complementary IT areas: Data cleansing, supervisory control and data acquisition, wireless sensor networks and complex event processing. Even though the deliverable’s authors have tried to avoid a too technical language and have tried to explain every concept referred to, the deliverable might seem rather technical to readers so far little familiar with the techniques it describes

Dark Matter Direct Detection with Non-Maxwellian Velocity Structure

The velocity distribution function of dark matter particles is expected to show significant departures from a Maxwell-Boltzmann distribution. This can have profound effects on the predicted dark matter - nucleon scattering rates in direct detection experiments, especially for dark matter models in which the scattering is sensitive to the high velocity tail of the distribution, such as inelastic dark matter (iDM) or light (few GeV) dark matter (LDM), and for experiments that require high energy recoil events, such as many directionally sensitive experiments. Here we determine the velocity distribution functions from two of the highest resolution numerical simulations of Galactic dark matter structure (Via Lactea II and GHALO), and study the effects for these scenarios. For directional detection, we find that the observed departures from Maxwell-Boltzmann increase the contrast of the signal and change the typical direction of incoming DM particles. For iDM, the expected signals at direct detection experiments are changed dramatically: the annual modulation can be enhanced by more than a factor two, and the relative rates of DAMA compared to CDMS can change by an order of magnitude, while those compared to CRESST can change by a factor of two. The spectrum of the signal can also change dramatically, with many features arising due to substructure. For LDM the spectral effects are smaller, but changes do arise that improve the compatibility with existing experiments. We find that the phase of the modulation can depend upon energy, which would help discriminate against background should it be found.Comment: 34 pages, 16 figures, submitted to JCAP. Tables of g(v_min), the integral of f(v)/v from v_min to infinity, derived from our simulations, are available for download at http://astro.berkeley.edu/~mqk/dmdd