Don't Repeat Yourself: Seamless Execution and Analysis of Extensive Network Experiments

This paper presents MACI, the first bespoke framework for the management, the scalable execution, and the interactive analysis of a large number of network experiments. Driven by the desire to avoid repetitive implementation of just a few scripts for the execution and analysis of experiments, MACI emerged as a generic framework for network experiments that significantly increases efficiency and ensures reproducibility. To this end, MACI incorporates and integrates established simulators and analysis tools to foster rapid but systematic network experiments. We found MACI indispensable in all phases of the research and development process of various communication systems, such as i) an extensive DASH video streaming study, ii) the systematic development and improvement of Multipath TCP schedulers, and iii) research on a distributed topology graph pattern matching algorithm. With this work, we make MACI publicly available to the research community to advance efficient and reproducible network experiments

Principles of building scalable and robust event-based systems

Event-based systems are of tremendous importance for a wide range of distributed applications interacting with physical processes, e.g., traffic management, financial services, manufacturing processes, or health services. Event-based systems support to monitor, analyze events of interest efficiently. Therefore, they enable distributed applications to respond to detected events in the form of appropriate actions. Event-based systems provide as part of the publish/subscribe paradigm, mechanisms for the scalable integration of a variety of information sources, e.g., dedicated sensor networks, mobile devices, or cameras. In addition, event-based systems allow as part of the event processing paradigm to detect correlations between events from distinct information sources. Event-based systems ensure two important forms of decoupling of importance building scalable distributed applications. Decoupling producers of information and consumers of information by ensuring that neither producers need to keep state on the interested consumers nor consumers need to know the producers of information, is a key principle for scalable communications. Furthermore, a step-wise correlation from primary events to events of importance for distributed applications is an enabler to specify distributed applications independent from the underlying sensor infrastructure at hand. In this thesis, we present and discuss principles of building scalable and robust event-based systems. On the one hand, this requires distributed mechanisms to fulfill a wide spectrum of distinct application requirements, e.g., being bandwidth efficient and providing events with low end-to-end latency. On the other hand, the underlying mechanisms for event-based systems need to deal with many levels of dynamics, e. g., dynamics in the rate at which events are produced, dynamics in the interest of producers and consumers, mobility of consumer and producer, failures and changing security privileges to access events. In the context of mechanisms for event distribution, operator execution, operator migration, operator recovery and secure access to events, we highlight problems in the scalable and robust design of those mechanisms. We give an overview on related work in the field and present in a tutorial manner the ideas of six own contributions for realizing distributed event-based systems

Window-based Parallel Operator Execution with In-Network Computing

Data parallel processing is a key concept to increase the scalability and elasticity in event streaming systems. Often data parallelism is accomplished in a splitter-merger architecture where the splitter divides incoming streams into partitions and forwards them to parallel operator instances. The splitter performance is a limiting factor to the system throughput and the parallelization degree. This work studies how to leverage novel methods of in-network computing to accelerate the splitter functionality by implementing it as an in-network function. While dedicated hardware for in-network computing has a high potential to enhance the splitter performance, in-network programming models like the P4 language are also highly limited in their expressiveness to support corresponding parallelization models. We propose P4 Splitter Switch (P4SS) which supports overlapping and non-overlapping count-based windows for multiple independent data streams and parallelizes them to a dynamically configurable number of operator instances. We validate in the context of a prototypical implementation our splitting strategy and its scalability in terms of switch resource consumption

Towards adaptive quality-aware Complex Event Processing in the Internet of Things

This paper investigates how to complement Complex Event Processing (CEP) with dynamic quality monitoring mechanisms and support the dynamic integration of suitable sensory data sources. In the proposed approach, queries to detect complex events are annotated with consumer-definable quality policies that are evaluated and used to autonomously assign (or even configure) suitable data sources of the sensing infrastructure. We present and study different forms of expressing quality policies and explore how they affect the process of quality monitoring including different modes of assessing and applying quality-related adaptations. A performance study in an IoT scenario shows that the proposed mechanisms in supporting quality policy monitoring and adaptively selecting suitable data sources succeed in enhancing the acquired quality of results while fulfilling consumers’ quality requirements. We show that the quality-based selection of sensor sources also extends the network’s lifetime by optimizing the data sources’ energy consumption

Towards Pattern-Level Privacy Protection in Distributed Complex Event Processing

In event processing systems, detected event patterns can revealprivacy-sensitive information. In this paper, we proposeand discuss how to integrate pattern-level privacy protection in event-based systems. Compared to state-of-the-art approaches, we aim to enforce privacy independent of the particularities of specific operators. We accomplish this by supporting the flexible integration of multiple obfuscation techniques and studying deployment strategies for privacy-enforcing mechanisms. Moreover, we share ideas on how to model the adversary’s knowledge to better select appropriate obfuscation techniques for the discussed deployment strategies. Initial results indicate that flexibly choosing obfuscation techniques and deployment strategies is essential to conceal privacy-sensitive event patterns accurately

Travel light:State shedding for efficient operator migration

Operator migration is a crucial concept to adapt event processing systems to dynamic changes. When the placement of a stateful operator changes, the operator state must be migrated to the new host. However, operator state size and time constraints can make it impossible to migrate the operator without severe Quality of Service (QoS) degradation. As a relief, we propose to perform state shedding in such a situation. The core idea of state shedding is to partition the operator state, assign a utility to each partial state, and use the utility and size of each partial state to identify the most useful partial states that can be migrated in a given time frame. Thus, state shedding can maintain a substantially higher QoS with a lower impact on query results than state-of-the-art solutions targeting consistent state at the old and new host. In this paper, we define this novel approach and in a simulation environment evaluate state shedding in migration scenarios with pattern-matching queries

Cordies:Expressive event correlation in distributed systems

Complex Event Processing (CEP) is the method of choice for the observation of system states and situations by means of events. A number of systems have been introduced that provide CEP in selected environments. Some are restricted to centralised systems, or to systems with synchronous communication, or to a limited space of event relations that are defined in advance. Many modern systems, though, are inherently distributed and asynchronous, and require a more powerful CEP. We present Cordies, a distributed system for the detection of correlated events that is designed for the operation in large-scale, heterogeneous networks and adapts dynamically to changing network conditions. With its expressive language to describe event relations, it is suitable for environments where neither the event space nor the situations of interest are predefined but are constantly adapted. In addition, Cordies supports Quality-of-Service (QoS) for communication in distributed event correlation detection