Engineering Crowdsourced Stream Processing Systems

A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort

LightBox: Full-stack Protected Stateful Middlebox at Lightning Speed

Running off-site software middleboxes at third-party service providers has been a popular practice. However, routing large volumes of raw traffic, which may carry sensitive information, to a remote site for processing raises severe security concerns. Prior solutions often abstract away important factors pertinent to real-world deployment. In particular, they overlook the significance of metadata protection and stateful processing. Unprotected traffic metadata like low-level headers, size and count, can be exploited to learn supposedly encrypted application contents. Meanwhile, tracking the states of 100,000s of flows concurrently is often indispensable in production-level middleboxes deployed at real networks. We present LightBox, the first system that can drive off-site middleboxes at near-native speed with stateful processing and the most comprehensive protection to date. Built upon commodity trusted hardware, Intel SGX, LightBox is the product of our systematic investigation of how to overcome the inherent limitations of secure enclaves using domain knowledge and customization. First, we introduce an elegant virtual network interface that allows convenient access to fully protected packets at line rate without leaving the enclave, as if from the trusted source network. Second, we provide complete flow state management for efficient stateful processing, by tailoring a set of data structures and algorithms optimized for the highly constrained enclave space. Extensive evaluations demonstrate that LightBox, with all security benefits, can achieve 10Gbps packet I/O, and that with case studies on three stateful middleboxes, it can operate at near-native speed.Comment: Accepted at ACM CCS 201

Theoretical vs. Empirical Classification and Prediction of Congested Traffic States

Starting from the instability diagram of a traffic flow model, we derive conditions for the occurrence of congested traffic states, their appearance, their spreading in space and time, and the related increase in travel times. We discuss the terminology of traffic phases and give empirical evidence for the existence of a phase diagram of traffic states. In contrast to previously presented phase diagrams, it is shown that "widening synchronized patterns" are possible, if the maximum flow is located inside of a metastable density regime. Moreover, for various kinds of traffic models with different instability diagrams it is discussed, how the related phase diagrams are expected to approximately look like. Apart from this, it is pointed out that combinations of on- and off-ramps create different patterns than a single, isolated on-ramp.Comment: See http://www.helbing.org for related wor

Mapping the interactions between rivers and sand dunes: Implications for fluvial and aeolian geomorphology

The interaction between fluvial and aeolian processes can significantly change Earth surface morphology. When rivers and sand dunes meet, the interaction of sediment transport between the two systems can lead to change in either or both systems. However, these two systems are usually studied independently, which leaves many questions unresolved in terms of how they interact. This paper carries out a global inventory, using satellite imagery, to identify 230 sites where there are significant fluvial-aeolian interactions. At each location key attributes such as wind/river direction, net sand transport direction, fluvial-aeolian meeting angle, dune type and river channel pattern were identified and relationships between each factor were analysed. From these data, six different types of interaction were classified that reflect a shift in dominance between the fluvial and aeolian systems. Results from this classification confirm that only certain types of interaction were significant: the meeting angle and dune type, the meeting angle and interaction type and finally the channel pattern and interaction type. However, the findings also indicate the difficulties of classifying dynamic geomorphic systems from snapshot satellite images

Hydraulic flow through a channel contraction: multiple steady states

We have investigated shallow water flows through a channel with a contraction by experimental and theoretical means. The horizontal channel consists of a sluice gate and an upstream channel of constant width $b_0$ ending in a linear contraction of minimum width $b_c$. Experimentally, we observe upstream steady and moving bores/shocks, and oblique waves in the contraction, as single and multiple steady states, as well as a steady reservoir with a complex hydraulic jump in the contraction occurring in a small section of the $b_c/b_0$ and Froude number parameter plane. One-dimensional hydraulic theory provides a comprehensive leading-order approximation, in which a turbulent frictional parametrization is used to achieve quantitative agreement. An analytical and numerical analysis is given for two-dimensional supercritical shallow water flows. It shows that the one-dimensional hydraulic analysis for inviscid flows away from hydraulic jumps holds surprisingly well, even though the two-dimensional oblique hydraulic jump patterns can show large variations across the contraction channel