Lightweight Asynchronous Snapshots for Distributed Dataflows

Distributed stateful stream processing enables the deployment and execution of large scale continuous computations in the cloud, targeting both low latency and high throughput. One of the most fundamental challenges of this paradigm is providing processing guarantees under potential failures. Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks. First, they often stall the overall computation which impacts ingestion. Second, they eagerly persist all records in transit along with the operation states which results in larger snapshots than required. In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements. ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics engine that supports stateful stream processing. Our evaluation shows that our algorithm does not have a heavy impact on the execution, maintaining linear scalability and performing well with frequent snapshots.Comment: 8 pages, 7 figure

One-bit Distributed Sensing and Coding for Field Estimation in Sensor Networks

This paper formulates and studies a general distributed field reconstruction problem using a dense network of noisy one-bit randomized scalar quantizers in the presence of additive observation noise of unknown distribution. A constructive quantization, coding, and field reconstruction scheme is developed and an upper-bound to the associated mean squared error (MSE) at any point and any snapshot is derived in terms of the local spatio-temporal smoothness properties of the underlying field. It is shown that when the noise, sensor placement pattern, and the sensor schedule satisfy certain weak technical requirements, it is possible to drive the MSE to zero with increasing sensor density at points of field continuity while ensuring that the per-sensor bitrate and sensing-related network overhead rate simultaneously go to zero. The proposed scheme achieves the order-optimal MSE versus sensor density scaling behavior for the class of spatially constant spatio-temporal fields.Comment: Fixed typos, otherwise same as V2. 27 pages (in one column review format), 4 figures. Submitted to IEEE Transactions on Signal Processing. Current version is updated for journal submission: revised author list, modified formulation and framework. Previous version appeared in Proceedings of Allerton Conference On Communication, Control, and Computing 200

Enduring Lagrangian coherence of a Loop Current ring assessed using independent observations

Ocean flows are routinely inferred from low-resolution satellite altimetry measurements of sea surface height assuming a geostrophic balance. Recent nonlinear dynamical systems techniques have revealed that surface currents derived from altimetry can support mesoscale eddies with material boundaries that do not filament for many months, thereby representing effective transport mechanisms. However, the long-range Lagrangian coherence assessed for mesoscale eddy boundaries detected from altimetry is constrained by the impossibility of current altimeters to resolve ageostrophic submesoscale motions. These may act to prevent Lagrangian coherence from manifesting in the rigorous form described by the nonlinear dynamical systems theories. Here we use a combination of satellite ocean color and surface drifter trajectory data, rarely available simultaneously over an extended period of time, to provide observational evidence for the enduring Lagrangian coherence of a Loop Current ring detected from altimetry. We also seek indications of this behavior in the flow produced by a data-assimilative system which demonstrated ability to reproduce observed relative dispersion statistics down into the marginally submesoscale range. However, the simulated flow, total surface and subsurface or subsampled emulating altimetry, is not found to support the long-lasting Lagrangian coherence that characterizes the observed ring. This highlights the importance of the Lagrangian metrics produced by the nonlinear dynamical systems tools employed here in assessing model performance.Comment: In press in nature.com/Scientific Report

Starling flock networks manage uncertainty in consensus at low cost

Flocks of starlings exhibit a remarkable ability to maintain cohesion as a group in highly uncertain environments and with limited, noisy information. Recent work demonstrated that individual starlings within large flocks respond to a fixed number of nearest neighbors, but until now it was not understood why this number is seven. We analyze robustness to uncertainty of consensus in empirical data from multiple starling flocks and show that the flock interaction networks with six or seven neighbors optimize the trade-off between group cohesion and individual effort. We can distinguish these numbers of neighbors from fewer or greater numbers using our systems-theoretic approach to measuring robustness of interaction networks as a function of the network structure, i.e., who is sensing whom. The metric quantifies the disagreement within the network due to disturbances and noise during consensus behavior and can be evaluated over a parameterized family of hypothesized sensing strategies (here the parameter is number of neighbors). We use this approach to further show that for the range of flocks studied the optimal number of neighbors does not depend on the number of birds within a flock; rather, it depends on the shape, notably the thickness, of the flock. The results suggest that robustness to uncertainty may have been a factor in the evolution of flocking for starlings. More generally, our results elucidate the role of the interaction network on uncertainty management in collective behavior, and motivate the application of our approach to other biological networks.Comment: 19 pages, 3 figures, 9 supporting figure