48,018 research outputs found
Lightweight Asynchronous Snapshots for Distributed Dataflows
Distributed stateful stream processing enables the deployment and execution
of large scale continuous computations in the cloud, targeting both low latency
and high throughput. One of the most fundamental challenges of this paradigm is
providing processing guarantees under potential failures. Existing approaches
rely on periodic global state snapshots that can be used for failure recovery.
Those approaches suffer from two main drawbacks. First, they often stall the
overall computation which impacts ingestion. Second, they eagerly persist all
records in transit along with the operation states which results in larger
snapshots than required. In this work we propose Asynchronous Barrier
Snapshotting (ABS), a lightweight algorithm suited for modern dataflow
execution engines that minimises space requirements. ABS persists only operator
states on acyclic execution topologies while keeping a minimal record log on
cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics
engine that supports stateful stream processing. Our evaluation shows that our
algorithm does not have a heavy impact on the execution, maintaining linear
scalability and performing well with frequent snapshots.Comment: 8 pages, 7 figure
One-bit Distributed Sensing and Coding for Field Estimation in Sensor Networks
This paper formulates and studies a general distributed field reconstruction
problem using a dense network of noisy one-bit randomized scalar quantizers in
the presence of additive observation noise of unknown distribution. A
constructive quantization, coding, and field reconstruction scheme is developed
and an upper-bound to the associated mean squared error (MSE) at any point and
any snapshot is derived in terms of the local spatio-temporal smoothness
properties of the underlying field. It is shown that when the noise, sensor
placement pattern, and the sensor schedule satisfy certain weak technical
requirements, it is possible to drive the MSE to zero with increasing sensor
density at points of field continuity while ensuring that the per-sensor
bitrate and sensing-related network overhead rate simultaneously go to zero.
The proposed scheme achieves the order-optimal MSE versus sensor density
scaling behavior for the class of spatially constant spatio-temporal fields.Comment: Fixed typos, otherwise same as V2. 27 pages (in one column review
format), 4 figures. Submitted to IEEE Transactions on Signal Processing.
Current version is updated for journal submission: revised author list,
modified formulation and framework. Previous version appeared in Proceedings
of Allerton Conference On Communication, Control, and Computing 200
Enduring Lagrangian coherence of a Loop Current ring assessed using independent observations
Ocean flows are routinely inferred from low-resolution satellite altimetry
measurements of sea surface height assuming a geostrophic balance. Recent
nonlinear dynamical systems techniques have revealed that surface currents
derived from altimetry can support mesoscale eddies with material boundaries
that do not filament for many months, thereby representing effective transport
mechanisms. However, the long-range Lagrangian coherence assessed for mesoscale
eddy boundaries detected from altimetry is constrained by the impossibility of
current altimeters to resolve ageostrophic submesoscale motions. These may act
to prevent Lagrangian coherence from manifesting in the rigorous form described
by the nonlinear dynamical systems theories. Here we use a combination of
satellite ocean color and surface drifter trajectory data, rarely available
simultaneously over an extended period of time, to provide observational
evidence for the enduring Lagrangian coherence of a Loop Current ring detected
from altimetry. We also seek indications of this behavior in the flow produced
by a data-assimilative system which demonstrated ability to reproduce observed
relative dispersion statistics down into the marginally submesoscale range.
However, the simulated flow, total surface and subsurface or subsampled
emulating altimetry, is not found to support the long-lasting Lagrangian
coherence that characterizes the observed ring. This highlights the importance
of the Lagrangian metrics produced by the nonlinear dynamical systems tools
employed here in assessing model performance.Comment: In press in nature.com/Scientific Report
Starling flock networks manage uncertainty in consensus at low cost
Flocks of starlings exhibit a remarkable ability to maintain cohesion as a
group in highly uncertain environments and with limited, noisy information.
Recent work demonstrated that individual starlings within large flocks respond
to a fixed number of nearest neighbors, but until now it was not understood why
this number is seven. We analyze robustness to uncertainty of consensus in
empirical data from multiple starling flocks and show that the flock
interaction networks with six or seven neighbors optimize the trade-off between
group cohesion and individual effort. We can distinguish these numbers of
neighbors from fewer or greater numbers using our systems-theoretic approach to
measuring robustness of interaction networks as a function of the network
structure, i.e., who is sensing whom. The metric quantifies the disagreement
within the network due to disturbances and noise during consensus behavior and
can be evaluated over a parameterized family of hypothesized sensing strategies
(here the parameter is number of neighbors). We use this approach to further
show that for the range of flocks studied the optimal number of neighbors does
not depend on the number of birds within a flock; rather, it depends on the
shape, notably the thickness, of the flock. The results suggest that robustness
to uncertainty may have been a factor in the evolution of flocking for
starlings. More generally, our results elucidate the role of the interaction
network on uncertainty management in collective behavior, and motivate the
application of our approach to other biological networks.Comment: 19 pages, 3 figures, 9 supporting figure
- …