4,325 research outputs found
Lightweight Asynchronous Snapshots for Distributed Dataflows
Distributed stateful stream processing enables the deployment and execution
of large scale continuous computations in the cloud, targeting both low latency
and high throughput. One of the most fundamental challenges of this paradigm is
providing processing guarantees under potential failures. Existing approaches
rely on periodic global state snapshots that can be used for failure recovery.
Those approaches suffer from two main drawbacks. First, they often stall the
overall computation which impacts ingestion. Second, they eagerly persist all
records in transit along with the operation states which results in larger
snapshots than required. In this work we propose Asynchronous Barrier
Snapshotting (ABS), a lightweight algorithm suited for modern dataflow
execution engines that minimises space requirements. ABS persists only operator
states on acyclic execution topologies while keeping a minimal record log on
cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics
engine that supports stateful stream processing. Our evaluation shows that our
algorithm does not have a heavy impact on the execution, maintaining linear
scalability and performing well with frequent snapshots.Comment: 8 pages, 7 figure
Asynchronous spiking neurons, the natural key to exploit temporal sparsity
Inference of Deep Neural Networks for stream signal (Video/Audio) processing in edge devices is still challenging. Unlike the most state of the art inference engines which are efficient for static signals, our brain is optimized for real-time dynamic signal processing. We believe one important feature of the brain (asynchronous state-full processing) is the key to its excellence in this domain. In this work, we show how asynchronous processing with state-full neurons allows exploitation of the existing sparsity in natural signals. This paper explains three different types of sparsity and proposes an inference algorithm which exploits all types of sparsities in the execution of already trained networks. Our experiments in three different applications (Handwritten digit recognition, Autonomous Steering and Hand-Gesture recognition) show that this model of inference reduces the number of required operations for sparse input data by a factor of one to two orders of magnitudes. Additionally, due to fully asynchronous processing this type of inference can be run on fully distributed and scalable neuromorphic hardware platforms
Monitoring Partially Synchronous Distributed Systems using SMT Solvers
In this paper, we discuss the feasibility of monitoring partially synchronous
distributed systems to detect latent bugs, i.e., errors caused by concurrency
and race conditions among concurrent processes. We present a monitoring
framework where we model both system constraints and latent bugs as
Satisfiability Modulo Theories (SMT) formulas, and we detect the presence of
latent bugs using an SMT solver. We demonstrate the feasibility of our
framework using both synthetic applications where latent bugs occur at any time
with random probability and an application involving exclusive access to a
shared resource with a subtle timing bug. We illustrate how the time required
for verification is affected by parameters such as communication frequency,
latency, and clock skew. Our results show that our framework can be used for
real-life applications, and because our framework uses SMT solvers, the range
of appropriate applications will increase as these solvers become more
efficient over time.Comment: Technical Report corresponding to the paper accepted at Runtime
Verification (RV) 201
Transport efficiency of metachronal waves in 3d cilia arrays immersed in a two-phase flow
The present work reports the formation and the characterization of
antipleptic and symplectic metachronal waves in 3D cilia arrays immersed in a
two-fluid environment, with a viscosity ratio of 20. A coupled
lattice-Boltzmann-Immersed-Boundary solver is used. The periciliary layer is
confined between the epithelial surface and the mucus. Its thickness is chosen
such that the tips of the cilia can penetrate the mucus. A purely
hydrodynamical feedback of the fluid is taken into account and a coupling
parameter is introduced allowing the tuning of both the direction of
the wave propagation, and the strength of the fluid feedback. A comparative
study of both antipleptic and symplectic waves, mapping a cilia inter-spacing
ranging from 1.67 up to 5 cilia length, is performed by imposing the
metachrony. Antipleptic waves are found to systematically outperform sympletic
waves. They are shown to be more efficient for transporting and mixing the
fluids, while spending less energy than symplectic, random, or synchronized
motions
- …