Lightweight Asynchronous Snapshots for Distributed Dataflows

Distributed stateful stream processing enables the deployment and execution of large scale continuous computations in the cloud, targeting both low latency and high throughput. One of the most fundamental challenges of this paradigm is providing processing guarantees under potential failures. Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks. First, they often stall the overall computation which impacts ingestion. Second, they eagerly persist all records in transit along with the operation states which results in larger snapshots than required. In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements. ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics engine that supports stateful stream processing. Our evaluation shows that our algorithm does not have a heavy impact on the execution, maintaining linear scalability and performing well with frequent snapshots.Comment: 8 pages, 7 figure

A type system for components

In modern distributed systems, dynamic reconfiguration, i.e., changing at runtime the communication pattern of a program, is chal- lenging. Generally, it is difficult to guarantee that such modifications will not disrupt ongoing computations. In a previous paper, a solution to this problem was proposed by extending the object-oriented language ABS with a component model allowing the programmer to: i) perform up- dates on objects by means of communication ports and their rebinding; and ii) precisely specify when such updates can safely occur in an object by means of critical sections. However, improper rebind operations could still occur and lead to runtime errors. The present paper introduces a type system for this component model that extends the ABS type system with the notion of ports and a precise analysis that statically enforces that no object will attempt illegal rebinding

Towards Reversible Sessions

In this work, we incorporate reversibility into structured communication-based programming, to allow parties of a session to automatically undo, in a rollback fashion, the effect of previously executed interactions. This permits taking different computation paths along the same session, as well as reverting the whole session and starting a new one. Our aim is to define a theoretical basis for examining the interplay in concurrent systems between reversible computation and session-based interaction. We thus enrich a session-based variant of pi-calculus with memory devices, dedicated to keep track of the computation history of sessions in order to reverse it. We discuss our initial investigation concerning the definition of a session type discipline for the proposed reversible calculus, and its practical advantages for static verification of safe composition in communication-centric distributed software performing reversible computations.Comment: In Proceedings PLACES 2014, arXiv:1406.331

The finite element machine: An experiment in parallel processing

The finite element machine is a prototype computer designed to support parallel solutions to structural analysis problems. The hardware architecture and support software for the machine, initial solution algorithms and test applications, and preliminary results are described