2,103 research outputs found
Lightweight Asynchronous Snapshots for Distributed Dataflows
Distributed stateful stream processing enables the deployment and execution
of large scale continuous computations in the cloud, targeting both low latency
and high throughput. One of the most fundamental challenges of this paradigm is
providing processing guarantees under potential failures. Existing approaches
rely on periodic global state snapshots that can be used for failure recovery.
Those approaches suffer from two main drawbacks. First, they often stall the
overall computation which impacts ingestion. Second, they eagerly persist all
records in transit along with the operation states which results in larger
snapshots than required. In this work we propose Asynchronous Barrier
Snapshotting (ABS), a lightweight algorithm suited for modern dataflow
execution engines that minimises space requirements. ABS persists only operator
states on acyclic execution topologies while keeping a minimal record log on
cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics
engine that supports stateful stream processing. Our evaluation shows that our
algorithm does not have a heavy impact on the execution, maintaining linear
scalability and performing well with frequent snapshots.Comment: 8 pages, 7 figure
An occam Style Communications System for UNIX Networks
This document describes the design of a communications system which provides occam style communications primitives under a Unix environment, using TCP/IP protocols, and any number of other protocols deemed suitable as underlying transport layers. The system will integrate with a low overhead scheduler/kernel without incurring significant costs to the execution of processes within the run time environment. A survey of relevant occam and occam3 features and related research is followed by a look at the Unix and TCP/IP facilities which determine our working constraints, and a description of the T9000 transputer's Virtual Channel Processor, which was instrumental in our formulation. Drawing from the information presented here, a design for the communications system is subsequently proposed. Finally, a preliminary investigation of methods for lightweight access control to shared resources in an environment which does not provide support for critical sections, semaphores, or busy waiting, is made. This is presented with relevance to mutual exclusion problems which arise within the proposed design. Future directions for the evolution of this project are discussed in conclusion
Distributed Queuing in Dynamic Networks
We consider the problem of forming a distributed queue in the adversarial
dynamic network model of Kuhn, Lynch, and Oshman (STOC 2010) in which the
network topology changes from round to round but the network stays connected.
This is a synchronous model in which network nodes are assumed to be fixed, the
communication links for each round are chosen by an adversary, and nodes do not
know who their neighbors are for the current round before they broadcast their
messages. Queue requests may arrive over rounds at arbitrary nodes and the goal
is to eventually enqueue them in a distributed queue. We present two algorithms
that give a total distributed ordering of queue requests in this model. We
measure the performance of our algorithms through round complexity, which is
the total number of rounds needed to solve the distributed queuing problem. We
show that in 1-interval connected graphs, where the communication links change
arbitrarily between every round, it is possible to solve the distributed
queueing problem in O(nk) rounds using O(log n) size messages, where n is the
number of nodes in the network and k <= n is the number of queue requests.
Further, we show that for more stable graphs, e.g. T-interval connected graphs
where the communication links change in every T rounds, the distributed queuing
problem can be solved in O(n+ (nk/min(alpha,T))) rounds using the same O(log n)
size messages, where alpha > 0 is the concurrency level parameter that captures
the minimum number of active queue requests in the system in any round. These
results hold in any arbitrary (sequential, one-shot concurrent, or dynamic)
arrival of k queue requests in the system. Moreover, our algorithms ensure
correctness in the sense that each queue request is eventually enqueued in the
distributed queue after it is issued and each queue request is enqueued exactly
once. We also provide an impossibility result for this distributed queuing
problem in this model. To the best of our knowledge, these are the first
solutions to the distributed queuing problem in adversarial dynamic networks.Comment: In Proceedings FOMC 2013, arXiv:1310.459
High performance computing of explicit schemes for electrofusion jointing process based on message-passing paradigm
The research focused on heterogeneous cluster workstations comprising of a number of CPUs in single and shared architecture platform. The problem statements under consideration involved one dimensional parabolic equations. The thermal process of electrofusion jointing was also discussed. Numerical schemes of explicit type such as AGE, Brian, and Charlies Methods were employed. The parallelization of these methods were based on the domain decomposition technique. Some parallel performance measurement for these methods were also addressed. Temperature profile of the one dimensional radial model of the electrofusion process were also given
- …