Causal distributed breakpoints

The authors define a causal distributed breakpoint, which is initiated by a sequential breakpoint in one process of a distributed computation and restores each process in the computation to its earliest state that reflects all events that happened before the breakpoint. An algorithm for finding the causal distributed breakpoint, given a sequential breakpoint in one of the processes, is presented. Approximately consistent checkpoint sets are used for efficiently restoring each process to its state in a causal distributed breakpoint. Causal distributed breakpoints assume deterministic processes that communicate solely by messages. The dependencies that arise from communication between processes are logged. Dependency logging and approximately consistent checkpoint sets are implemented on a network of SUN workstations running the V-System. Overhead on the message-passing primitives varies between 1% and 14% for dependency logging. Execution time overhead for a 200×200 Gaussian elimination is less than 4% and generates a dependency log of 288 kbyte

Causality Diagrams using Hybrid Vector Clocks

Causality in distributed systems is a concept that has long been explored and numerous approaches have been made to use causality as a way to trace distributed system execution. Traditional approaches usually used system profiling and newer approaches profiled clocks of systems to detect failures and construct timelines that caused those failures. Since the advent of logical clocks, these profiles have become more and more accurate with ways to characterize concurrency and distributions, with accurate diagrams for message passing. Vector clocks addressed the shortcomings of using traditional logical clocks, by storing information about other processes in the system as well. Hybrid vector clocks are a novel approach to this concept where clocks need not store all the process information. Rather, we store information of processes within an acceptable skew of the focused process. This gives us an efficient way of profiling with substantially reduced costs to the system. Building on this idea, we propose the idea of building causal traces using information generated from the hybrid vector clock. The hybrid vector clock would provide us with a strong sense of concurrency and distribution, and we theorize that all the information generated from the clock is sufficient to develop a causal trace for debugging. We post-process and parse the clocks generated from an execution trace to develop a swimlane on a web interface, that traces the points of failure of a distributed system. We also provide an API to reuse this concept for any generic distributed system framework

Replay Clocks

In this work, we focus on the problem of replay clocks (RepCL). The need for replay clocks arises from the observation that analyzing distributed computation for all desired properties of interest may not be feasible in an online environment. These properties can be analyzed by replaying the computation. However, to be beneficial, such replay must account for all the uncertainty that is possible in a distributed computation. Specifically, if event 'e' must occur before 'f' then the replay clock must ensure that 'e' is replayed before 'f'. On the other hand, if 'e' and 'f' could occur in any order then replay should not force an order between them. After identifying the limitations of existing clocks to provide the replay primitive, we present RepCL and identify an efficient representation for the same. We demonstrate that RepCL can be implemented with less than four integers for 64 processes for various system parameters if clocks are synchronized within 1 ms. Furthermore, the overhead of RepCL (for computing/comparing timestamps and message size) is proportional to the size of the clock. Using simulations, we identify the expected overhead of RepCL based on the given system settings. We also identify how a user can the identify feasibility region for RepCL. Specifically, given the desired overhead of RepCL, it identifies the region where unabridged replay is possible

Optimal Asynchronous Garbage Collection for RDT Checkpointing Protocols

Communication-induced checkpointing protocols that ensure rollback-dependency trackability (RDT) guarantee important properties to the recovery system without explicit coordination. However, to the best of our knowledge, there was no garbage collection algorithm for them which did not use some type of process synchronization, like time assumptions or reliable control message exchanges. This paper addresses the problem of garbage collection for RDT checkpointing protocols and presents an optimal solution for the case where coordination is done only by means of timestamps piggybacked in application messages. Our algorithm uses the same timestamps as off-the-shelf RDT protocols and ensures the tight upper bound on the number of uncollected checkpoints for each process during all the system execution

Estudo comparativo de algoritmos para checkpointing

Orientador : Luiz Eduardo BuzatoDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Esta dissertação fornece um estudo comparativo abrangente de algoritmos quase-síncronos para checkpointing. Para tanto, utilizamos a simulação de sistemas distribuídos que nos oferece liberdade para construirmos modelos de sistemas com grande facilidade. O estudo comparativo avaliou pela primeira vez de forma uniforme o impacto sobre o desempenho dos algoritmos de fatores como a escala do sistema, a freqüência de check points básicos e a diferença na velocidade dos processos da aplicação. Com base nestes dados obtivemos um profundo conhecimento sobre o comportamento destes algoritmos e produzimos um valioso referencial para projetistas de sistemas em busca de algoritmos para check pointing para as suas aplicações distribuídasAbstract: This dissertation provides a comprehensive comparative study ofthe performance of quase synchronous check pointing algorithms. To do so we used the simulation of distributed systems, which provides freedom to build system models easily. The comparative study assessed for the first time in an uniform environment the impact of the algorithms' performance with respect to factors such as the system's scale, the basic checkpoint rate and the relative processes' speed. By analyzing these data we acquired a deep understanding of the behavior of these algorithms and were able to produce a valuable reference to system architects looking for check pointing algorithms for their distributed applicationsMestradoMestre em Ciência da Computaçã

Online Trace Reordering for Efficient Representation of Event Partial Orders

Distributed and parallel applications not only have distributed state but are often inherently non-deterministic, making them significantly more challenging to monitor and debug. Additionally, a significant challenge when working with distributed and parallel applications has to do with the fundamental requirement of determining the order in which certain actions are performed by the application. A naive approach for ordering actions would be to impose a single order on all actions, i.e., given any two actions or events, one must happen before the other. A global order, however, is often misleading, e.g., two events in two different processes may be causally independent yet one may have occurred before the other. A partial order of events, therefore, serves as the fundamental data structure for ordering events in distributed and parallel applications. Traditionally, Fidge/Mattern timestamps have been used for representing event partial orders. The size of the vector timestamp depends on the number of parallel entities (traces) in the application, e.g., processes or threads. A major limitation of Fidge/Mattern time- stamps is that the total size of timestamps does not scale for large systems with hundreds or thousands of traces. Taylor proposed an efficient offset-based scheme for representing large event partial orders by representing deltas between timestamps of successive events. The offset-based schemes have been shown to be significantly more space efficient when traces that communicate the most are close to each other for generating the deltas (offsets). In Taylor’s offset-based schemes the optimal order of traces is computed offline. In this work we adapt the offset-based schemes to dynamically reorder traces and demonstrate that very efficient scalable representations of event partial orders can be generated in an online setting, requiring as few as 100 bytes/event for storing partial order event data for applications with around 1000 processes

Causal Distributed Breakpoints

A causal distributed breakpoint is initiated by a sequential breakpoint in one process of a distributed computation, and restores each process in the computation to its earliest state that reflects all events that "happened before" the breakpoint. A causal distributed breakpoint is the natural extension for distributed programs of the conventional notion of a breakpoint in a sequential program. We present an algorithm for finding the causal distributed breakpoint given a sequential breakpoint in one of the processes. Approximately consistent checkpoint sets are used for efficiently restoring each process to its state in a causal distributed breakpoint. Causal distributed breakpoints assume deterministic processes that communicate solely by messages. The dependencies that arise from communication between processes are logged. Dependency logging and approximately consistent checkpoints sets have been implemented on a network of SUN workstations running the V-System. Overhead on the mess..