Leveraging Distributed Tracing and Container Cloning for Replay Debugging of Microservices

Microservice architectures have gained prominence in recent years for building large-scale industrial distributed systems. However, microservice architectures make the usage of replay debugging, a powerful technique for finding root causes of faults, very challenging because of the polyglot (written in several languages) services, large accumulated state of services, and tight latency limits imposed by long hop-chains. This work attempts to provide a framework for enabling replay debugging in production microservice applications. We study 25 real-world faults in microservice systems collected from diverse sources, categorize these faults by fault symptoms, and create 15 application agnostic mutation operators for microservices. We then propose a language agnostic replay debugging framework for microservice applications that uses a distributed tracing system to record network requests and enables replay of those requests on cloned service containers running in a debug environment. A key component of this framework is an anomaly detector that uses span-level and container-level monitoring to detect fault symptoms found in our study and localizes faults to trace level so that faulty traces can be easily replayed to find the root cause. An open-source microservices application injected successively with the mutation operators is used for an evaluation that shows that our framework is upto an order of magnitude lighter-weight than language-specific recording tools such as Chrome DevTools or VisualVM and can help in finding root causes of 9 out of 15 mutations at a line or function level

Dynamic Slicing by On-demand Re-execution

In this paper, we propose a novel approach that aims to offer an alternative to the prevalent paradigm to dynamic slicing construction. Dynamic slicing requires dynamic data and control dependencies that arise in an execution. During a single execution, memory reference information is recorded and then traversed to extract dependencies. Execute-once approaches and tools are challenged even by executions of moderate size of simple and short programs. We propose to shift practical time complexity from execution size to slice size. In particular, our approach executes the program multiple times while tracking targeted information at each execution. We present a concrete algorithm that follows an on-demand re-execution paradigm that uses a novel concept of frontier dependency to incrementally build a dynamic slice. To focus dependency tracking, the algorithm relies on static analysis. We show results of an evaluation on the SV-COMP benchmark and Antrl4 unit tests that provide evidence that on-demand re-execution can provide performance gains particularly when slice size is small and execution size is large

Replaying and Isolating Failure-Inducing Program Interactions

A program fails. What now? A programmer must debug the program to fix the problem, by completing two fundamental debugging tasks: first, the programmer has to reproduce the failure; second, she has to find the failure cause. Both tasks can result in a tedious, long-lasting, and boring work on the one hand, and can be a factor that significantly drives up costs and risks on the other hand. The field of automated debugging aims to ease the search for failure causes. This work presents JINSI, taking a new twist on automated debugging that aims to combine ease of use with unprecedented effectiveness. Taking a single failing run, we automatically record and minimize the interactions between objects to the set of calls relevant for the failure. The result is a minimal unit test that faithfully reproduces the failure at will: "Out of these 14,628 calls, only 2 are required". In a study of 17 real-life bugs, JINSI reduced the search space to 13.7 % of the dynamic slice or 0.22 % of the source code, with only 1 to 12 calls left to examine—a precision not only significantly above the state of the art, but also at a level at which fault localization ceases to be a problem. Moreover, by combining delta debugging, event slicing, and dynamic slicing, we are able to automatically compute failure reproductions along cause-effect chains eventually leading to the defect—both efficiently and effectively. JINSI provides minimal unit tests with related calls which pinpoints the circumstances under which the failure occurs at different abstraction levels. In that way, the approach discussed in this thesis ensures a high diagnostic quality and enables the programmer to concentrate on automatically selected parts of the program relevant to the failure.Ein Programm liefert ein falsches Ergebnis. Was nun? Ein Entwickler muss das Programm, mit dem Ziel den eigentlichen Defekt im Programmcode zu beheben, debuggen. Hierzu sind zwei fundamentale Schritte erforderlich: der Entwickler muss den Fehler zunächst reproduzieren, und er muss schließlich die genaue Fehlerursache finden. Diese Aufgabenstellung kann einerseits in einer mühsamen, langwierigen und auch langweiligen Suche münden, anderseits kann sie einen Faktor darstellen, der die Kosten der Entwicklung erheblich in die Höhe treibt und die damit verbundenen Risiken schwer kalkulierbar macht. Ziel des Gebietes der Automatischen Fehlersuche ist es, jene Suche nach den genauen Ursachen von Programmfehlern zu vereinfachen. Die vorliegende Dissertation beschreibt JINSI, ein neuartiges Verfahren zur automatischen Fehlersuche, welches Anwenderfreundlichkeit mit beispielloser Leistungsfähigkeit vereint. Anhand eines einzelnen fehlschlagenden Programmlaufes zeichnen wir Objektinteraktionen auf und vereinfachen diese, bis lediglich jene Interaktionen verbleiben, die für den Fehler relevant sind. Das Ergebnis ist ein minimaler Unittest, der den Fehler originalgetreu und nach Belieben reproduzieren kann: "Von den 14.628 Methodenaufrufen werden nur 2 benötigt." In einer Untersuchung von 17 Fehlern aus der Praxis reduzierte JINSI den Suchraum auf 13,7% des dynamischen Slices bzw. auf 0,22% des Quelltextes. Dabei blieben 1 bis 12 Interaktionen übrig, die zum Auffinden der Fehlerursache untersucht werden müssen. Diese Präzision übertrifft nicht nur den bis dato letzten Stand der Forschung, sondern ist auch auf einem Niveau, auf dem das Problem der Fehlerursachenbestimmung aufhört zu bestehen. Durch die Verknüpfung von Delta Debugging, Event Slicing und Dynamischem Slicing sind wir in der Lage Fehlerreproduktionen vollautomatisch entlang von Ursache-Wirkungsketten zu bestimmen — vom Symptom bis zur Ursache des Fehlers. JINSI bestimmt Unittests mit zusammenhängenden Interaktionen die genau festlegen, unter welchen Umständen ein Fehler auf verschiedenen Abstraktionsebenen innerhalb des Programmes auftritt. Auf diese Weise erreicht das in dieser Arbeit diskutierte Verfahren eine hohe Diagnosequalität und hilft dem Programmierer sich auf die Teile des Programmes zu konzentrieren, die für das Fehlschlagen relevant sind

Doctor of Philosophy

dissertationA modern software system is a composition of parts that are themselves highly complex: operating systems, middleware, libraries, servers, and so on. In principle, compositionality of interfaces means that we can understand any given module independently of the internal workings of other parts. In practice, however, abstractions are leaky, and with every generation, modern software systems grow in complexity. Traditional ways of understanding failures, explaining anomalous executions, and analyzing performance are reaching their limits in the face of emergent behavior, unrepeatability, cross-component execution, software aging, and adversarial changes to the system at run time. Deterministic systems analysis has a potential to change the way we analyze and debug software systems. Recorded once, the execution of the system becomes an independent artifact, which can be analyzed offline. The availability of the complete system state, the guaranteed behavior of re-execution, and the absence of limitations on the run-time complexity of analysis collectively enable the deep, iterative, and automatic exploration of the dynamic properties of the system. This work creates a foundation for making deterministic replay a ubiquitous system analysis tool. It defines design and engineering principles for building fast and practical replay machines capable of capturing complete execution of the entire operating system with an overhead of several percents, on a realistic workload, and with minimal installation costs. To enable an intuitive interface of constructing replay analysis tools, this work implements a powerful virtual machine introspection layer that enables an analysis algorithm to be programmed against the state of the recorded system through familiar terms of source-level variable and type names. To support performance analysis, the replay engine provides a faithful performance model of the original execution during replay

Break the dead end of dynamic slicing: localizing data and control omission bug

Dynamic slicing is a common way of identifying the root cause when a program fault is revealed. With the dynamic slicing technique, the programmers can follow data and control flow along the program execution trace to the root cause. However, the technique usually fails to work on omission bugs, i.e., the faults which are caused by missing executing some code. In many cases, dynamic slicing over-skips the root cause when an omission bug happens, leading the debugging process to a dead end. In this work, we conduct an empirical study on the omission bugs in the Defects4J bug repository. Our study shows that (1) omission bugs are prevalent (46.4%) among all the studied bugs; (2) there are repeating patterns on causes and fixes of the omission bugs; (3) the patterns of fixing omission bugs serve as a strong hint to break the slicing dead end. Based on our findings, we train a neural network model on the omission bugs in Defects4J repository to recommend where to approach when slicing can no long work. We conduct an experiment by applying our approach on 3193 mutated omission bugs which slicing fails to locate. The results show that our approach outperforms random benchmark on breaking the dead end and localizing the mutated omission bugs (63.8% over 2.8%).No Full Tex

Capture-based Automated Test Input Generation

Testing object-oriented software is critical because object-oriented languages have been commonly used in developing modern software systems. Many efficient test input generation techniques for object-oriented software have been proposed; however, state-of-the-art algorithms yield very low code coverage (e.g., less than 50%) on large-scale software. Therefore, one important and yet challenging problem is to generate desirable input objects for receivers and arguments that can achieve high code coverage (such as branch coverage) or help reveal bugs. Desirable objects help tests exercise the new parts of the code. However, generating desirable objects has been a significant challenge for automated test input generation tools, partly because the search space for such desirable objects is huge. To address this significant challenge, we propose a novel approach called Capture-based Automated Test Input Generation for Objected-Oriented Unit Testing (CAPTIG). The contributions of this proposed research are the following. First, CAPTIG enhances method-sequence generation techniques. Our approach intro-duces a set of new algorithms for guided input and method selection that increase code coverage. In addition, CAPTIG efficently reduces the amount of generated input. Second, CAPTIG captures objects dynamically from program execution during either system testing or real use. These captured inputs can support existing automated test input generation tools, such as a random testing tool called Randoop, to achieve higher code coverage. Third, CAPTIG statically analyzes the observed branches that had not been covered and attempts to exercise them by mutating existing inputs, based on the weakest precon-dition analysis. This technique also contributes to achieve higher code coverage. Fourth, CAPTIG can be used to reproduce software crashes, based on crash stack trace. This feature can considerably reduce cost for analyzing and removing causes of the crashes. In addition, each CAPTIG technique can be independently applied to leverage existing testing techniques. We anticipate our approach can achieve higher code coverage with a reduced duration of time with smaller amount of test input. To evaluate this new approach, we performed experiments with well-known large-scale open-source software and discovered our approach can help achieve higher code coverage with fewer amounts of time and test inputs

Automated Testing and Debugging for Big Data Analytics

The prevalence of big data analytics in almost every large-scale software system has generated a substantial push to build data-intensive scalable computing (DISC) frameworks such as Google MapReduce and Apache Spark that can fully harness the power of existing data centers. However, frameworks once used by domain experts are now being leveraged by data scientists, business analysts, and researchers. This shift in user demographics calls for immediate advancements in the development, debugging, and testing practices of big data applications, which are falling behind compared to the DISC framework design and implementation. In practice, big data applications often fail as users are unable to test all behaviors emerging from interleaving dataflow operators, user-defined functions, and framework's code. "Testing based on a random sample" rarely guarantees the reliability and "trial and error" and "print" debugging methods are expensive and time-consuming. Thus, the current practice of developing a big data application must be improved and the tools built to enhance the developer's productivity must adapt to the distinct characteristics of data-intensive scalable computing. By synthesizing ideas from software engineering and database systems, our hypothesis is that we can design effective and scalable testing and debugging algorithms for big data analytics without compromising the performance and efficiency of the underlying DISC framework. To design such techniques, we investigate how we can build interactive and responsive debugging primitives that significantly reduce the debugging time, yet do not pose much performance overhead on big data applications. Furthermore, we investigate how we can leverage data provenance techniques from databases and fault-isolation algorithms from software engineering to pinpoint the minimal subset of failure-inducing inputs efficiently. To improve the reliability of big data analytics, we investigate how we can abstract the semantics of dataflow operators and use them in tandem with the semantics of user-defined functions to generate a minimum set of synthetic test inputs capable of revealing more defects than the entire input dataset.To examine the first hypothesis, we introduce interactive, real-time debugging primitives for big data analytics through innovative and scalable debugging features such as simulated breakpoint, dynamic watchpoint, and crash culprit identification. Second, we design a new automated fault localization approach that combines insights from both the software engineering and database literature to bring delta debugging closer to a reality in the big data applications by leveraging data provenance and by constructing systems optimizations for debugging provenance queries. Lastly, we devise a new symbolic-execution based white-box testing algorithm for big data applications that abstracts the implementation of dataflow operators using logical specifications instead of modeling their implementations and combines them with the semantics of any arbitrary user-defined function. We instantiate the idea of an interactive debugging algorithm as BigDebug, the idea of an automated debugging algorithm as BigSift, and the idea of symbolic execution-based testing as BigTest. Our investigation shows that the interactive debugging primitives can scale to terabytes---our record-level tracing incurs less than 25% overhead on average and provides up to 100% time saving compared to the baseline replay debugger. Second, we observe that by combining data provenance with delta debugging, we can identify the minimum faulty input in just under 30% of the original job execution time. Lastly, we verify that by abstracting dataflow operators using logical specifications, we can efficiently generate the most concise test data suitable for local testing while revealing twice as many faults as prior approaches. Our investigations collectively demonstrate that developer productivity can be significantly improved through effective and scalable testing and debugging techniques for big data analytics, without impacting the DISC framework's performance. This dissertation affirms the feasibility of automated debugging and testing techniques for big data analytics---techniques that were previously considered infeasible for large-scale data processing

Semantic Analyses to Detect and Localize Software Regression Errors

Ph.DDOCTOR OF PHILOSOPH