Leveraging Distributed Tracing and Container Cloning for Replay Debugging of Microservices

Microservice architectures have gained prominence in recent years for building large-scale industrial distributed systems. However, microservice architectures make the usage of replay debugging, a powerful technique for finding root causes of faults, very challenging because of the polyglot (written in several languages) services, large accumulated state of services, and tight latency limits imposed by long hop-chains. This work attempts to provide a framework for enabling replay debugging in production microservice applications. We study 25 real-world faults in microservice systems collected from diverse sources, categorize these faults by fault symptoms, and create 15 application agnostic mutation operators for microservices. We then propose a language agnostic replay debugging framework for microservice applications that uses a distributed tracing system to record network requests and enables replay of those requests on cloned service containers running in a debug environment. A key component of this framework is an anomaly detector that uses span-level and container-level monitoring to detect fault symptoms found in our study and localizes faults to trace level so that faulty traces can be easily replayed to find the root cause. An open-source microservices application injected successively with the mutation operators is used for an evaluation that shows that our framework is upto an order of magnitude lighter-weight than language-specific recording tools such as Chrome DevTools or VisualVM and can help in finding root causes of 9 out of 15 mutations at a line or function level

A study of testing a microservice system based on code coverage

The microservice architecture is a service-oriented architecture that supports the development of small loosely-coupled software services. In contrast to the monolithic architecture, the microservice architecture is increasingly popular among developers since it allows for an easier development process, reduces the barrier to adding new technologies to a system, and increases the efficiency of a system's scalability. Testing a microservice system with a test suite (i.e., a collection of test cases) is an important means for improving the quality of the microservice system. To guide the assessment and improvement of the test suite, code-coverage information is commonly used. However, there exists no research on investigating how code-coverage information can be used to measure and improve a test suite for a microservice system. To fill this gap, we conduct an empirical study on testing a benchmark microservice system by measuring code-coverage achieved by its test suite based on a coverage-measurement infrastructure that we build. The line-coverage results show that covering a major portion of the lines not being covered is blocked by error-handling branches (i.e., branches related to error handling). Based on the code-coverage information, we explore ways of augmenting the existing test suite to achieve higher code-coverage. In particular, guided by the code-coverage information, we augment the existing test suite by adding new test cases to aim for covering those functionalities whose related code portions are not covered by the existing test suite. In addition, to covering those error-handling branches, we augment the test suite by manipulating its execution environment, i.e., mutating the messages sent by a microservice to another microservice in order to aim for covering not-covered error-handling branches. Such coverage-guided test-suite augmentation helps achieve higher code-coverage of the microservice system