Testing in the distributed test architecture: An extended abstract

Some systems interact with their environment at a number of physically distributed interfaces/ports and when testing such a system it is normal to place a local tester at each port. If the local testers cannot interact with one another and there is no global clock then we are testing in the distributed test architecture and this can introduce additional controllability and observability problems. While there has been interest in test generation algorithms that overcome controllability and observability problems, such algorithms lack generality since controllability and observability problems cannot always be overcome. In addition, traditionally only deterministic systems and models have been considered despite distributed systems often being non-deterministic. This paper describes recent work that characterized the power of testing in the distributed test architecture in the context of testing from a deterministic finite state machine and also work that investigated testing from a non-deterministic finite state machine and testing from an input output transition system. This work has the potential to lead to more general test generation algorithms for the distributed test architecture

Using status messages in the distributed test architecture

If the system under test has multiple interfaces/ports and these are physically distributed then in testing we place a tester at each port. If these testers cannot directly communicate with one another and there is no global clock then we are testing in the distributed test architecture. If the distributed test architecture is used then there may be input sequences that cannot be applied in testing without introducing controllability problems. Additionally, observability problems can allow fault masking. In this paper we consider the situation in which the testers can apply a status message: an input that causes the system under test to identify its current state. We show how such a status message can be used in order to overcome controllability and observability problems

Uncovering Bugs in Distributed Storage Systems during Testing (not in Production!)

Testing distributed systems is challenging due to multiple sources of nondeterminism. Conventional testing techniques, such as unit, integration and stress testing, are ineffective in preventing serious but subtle bugs from reaching production. Formal techniques, such as TLA+, can only verify high-level specifications of systems at the level of logic-based models, and fall short of checking the actual executable code. In this paper, we present a new methodology for testing distributed systems. Our approach applies advanced systematic testing techniques to thoroughly check that the executable code adheres to its high-level specifications, which significantly improves coverage of important system behaviors. Our methodology has been applied to three distributed storage systems in the Microsoft Azure cloud computing platform. In the process, numerous bugs were identified, reproduced, confirmed and fixed. These bugs required a subtle combination of concurrency and failures, making them extremely difficult to find with conventional testing techniques. An important advantage of our approach is that a bug is uncovered in a small setting and witnessed by a full system trace, which dramatically increases the productivity of debugging

Methodology for testing loss of mains detection algorithms for microgrids and distributed generation using real-time power hardware–in-the-loop based technique

The effective integration of distributed energy resources in distribution networks demands powerful simulation and test methods in order to determine both system and component behaviour, and understand their interaction. Unexpected disconnection of a significant volume of distributed generation (DG) could have potentially serious consequences for the entire system [1], this means DG sources can no longer be treated as purely negative load. This paper proposes a method of testing loss-of-mains (LOM) detection and protection schemes for distributed energy resources (DER) using real-time power hardware-in-the-loop (RT PHIL). The approach involves connecting the generator and interface under test (e.g. motor-generator set or inverter, controlled by an RTS – Real Time Station[3]) to a real-time simulator (an RTDS – Real Time Digital Simulator[2]) which simulates the local loads and upstream power system. This arrangement allows observation of the interaction with other controls in the network beyond the local microgrid area. These LOM schemes are of increasing importance because with growing penetration levels of distributed generation the network operator has less visibility and control of the connected generation. Furthermore when the generation and load in a particular network area are closely matched (e.g. a grid-connected microgrid), it becomes increasingly difficult to detect a loss of grid supply at the generator. This work builds upon the existing LOM testing methodology proposed in [4]. By utilising RT PHIL and a laboratory microgrid, the testing environment has been brought to a new level of functionality where system integrity can be more rigorously and realistically evaluated

Testing a distributed system: Generating minimal synchronised test sequences that detect output-shifting faults

A distributed system may have a number of separate interfaces called ports and in testing it may be necessary to have a separate tester at each port. This introduces a number of issues, including the necessity to use synchronised test sequences and the possibility that output-shifting faults go undetected. This paper considers the problem of generating a minimal synchronised test sequence that detects output-shifting faults when the system is specified using a finite state machine with multiple ports. The set of synchronised test sequences that detect output-shifting faults is represented by a directed graph G and test generation involves finding appropriate tours of G. This approach is illustrated using the test criterion that the test sequence contains a test segment for each transition

Overcoming controllability problems with fewest channels between testers

When testing a system that has multiple physically distributed ports/interfaces it is normal to place a tester at each port. Each tester observes only the events at its port and it is known that this can lead to additional controllability problems. While such controllability problems can be overcome by the exchange of external coordination messages between the testers, this requires the deployment of an external network and may thus increase the costs of testing. The problem studied in this paper is finding a minimum number of coordination channels to overcome controllability problems in distributed testing. Three instances of this problem are considered. The first problem is to find a minimum number of channels between testers in order to overcome the controllability problems in a given test sequence to be applied in testing. The second problem is finding a minimal set of channels that allow us to overcome controllability problems in any test sequence that may be selected from the specification of the system under test. The last problem is to find a test sequence that achieves a particular test objective and in doing so allows fewest channels to be used

A CORBA-based system for testing distributed systems

Le test est considéré comme une des étapes du cycle de vie d'un logiciel, et la dernière phase de la méthodologie de création de logiciel (analyse, conception, développement, et test). Dans ce mémoire, nous contribuons à la phase de test. Nous définissons les systèmes repartis et étudions les méthodes et les architectures pour tester un système reparti, à savoir : l'architecture de test centralise, l'architecture de test réparti (ou distant), et l'architecture de test coordonné. Si l'architecture centralisée ne pose pas de problème particulier, l'architecture repartie cause plusieurs problèmes en terme de contrôlabilité et d'observabilité, qui sont des caractéristiques fondamentales du test de conformité. Après une présentation des problèmes de contrôlabilité et d'observabilité, nous proposons une solution a ces deux problèmes, qui consiste a utiliser une architecture de test coordonné. Ensuite, nous proposons et concevons une architecture de test coordonné constituée de trois parties : Ie contrôleur de test. Ie système de test, et l'implementation sous test. Ensuite, nous présentons CORBA (Common Object Request Broker Architecture), qui s'occupe de la communication entre les trois parties de notre architecture de test. Nous présentons une implémentation en Java et CORBA de notre architecture de test. Et enfin, nous illustrons l'application de notre architecture pour Ie test d'une version temporisée du protocole X.25.Abstract: Testing is considered as one of the steps in software life cycle and is the last phase in software creation methodology (Analysis, Design, Development, Testing). In this work, we contribute to testing phase. We define distributed systems, and study methods and architectures to test a distributed system, namely: centralized test architecture, distributed (or remote) test architecture, and coordinated test architecture. If the centralized architecture does not pose any particular problem, the distributed architecture raises several problems in terms of controllability and observability , which are fundamental features of conformance testing. After presenting controllability and observability problems, we propose a solution to these two problems, which consists of using a coordinated test architecture. Then, we propose and design a coordinated test architecture, consisting of three parts: Test Controller, Test System and Implementation Under Test. Then, we introduce CORBA (Common Object Request Broker Architecture), which is responsible for communications between the three parts of our test architecture. Then, we present an implementation in Java and CORBA of our test architecture. And finally, we illustrate the application of our architecture for testing a timed version of the X.25 protocol

Performance Testing of Distributed Component Architectures

Performance characteristics, such as response time, throughput andscalability, are key quality attributes of distributed applications. Current practice,however, rarely applies systematic techniques to evaluate performance characteristics.We argue that evaluation of performance is particularly crucial in early developmentstages, when important architectural choices are made. At first glance, thiscontradicts the use of testing techniques, which are usually applied towards the endof a project. In this chapter, we assume that many distributed systems are builtwith middleware technologies, such as the Java 2 Enterprise Edition (J2EE) or theCommon Object Request Broker Architecture (CORBA). These provide servicesand facilities whose implementations are available when architectures are defined.We also note that it is the middleware functionality, such as transaction and persistenceservices, remote communication primitives and threading policy primitives,that dominates distributed system performance. Drawing on these observations, thischapter presents a novel approach to performance testing of distributed applications.We propose to derive application-specific test cases from architecture designs so thatthe performance of a distributed application can be tested based on the middlewaresoftware at early stages of a development process. We report empirical results thatsupport the viability of the approach