Testing in the distributed test architecture: An extended abstract

Some systems interact with their environment at a number of physically distributed interfaces/ports and when testing such a system it is normal to place a local tester at each port. If the local testers cannot interact with one another and there is no global clock then we are testing in the distributed test architecture and this can introduce additional controllability and observability problems. While there has been interest in test generation algorithms that overcome controllability and observability problems, such algorithms lack generality since controllability and observability problems cannot always be overcome. In addition, traditionally only deterministic systems and models have been considered despite distributed systems often being non-deterministic. This paper describes recent work that characterized the power of testing in the distributed test architecture in the context of testing from a deterministic finite state machine and also work that investigated testing from a non-deterministic finite state machine and testing from an input output transition system. This work has the potential to lead to more general test generation algorithms for the distributed test architecture

Using status messages in the distributed test architecture

If the system under test has multiple interfaces/ports and these are physically distributed then in testing we place a tester at each port. If these testers cannot directly communicate with one another and there is no global clock then we are testing in the distributed test architecture. If the distributed test architecture is used then there may be input sequences that cannot be applied in testing without introducing controllability problems. Additionally, observability problems can allow fault masking. In this paper we consider the situation in which the testers can apply a status message: an input that causes the system under test to identify its current state. We show how such a status message can be used in order to overcome controllability and observability problems

Controllable testing from nondeterministic finite state machines with multiple ports

Copyright @ 2011 IEEESome systems have physically distributed interfaces, called ports, at which they interact with their environment. We place a tester at each port and if the testers cannot directly communicate and there is no global clock then we are using the distributed test architecture. It is known that this test architecture introduces controllability problems when testing from a deterministic finite state machine. This paper investigates the problem of testing from a nondeterministic finite state machine in the distributed test architecture and explores controllability. It shows how we can decide in polynomial time whether an input sequence is controllable. It also gives an algorithm for generating such an input sequence bar{x} and shows how we can produce testers that implement bar{x}

The effect of the distributed test architecture on the power of testing

Copyright @ 2008 Oxford University PressThere has been much interest in testing from finite-state machines (FSMs). If the system under test can be modelled by the (minimal) FSM N then testing from an (minimal) FSM M is testing to check that N is isomorphic to M. In the distributed test architecture, there are multiple interfaces/ports and there is a tester at each port. This can introduce controllability/synchronization and observability problems. This paper shows that the restriction to test sequences that do not cause controllability problems and the inability to observe the global behaviour in the distributed test architecture, and thus relying only on the local behaviour at remote testers, introduces fundamental limitations into testing. There exist minimal FSMs that are not equivalent, and so are not isomorphic, and yet cannot be distinguished by testing in this architecture without introducing controllability problems. Similarly, an FSM may have non-equivalent states that cannot be distinguished in the distributed test architecture without causing controllability problems: these are said to be locally s-equivalent and otherwise they are locally s-distinguishable. This paper introduces the notion of two states or FSMs being locally s-equivalent and formalizes the power of testing in the distributed test architecture in terms of local s-equivalence. It introduces a polynomial time algorithm that, given an FSM M, determines which states of M are locally s-equivalent and produces minimal length input sequences that locally s-distinguish states that are not locally s-equivalent. An FSM is locally s-minimal if it has no pair of locally s-equivalent states. This paper gives an algorithm that takes an FSM M and returns a locally s-minimal FSM M′ that is locally s-equivalent to M.This work was supported in part by Leverhulme Trust grant number F/00275/D, Testing State Based Systems, Natural Sciences and Engineering Research Council (NSERC) of Canada grant number RGPIN 976, and Engineering and Physical Sciences Research Council grant number GR/R43150, Formal Methods and Testing (FORTEST)

A CORBA-based system for testing distributed systems

Le test est considéré comme une des étapes du cycle de vie d'un logiciel, et la dernière phase de la méthodologie de création de logiciel (analyse, conception, développement, et test). Dans ce mémoire, nous contribuons à la phase de test. Nous définissons les systèmes repartis et étudions les méthodes et les architectures pour tester un système reparti, à savoir : l'architecture de test centralise, l'architecture de test réparti (ou distant), et l'architecture de test coordonné. Si l'architecture centralisée ne pose pas de problème particulier, l'architecture repartie cause plusieurs problèmes en terme de contrôlabilité et d'observabilité, qui sont des caractéristiques fondamentales du test de conformité. Après une présentation des problèmes de contrôlabilité et d'observabilité, nous proposons une solution a ces deux problèmes, qui consiste a utiliser une architecture de test coordonné. Ensuite, nous proposons et concevons une architecture de test coordonné constituée de trois parties : Ie contrôleur de test. Ie système de test, et l'implementation sous test. Ensuite, nous présentons CORBA (Common Object Request Broker Architecture), qui s'occupe de la communication entre les trois parties de notre architecture de test. Nous présentons une implémentation en Java et CORBA de notre architecture de test. Et enfin, nous illustrons l'application de notre architecture pour Ie test d'une version temporisée du protocole X.25.Abstract: Testing is considered as one of the steps in software life cycle and is the last phase in software creation methodology (Analysis, Design, Development, Testing). In this work, we contribute to testing phase. We define distributed systems, and study methods and architectures to test a distributed system, namely: centralized test architecture, distributed (or remote) test architecture, and coordinated test architecture. If the centralized architecture does not pose any particular problem, the distributed architecture raises several problems in terms of controllability and observability , which are fundamental features of conformance testing. After presenting controllability and observability problems, we propose a solution to these two problems, which consists of using a coordinated test architecture. Then, we propose and design a coordinated test architecture, consisting of three parts: Test Controller, Test System and Implementation Under Test. Then, we introduce CORBA (Common Object Request Broker Architecture), which is responsible for communications between the three parts of our test architecture. Then, we present an implementation in Java and CORBA of our test architecture. And finally, we illustrate the application of our architecture for testing a timed version of the X.25 protocol

Performance Testing of Distributed Component Architectures

Performance characteristics, such as response time, throughput andscalability, are key quality attributes of distributed applications. Current practice,however, rarely applies systematic techniques to evaluate performance characteristics.We argue that evaluation of performance is particularly crucial in early developmentstages, when important architectural choices are made. At first glance, thiscontradicts the use of testing techniques, which are usually applied towards the endof a project. In this chapter, we assume that many distributed systems are builtwith middleware technologies, such as the Java 2 Enterprise Edition (J2EE) or theCommon Object Request Broker Architecture (CORBA). These provide servicesand facilities whose implementations are available when architectures are defined.We also note that it is the middleware functionality, such as transaction and persistenceservices, remote communication primitives and threading policy primitives,that dominates distributed system performance. Drawing on these observations, thischapter presents a novel approach to performance testing of distributed applications.We propose to derive application-specific test cases from architecture designs so thatthe performance of a distributed application can be tested based on the middlewaresoftware at early stages of a development process. We report empirical results thatsupport the viability of the approach

Canonical finite state machines for distributed systems

There has been much interest in testing from finite state machines (FSMs) as a result of their suitability for modelling or specifying state-based systems. Where there are multiple ports/interfaces a multi-port FSM is used and in testing, a tester is placed at each port. If the testers cannot communicate with one another directly and there is no global clock then we are testing in the distributed test architecture. It is known that the use of the distributed test architecture can affect the power of testing and recent work has characterised this in terms of local s-equivalence: in the distributed test architecture we can distinguish two FSMs, such as an implementation and a specification, if and only if they are not locally s-equivalent. However, there may be many FSMs that are locally s-equivalent to a given FSM and the nature of these FSMs has not been explored. This paper examines the set of FSMs that are locally s-equivalent to a given FSM M. It shows that there is a unique smallest FSM χmin(M) and a unique largest FSM χmax(M) that are locally s-equivalent to M. Here smallest and largest refer to the set of traces defined by an FSM and thus to its semantics. We also show that for a given FSM M the set of FSMs that are locally s-equivalent to M defines a bounded lattice. Finally, we define an FSM that, amongst all FSMs locally s-equivalent to M, has fewest states. We thus give three alternative canonical FSMs that are locally s-equivalent to an FSM M: one that defines the smallest set of traces, one that defines the largest set of traces, and one with fewest states. All three provide valuable information and the first two can be produced in time that is polynomial in terms of the number of states of M. We prove that the problem of finding an s-equivalent FSM with fewest states is NP-hard in general but can be solved in polynomial time for the special case where there are two ports

Minimizing coordination channels in distributed testing

Testing may be used to show that a system under test conforms to its specication. In the case of a distributed system, one may have to use a distributed test architecture, involving p testers in order to test the system under test. These p testers may under some circumstances have to coordinate their actions with each other using external coordination channels. This may require the use of up to p2 p unidirectional coordination channels in the test architecture, which can be an extensive and expensive setup. In this paper, we propose a method to generate checking sequences while minimizing the number of required coordination channels, by adapting existing methods that generate checking sequences to be applied in a centralized test architecture. We consider the case of unidirectional and bidirectional coordination channels, and the case of transitive coordination