Overcoming controllability problems in distributed testing from an input output transition system

This is the Pre-print version of the Article. The official published version can be accessed from the link below - Copyright @ 2012 Springer VerlagThis paper concerns the testing of a system with physically distributed interfaces, called ports, at which it interacts with its environment. We place a tester at each port and the tester at port p observes events at p only. This can lead to controllability problems, where the observations made by the tester at a port p are not sufficient for it to be able to know when to send an input. It is known that there are test objectives, such as executing a particular transition, that cannot be achieved if we restrict attention to test cases that have no controllability problems. This has led to interest in schemes where the testers at the individual ports send coordination messages to one another through an external communications network in order to overcome controllability problems. However, such approaches have largely been studied in the context of testing from a deterministic finite state machine. This paper investigates the use of coordination messages to overcome controllability problems when testing from an input output transition system and gives an algorithm for introducing sufficient messages. It also proves that the problem of minimising the number of coordination messages used is NP-hard

Testing in the distributed test architecture: An extended abstract

Some systems interact with their environment at a number of physically distributed interfaces/ports and when testing such a system it is normal to place a local tester at each port. If the local testers cannot interact with one another and there is no global clock then we are testing in the distributed test architecture and this can introduce additional controllability and observability problems. While there has been interest in test generation algorithms that overcome controllability and observability problems, such algorithms lack generality since controllability and observability problems cannot always be overcome. In addition, traditionally only deterministic systems and models have been considered despite distributed systems often being non-deterministic. This paper describes recent work that characterized the power of testing in the distributed test architecture in the context of testing from a deterministic finite state machine and also work that investigated testing from a non-deterministic finite state machine and testing from an input output transition system. This work has the potential to lead to more general test generation algorithms for the distributed test architecture

Controllable testing from nondeterministic finite state machines with multiple ports

Copyright @ 2011 IEEESome systems have physically distributed interfaces, called ports, at which they interact with their environment. We place a tester at each port and if the testers cannot directly communicate and there is no global clock then we are using the distributed test architecture. It is known that this test architecture introduces controllability problems when testing from a deterministic finite state machine. This paper investigates the problem of testing from a nondeterministic finite state machine in the distributed test architecture and explores controllability. It shows how we can decide in polynomial time whether an input sequence is controllable. It also gives an algorithm for generating such an input sequence bar{x} and shows how we can produce testers that implement bar{x}

Using status messages in the distributed test architecture

If the system under test has multiple interfaces/ports and these are physically distributed then in testing we place a tester at each port. If these testers cannot directly communicate with one another and there is no global clock then we are testing in the distributed test architecture. If the distributed test architecture is used then there may be input sequences that cannot be applied in testing without introducing controllability problems. Additionally, observability problems can allow fault masking. In this paper we consider the situation in which the testers can apply a status message: an input that causes the system under test to identify its current state. We show how such a status message can be used in order to overcome controllability and observability problems

Overcoming observability problems in distributed test architectures

This paper investigates conditions that must be satisfied by an FSM for the existence of input sequences that can be applied in a distributed test architecture without encountering controllability and observability problems and without using external coordination messages. Such conditions have two potential values. First, they can be used to determine whether we require coordination messages and thus a network that connects the testers. Second, if we wish to avoid the use of coordination messages in testing then these conditions can be seen as testability conditions that can inform the design process. Results given in this paper differ from those in the following ways. First, the conditions are strictly weaker than those in since we are less restrictive in the ways we achieve our goals. Second, only considered observability problems; we consider both controllability and observability problems. In addition, only considered a particular type of observability problem and we generalize this. Finally, we investigate the situation in which we need only add input sequences to complement a given test/checking sequence ρ and prove that the conditions for this problem are equivalent to those for the original problem

Checking sequences for distributed test architectures

Controllability and observability problems may manifest themselves during the application of a checking sequence in a test architecture where there are multiple remote testers. These problems often require the use of external coordination message exchanges among testers during testing. However, the use of coordination messages requires the existence of an external network that can increase the cost of testing and can be difficult to implement. In addition, the use of coordination messages introduces delays and this can cause problems where there are timing constraints. Thus, sometimes it is desired to construct a checking sequence from the specification of the system under test that will be free from controllability and observability problems without requiring the use of external coordination message exchanges. This paper gives conditions under which it is possible to produce such a checking sequence, using multiple distinguishing sequences, and an algorithm that achieves this

Overcoming controllability problems with fewest channels between testers

When testing a system that has multiple physically distributed ports/interfaces it is normal to place a tester at each port. Each tester observes only the events at its port and it is known that this can lead to additional controllability problems. While such controllability problems can be overcome by the exchange of external coordination messages between the testers, this requires the deployment of an external network and may thus increase the costs of testing. The problem studied in this paper is finding a minimum number of coordination channels to overcome controllability problems in distributed testing. Three instances of this problem are considered. The first problem is to find a minimum number of channels between testers in order to overcome the controllability problems in a given test sequence to be applied in testing. The second problem is finding a minimal set of channels that allow us to overcome controllability problems in any test sequence that may be selected from the specification of the system under test. The last problem is to find a test sequence that achieves a particular test objective and in doing so allows fewest channels to be used

Conformance relations for distributed testing based on CSP

Copyright @ 2011 Springer Berlin HeidelbergCSP is a well established process algebra that provides comprehensive theoretical and practical support for refinement-based design and verification of systems. Recently, a testing theory for CSP has also been presented. In this paper, we explore the problem of testing from a CSP specification when observations are made by a set of distributed testers. We build on previous work on input-output transition systems, but the use of CSP leads to significant differences, since some of its conformance (refinement) relations consider failures as well as traces. In addition, we allow events to be observed by more than one tester. We show how the CSP notions of refinement can be adapted to distributed testing. We consider two contexts: when the testers are entirely independent and when they can cooperate. Finally, we give some preliminary results on test-case generation and the use of coordination messages. © 2011 IFIP International Federation for Information Processing

Model-Based Testing for the Cloud

Software in the cloud is characterised by the need to be highly adaptive and continuously available. Incremental changes are applied to the deployed system and need to be tested in the field. Different configurations need to be tested. Higher quality standards regarding both functional and non-functional properties are put on those systems, as they often face large and diverse customer bases and/or are used as services from different peer service implementations. The properties of interest include interoperability, privacy, security, reliability, performance, resource use, timing constraints, service dependencies, availability, and so on. This paper discusses the state of the art in model-based testing of cloud systems. It focuses on two central aspects of the problem domain: (a) dealing with the adaptive and dynamic character of cloud software when tested with model-based testing, by developing new online and offline test strategies, and (b) dealing with the variety of modeling concerns for functional and non-functional properties, by devising a unified framework for them where this is possible. Having discussed the state of the art we identify challenges and future directions

Using schedulers to test probabilistic distributed systems

This is the author's accepted manuscript. The final publication is available at Springer via http://dx.doi.org/10.1007/s00165-012-0244-5. Copyright © 2012, British Computer Society.Formal methods are one of the most important approaches to increasing the confidence in the correctness of software systems. A formal specification can be used as an oracle in testing since one can determine whether an observed behaviour is allowed by the specification. This is an important feature of formal testing: behaviours of the system observed in testing are compared with the specification and ideally this comparison is automated. In this paper we study a formal testing framework to deal with systems that interact with their environment at physically distributed interfaces, called ports, and where choices between different possibilities are probabilistically quantified. Building on previous work, we introduce two families of schedulers to resolve nondeterministic choices among different actions of the system. The first type of schedulers, which we call global schedulers, resolves nondeterministic choices by representing the environment as a single global scheduler. The second type, which we call localised schedulers, models the environment as a set of schedulers with there being one scheduler for each port. We formally define the application of schedulers to systems and provide and study different implementation relations in this setting