Verifying and comparing finite state machines for systems that have distributed interfaces

This paper concerns state-based systems that interact with their environment at physically distributed interfaces, called ports. When such a system is used a projection of the global trace, a local trace, is observed at each port. As a result the environment has reduced observational power: the set of local traces observed need not define the global trace that occurred. We consider the previously defined implementation relation ⊆s and prove that it is undecidable whether N ⊆s M and so it is also undecidable whether testing can distinguishing two states or FSMs. We also prove that a form of model-checking is undecidable when we have distributed observations and give conditions under which N ⊆s M is decidable. We then consider implementation relation ⊆sk that concerns input sequences of length κ or less. If we place bounds on κ and the number of ports then we can decide N ⊆sk M in polynomial time but otherwise this problem is NP-hard

Using status messages in the distributed test architecture

If the system under test has multiple interfaces/ports and these are physically distributed then in testing we place a tester at each port. If these testers cannot directly communicate with one another and there is no global clock then we are testing in the distributed test architecture. If the distributed test architecture is used then there may be input sequences that cannot be applied in testing without introducing controllability problems. Additionally, observability problems can allow fault masking. In this paper we consider the situation in which the testers can apply a status message: an input that causes the system under test to identify its current state. We show how such a status message can be used in order to overcome controllability and observability problems

Testing from a nondeterministic finite state machine using adaptive state counting

The problem of generating a checking experiment from a nondeterministic finite state machine has been represented in terms of state counting. However, test techniques that use state counting traditionally produce preset test suites. This paper extends the notion of state counting in order to allow the input/output sequences observed in testing to be utilized: Adaptive state counting is introduced. The main benefit of the proposed approach is that it may result in a reduction in the size of the test suite used. An additional benefit is that, where a failure is observed, it is possible to terminate test generation at this point

Oracles for distributed testing

Copyright @ 2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.The problem of deciding whether an observed behaviour is acceptable is the oracle problem. When testing from a finite state machine (FSM) it is easy to solve the oracle problem and so it has received relatively little attention for FSMs. However, if the system under test has physically distributed interfaces, called ports, then in distributed testing we observe a local trace at each port and we compare the set of local traces with the set of allowed behaviours (global traces). This paper investigates the oracle problem for deterministic and non-deterministic FSMs and for two alternative definitions of conformance for distributed testing. We show that the oracle problem can be solved in polynomial time for the weaker notion of conformance but is NP-hard for the stronger notion of conformance, even if the FSM is deterministic. However, when testing from a deterministic FSM with controllable input sequences the oracle problem can be solved in polynomial time and similar results hold for nondeterministic FSMs. Thus, in some cases the oracle problem can be efficiently solved when using stronger notion of conformance and where this is not the case we can use the decision procedure for weaker notion of conformance as a sound approximation

FSM quasi-equivalence testing via reduction and observing absence

There has been significant interest in automatically generating test cases from a non-deterministic finite state machine (FSM). Most approaches check that the behaviours of the system under test (SUT) are allowed by the specification FSM; they therefore test for reduction. However, sometimes one wants all of the behaviours, and so features, of the specification to be implemented and then one is testing for equivalence. In this paper we first note that in order to test for equivalence one must effectively be able to observe the SUT not being able to produce an output $y$ in response to an input $x$ after trace $\sigma$; we model this as the absence of an output. We prove that the problem of testing for equivalence to FSM $M$ can be mapped to testing for reduction to an FSM $R(M)$ that extends $M$ with absences. Thus, one can use techniques developed for testing for reduction when testing for equivalence. We then consider the case where the specification is partial, generalising the result to quasi-equivalence. These results are proved for observable specifications and so we also show how a partial FSM can be mapped to an observable partial FSM from which we can test

Implementation relations for testing through asynchronous channels

This paper concerns testing from an input output transition system (IOTS) model of a system under test that interacts with its environment through asynchronous first in first out (FIFO) channels. It explores methods for analysing an IOTS without modelling the channels. If IOTS M produces sequence $\sigma$ then, since communications are asynchronous, output can be delayed and so a different sequence might be observed. Thus M defines a language Tr(M) of sequences that can be observed when interacting with M through FIFO channels. We define implementation relations and equivalences in terms of Tr(M): an implementation relation says how IOTS N must relate to IOTS M in order for N to be a correct implementation of M. It is important to use an appropriate implementation relation since otherwise the verdict from a test run might be incorrect and because it influences test generation. It is undecidable whether IOTS N conforms to IOTS M and so also whether there is a test case that can distinguish between two IOTSs. We also investigate the situation in which we have a finite automaton P and either wish to know whether $Tr(M) \cap L(P)$ is empty or whether Tr(M) \cap \tr(P) is empty and prove that these are undecidable. In addition, we give conditions under which conformance and intersection are decidable.This work was partially supported by EPSRC grant EP/G04354X/1:The Birth, Life and Death of Semantic Mutants

The complexity of asynchronous model based testing

This is the post-print version of the final paper published in Theoretical Computer Science. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2012 Elsevier B.V.In model based testing (MBT), testing is based on a model MM that typically is expressed using a state-based language such as an input output transition system (IOTS). Most approaches to MBT assume that communications between the system under test (SUT) and its environment are synchronous. However, many systems interact with their environment through asynchronous channels and the presence of such channels changes the nature of testing. In this paper we investigate the situation in which the SUT interacts with its environment through asynchronous channels and the problems of producing test cases to reach a state, execute a transition, or to distinguish two states. In addition, we investigate the Oracle Problem. All four problems are explored for both FIFO and non-FIFO channels. It is known that the Oracle Problem can be solved in polynomial time for FIFO channels but we also show that the three test case generation problems can also be solved in polynomial time in the case where the IOTS is observable but the general test generation problems are EXPTIME-hard. For non-FIFO channels we prove that all of the test case generation problems are EXPTIME-hard and the Oracle Problem in NP-hard, even if we restrict attention to deterministic IOTSs