Combining centralised and distributed testing

Many systems interact with their environment at distributed interfaces (ports) and sometimes it is not possible to place synchronised local testers at the ports of the system under test (SUT). There are then two main approaches to testing: having independent local testers or a single centralised tester that interacts asynchronously with the SUT. The power of using independent testers has been captured using implementation relation \dioco. In this paper we define implementation relation \diococ for the centralised approach and prove that \dioco and \diococ are incomparable. This shows that the frameworks detect different types of faults and so we devise a hybrid framework and define an implementation relation \diocos for this. We prove that the hybrid framework is more powerful than the distributed and centralised approaches. We then prove that the Oracle problem is NP-complete for \diococ and \diocos but can be solved in polynomial time if we place an upper bound on the number of ports. Finally, we consider the problem of deciding whether there is a test case that is guaranteed to force a finite state model into a particular state or to distinguish two states, proving that both problems are undecidable for the centralised and hybrid frameworks

Action Refinement as an Implementation Relation

We propose a theory of process refinement which relates behavioural descriptions belonging to conceptually different abstraction levels, through a so-called vertical implementation relation. The theory is based on action refinement, which permits to relate abstract actions of the implementation to concrete computations of the implementation; it is developed in the standard interleaving approach. A number of proof rules is shown to be sound for the particular vertical implementation relation (based on observation congruence) we study in this paper. We give an illustrative example

Testing from a stochastic timed system with a fault model

In this paper we present a method for testing a system against a non-deterministic stochastic finite state machine. As usual, we assume that the functional behaviour of the system under test (SUT) is deterministic but we allow the timing to be non-deterministic. We extend the state counting method of deriving tests, adapting it to the presence of temporal requirements represented by means of random variables. The notion of conformance is introduced using an implementation relation considering temporal aspects and the limitations imposed by a black-box framework. We propose an algorithm for generating a test suite that determines the conformance of a deterministic SUT with respect to a non-deterministic specification. We show how previous work on testing from stochastic systems can be encoded into the framework presented in this paper as an instantiation of our parameterized implementation relation. In this setting, we use a notion of conformance up to a given confidence level

Verifying and comparing finite state machines for systems that have distributed interfaces

This paper concerns state-based systems that interact with their environment at physically distributed interfaces, called ports. When such a system is used a projection of the global trace, a local trace, is observed at each port. As a result the environment has reduced observational power: the set of local traces observed need not define the global trace that occurred. We consider the previously defined implementation relation ⊆s and prove that it is undecidable whether N ⊆s M and so it is also undecidable whether testing can distinguishing two states or FSMs. We also prove that a form of model-checking is undecidable when we have distributed observations and give conditions under which N ⊆s M is decidable. We then consider implementation relation ⊆sk that concerns input sequences of length κ or less. If we place bounds on κ and the number of ports then we can decide N ⊆sk M in polynomial time but otherwise this problem is NP-hard

Test Derivation from Timed Automata

A real-time system is a discrete system whose state changes occur in real-numbered time [AH97]. For testing real-time systems, specification languages must be extended with constructs for expressing real-time constraints, the implementation relation must be generalized to consider the temporal dimension, and the data structures and algorithms used to generate tests must be revised to operate on a potentially infinite set of states

A more precise implementation relation for distributed testing

There has been significant interest in distributed testing from an input output transition system. Previous work introduced an implementation relation \dioco that was defined in terms of an equivalence relation on traces (sequences of observations). This paper considers an alternative approach in which an observation made in testing is a tuple of local traces, one for each tester. This paper defines such an implementation relation \newdioco in terms of the possible observations regarding the system under test and the specification. It shows that \newdioco is strictly weaker than \dioco but is equivalent to \dioco if processes cannot be output-divergent. Interestingly, this shows that the previous definition of \dioco is too strong for output-divergent processes. We also prove that the Oracle problem is NP-complete but can be solved in polynomial time if there is an upper bound on the number of local testers

Implementation relations for testing through asynchronous channels

This paper concerns testing from an input output transition system (IOTS) model of a system under test that interacts with its environment through asynchronous first in first out (FIFO) channels. It explores methods for analysing an IOTS without modelling the channels. If IOTS M produces sequence $\sigma$ then, since communications are asynchronous, output can be delayed and so a different sequence might be observed. Thus M defines a language Tr(M) of sequences that can be observed when interacting with M through FIFO channels. We define implementation relations and equivalences in terms of Tr(M): an implementation relation says how IOTS N must relate to IOTS M in order for N to be a correct implementation of M. It is important to use an appropriate implementation relation since otherwise the verdict from a test run might be incorrect and because it influences test generation. It is undecidable whether IOTS N conforms to IOTS M and so also whether there is a test case that can distinguish between two IOTSs. We also investigate the situation in which we have a finite automaton P and either wish to know whether $Tr(M) \cap L(P)$ is empty or whether Tr(M) \cap \tr(P) is empty and prove that these are undecidable. In addition, we give conditions under which conformance and intersection are decidable.This work was partially supported by EPSRC grant EP/G04354X/1:The Birth, Life and Death of Semantic Mutants

A test generation framework for quiescent real-time systems

We present an extension of Tretmans theory and algorithm for test generation for input-output transition systems to real-time systems. Our treatment is based on an operational interpretation of the notion of quiescence in the context of real-time behaviour. This gives rise to a family of implementation relations parameterized by observation durations for quiescence. We define a nondeterministic (parameterized) test generation algorithm that generates test cases that are sound with respect to the corresponding implementation relation. Also, the test generation is exhaustive in the sense that for each non-conforming implementation a test case can be generated that detects the non-conformance

Preserving Liveness Guarantees from Synchronous Communication to Asynchronous Unstructured Low-Level Languages

In the implementation of abstract synchronous communication in asynchronous unstructured low-level languages, e.g. using shared variables, the preservation of safety and especially liveness properties is a hitherto open problem due to inherently different abstraction levels. Our approach to overcome this problem is threefold: First, we present our notion of handshake refinement with which we formally prove the correctness of the implementation relation of a handshake protocol. Second, we verify the soundness of our handshake refinement, i.e., all safety and liveness properties are preserved to the lower level. Third, we apply our handshake refinement to show the correctness of all implementations that realize the abstract synchronous communication with the handshake protocol. To this end, we employ an exemplary language with asynchronous shared variable communication. Our approach is scalable and closes the verification gap between different abstraction levels of communication

Vertical Implementation

We investigate criteria to relate specifications and implementations belonging to conceptually different levels of abstraction. For this purpose, we introduce the generic concept of a vertical implementation relation, which is a family of binary relations indexed by a refinement function that maps abstract actions onto concrete processes and thus determines the basic connection between the abstraction levels. If the refinement function is the identity, the vertical implementation relation collapses to a standard (horizontal) implementation relation. As desiderata for vertical implementation relations we formulate a number of congruence-like proof rules (notably a structural rule for recursion) that offer a powerful, compositional proof technique for vertical implementation. As a candidate vertical implementation relation we propose vertical bisimulation. Vertical bisimulation is compatible with the standard interleaving semantics of process algebra; in fact, the corresponding horizontal relation is rooted weak bisimulation. We prove that vertical bisimulation satisfies the proof rules for vertical implementation, thus establishing the consistency of the rules. Moreover, we define a corresponding notion of abstraction that strengthens the intuition behind vertical bisimulation and also provides a decision algorithm for finite-state systems. Finally, we give a number of small examples to demonstrate the advantages of vertical implementation in general and vertical bisimulation in particular.\u