Checking experiments for stream X-machines

This article is a post-print version of the published article which may be accessed at the link below. Copyright © 2010 Elsevier B.V. All rights reserved.Stream X-machines are a state based formalism that has associated with it a particular development process in which a system is built from trusted components. Testing thus essentially checks that these components have been combined in a correct manner and that the orders in which they can occur are consistent with the specification. Importantly, there are test generation methods that return a checking experiment: a test that is guaranteed to determine correctness as long as the implementation under test (IUT) is functionally equivalent to an unknown element of a given fault domain Ψ. Previous work has show how three methods for generating checking experiments from a finite state machine (FSM) can be adapted to testing from a stream X-machine. However, there are many other methods for generating checking experiments from an FSM and these have a variety of benefits that correspond to different testing scenarios. This paper shows how any method for generating a checking experiment from an FSM can be adapted to generate a checking experiment for testing an implementation against a stream X-machine. This is the case whether we are testing to check that the IUT is functionally equivalent to a specification or we are testing to check that every trace (input/output sequence) of the IUT is also a trace of a nondeterministic specification. Interestingly, this holds even if the fault domain Ψ used is not that traditionally associated with testing from a stream X-machine. The results also apply for both deterministic and nondeterministic implementations

Implementation relations for testing through asynchronous channels

This paper concerns testing from an input output transition system (IOTS) model of a system under test that interacts with its environment through asynchronous first in first out (FIFO) channels. It explores methods for analysing an IOTS without modelling the channels. If IOTS M produces sequence $\sigma$ then, since communications are asynchronous, output can be delayed and so a different sequence might be observed. Thus M defines a language Tr(M) of sequences that can be observed when interacting with M through FIFO channels. We define implementation relations and equivalences in terms of Tr(M): an implementation relation says how IOTS N must relate to IOTS M in order for N to be a correct implementation of M. It is important to use an appropriate implementation relation since otherwise the verdict from a test run might be incorrect and because it influences test generation. It is undecidable whether IOTS N conforms to IOTS M and so also whether there is a test case that can distinguish between two IOTSs. We also investigate the situation in which we have a finite automaton P and either wish to know whether $Tr(M) \cap L(P)$ is empty or whether Tr(M) \cap \tr(P) is empty and prove that these are undecidable. In addition, we give conditions under which conformance and intersection are decidable.This work was partially supported by EPSRC grant EP/G04354X/1:The Birth, Life and Death of Semantic Mutants

Avoiding coincidental correctness in boundary value analysis

In partition analysis we divide the input domain to form subdomains on which the system's behaviour should be uniform. Boundary value analysis produces test inputs near each subdomain's boundaries to find failures caused by incorrect implementation of the boundaries. However, boundary value analysis can be adversely affected by coincidental correctness---the system produces the expected output, but for the wrong reason. This article shows how boundary value analysis can be adapted in order to reduce the likelihood of coincidental correctness. The main contribution is to cases of automated test data generation in which we cannot rely on the expertise of a tester

Controllable testing from nondeterministic finite state machines with multiple ports

Copyright @ 2011 IEEESome systems have physically distributed interfaces, called ports, at which they interact with their environment. We place a tester at each port and if the testers cannot directly communicate and there is no global clock then we are using the distributed test architecture. It is known that this test architecture introduces controllability problems when testing from a deterministic finite state machine. This paper investigates the problem of testing from a nondeterministic finite state machine in the distributed test architecture and explores controllability. It shows how we can decide in polynomial time whether an input sequence is controllable. It also gives an algorithm for generating such an input sequence bar{x} and shows how we can produce testers that implement bar{x}

Oracles for distributed testing

Copyright @ 2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.The problem of deciding whether an observed behaviour is acceptable is the oracle problem. When testing from a finite state machine (FSM) it is easy to solve the oracle problem and so it has received relatively little attention for FSMs. However, if the system under test has physically distributed interfaces, called ports, then in distributed testing we observe a local trace at each port and we compare the set of local traces with the set of allowed behaviours (global traces). This paper investigates the oracle problem for deterministic and non-deterministic FSMs and for two alternative definitions of conformance for distributed testing. We show that the oracle problem can be solved in polynomial time for the weaker notion of conformance but is NP-hard for the stronger notion of conformance, even if the FSM is deterministic. However, when testing from a deterministic FSM with controllable input sequences the oracle problem can be solved in polynomial time and similar results hold for nondeterministic FSMs. Thus, in some cases the oracle problem can be efficiently solved when using stronger notion of conformance and where this is not the case we can use the decision procedure for weaker notion of conformance as a sound approximation

Verdict functions in testing with a fault domain or test hypotheses

In state based testing it is common to include verdicts within test cases, the result of the test case being the verdict reached by the test run. In addition, approaches that reason about test effectiveness or produce tests that are guaranteed to find certain classes of faults are often based on either a fault domain or a set of test hypotheses. This paper considers how the presence of a fault domain or test hypotheses affects our notion of a test verdict. The analysis reveals the need for new verdicts that provide more information than the current verdicts and for verdict functions that return a verdict based on a set of test runs rather than a single test run. The concepts are illustrated in the contexts of testing from a non-deterministic finite state machine and the testing of a datatype specified using an algebraic specification language but are potentially relevant whenever fault domains or test hypotheses are used

Verifying and comparing finite state machines for systems that have distributed interfaces

This paper concerns state-based systems that interact with their environment at physically distributed interfaces, called ports. When such a system is used a projection of the global trace, a local trace, is observed at each port. As a result the environment has reduced observational power: the set of local traces observed need not define the global trace that occurred. We consider the previously defined implementation relation ⊆s and prove that it is undecidable whether N ⊆s M and so it is also undecidable whether testing can distinguishing two states or FSMs. We also prove that a form of model-checking is undecidable when we have distributed observations and give conditions under which N ⊆s M is decidable. We then consider implementation relation ⊆sk that concerns input sequences of length κ or less. If we place bounds on κ and the number of ports then we can decide N ⊆sk M in polynomial time but otherwise this problem is NP-hard

On the testability of SDL specifications

The problem of testing from an SDL specification is often complicated by the presence of infeasible paths. This paper introduces an approach for transforming a class of SDL specification in order to eliminate or reduce the infeasible path problem. This approach is divided into two phases in order to aid generality. First the SDL specification is rewritten to create a normal form extended finite state machine (NF-EFSM). This NF-EFSM is then expanded in order to produce a state machine in which the test criterion may be satisfied using paths that are known to be feasible. The expansion process is guaranteed to terminate. Where the expansion process may lead to an excessively large state machine, this process may be terminated early and feasible paths added. The approach is illustrated through being applied to the Initiator process of the Inres protocol

Applying adaptive test cases to nondeterministic implementations

The testing of a state-based system involves the application of sequences of inputs and the observation of the resultant input/output sequences (traces). These traces can result from preset input sequences or adaptive test cases in which the choice of the next input depends on the trace that has observed up to that input. Adaptive test cases are used in a number of areas including protocol conformance testing and adaptivity forms the basis of the standardised test language TTCN. Suppose that we apply adaptive test case ° to the system under test (SUT) and observe the trace ¯¾. If the SUT is deterministic and we apply ° again, after resetting the SUT, then we will observe ¯¾ again. Further, if we have another adaptive test case °0 where a prefix ¯¾0 of ¯¾ is a possible response to °0 then we know that the application of °0 must lead to ¯¾0. Thus, for a deterministic SUT the response of the SUT to an adaptive test case °0 might be deduced from the response of the SUT to another adaptive test case. This observation can be used to reduce the cost of testing: we only apply adaptive test case °0 if we cannot deduce the response to °0 from the set of observations. While many systems are deterministic, nondeterminism is becoming increasingly common. Nondeterminism in the SUT is typically a consequence of limits in the ability to observe the SUT. For example, it could be a result of information hiding, real time properties, or of different possible interleavings in a concurrent system (see, for example. This paper investigates the case where the SUT is nondeterministic. We consider the situation in which a set O of traces has been observed in testing and we are considering applying an adaptive test case °. In general we cannot expect to be able to deduce the response of a nondeterministic SUT to an adaptive test case ° since there may be more than one possible response. Instead we consider the question of how we can decide whether the application of ° could lead to a trace that has not been observed. A solution to this would allow us to reduce the cost of testing: if all possible responses of the SUT to ° have already been observed then we do not have to apply ° in testing and thus reduce the cost of test execution. This paper considers three cases. Section 3 considers the case where we can apply a fairness assumption. Section 4 weakens this assumption to us having a lower bound p on the probability of observing alternative responses of the SUT to any input and in any state. Section 5 then considers the general case