The effect of the distributed test architecture on the power of testing

Copyright @ 2008 Oxford University PressThere has been much interest in testing from finite-state machines (FSMs). If the system under test can be modelled by the (minimal) FSM N then testing from an (minimal) FSM M is testing to check that N is isomorphic to M. In the distributed test architecture, there are multiple interfaces/ports and there is a tester at each port. This can introduce controllability/synchronization and observability problems. This paper shows that the restriction to test sequences that do not cause controllability problems and the inability to observe the global behaviour in the distributed test architecture, and thus relying only on the local behaviour at remote testers, introduces fundamental limitations into testing. There exist minimal FSMs that are not equivalent, and so are not isomorphic, and yet cannot be distinguished by testing in this architecture without introducing controllability problems. Similarly, an FSM may have non-equivalent states that cannot be distinguished in the distributed test architecture without causing controllability problems: these are said to be locally s-equivalent and otherwise they are locally s-distinguishable. This paper introduces the notion of two states or FSMs being locally s-equivalent and formalizes the power of testing in the distributed test architecture in terms of local s-equivalence. It introduces a polynomial time algorithm that, given an FSM M, determines which states of M are locally s-equivalent and produces minimal length input sequences that locally s-distinguish states that are not locally s-equivalent. An FSM is locally s-minimal if it has no pair of locally s-equivalent states. This paper gives an algorithm that takes an FSM M and returns a locally s-minimal FSM M′ that is locally s-equivalent to M.This work was supported in part by Leverhulme Trust grant number F/00275/D, Testing State Based Systems, Natural Sciences and Engineering Research Council (NSERC) of Canada grant number RGPIN 976, and Engineering and Physical Sciences Research Council grant number GR/R43150, Formal Methods and Testing (FORTEST)

Testing in the distributed test architecture: An extended abstract

Some systems interact with their environment at a number of physically distributed interfaces/ports and when testing such a system it is normal to place a local tester at each port. If the local testers cannot interact with one another and there is no global clock then we are testing in the distributed test architecture and this can introduce additional controllability and observability problems. While there has been interest in test generation algorithms that overcome controllability and observability problems, such algorithms lack generality since controllability and observability problems cannot always be overcome. In addition, traditionally only deterministic systems and models have been considered despite distributed systems often being non-deterministic. This paper describes recent work that characterized the power of testing in the distributed test architecture in the context of testing from a deterministic finite state machine and also work that investigated testing from a non-deterministic finite state machine and testing from an input output transition system. This work has the potential to lead to more general test generation algorithms for the distributed test architecture

Controllable testing from nondeterministic finite state machines with multiple ports

Copyright @ 2011 IEEESome systems have physically distributed interfaces, called ports, at which they interact with their environment. We place a tester at each port and if the testers cannot directly communicate and there is no global clock then we are using the distributed test architecture. It is known that this test architecture introduces controllability problems when testing from a deterministic finite state machine. This paper investigates the problem of testing from a nondeterministic finite state machine in the distributed test architecture and explores controllability. It shows how we can decide in polynomial time whether an input sequence is controllable. It also gives an algorithm for generating such an input sequence bar{x} and shows how we can produce testers that implement bar{x}

Oracles for distributed testing

Copyright @ 2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.The problem of deciding whether an observed behaviour is acceptable is the oracle problem. When testing from a finite state machine (FSM) it is easy to solve the oracle problem and so it has received relatively little attention for FSMs. However, if the system under test has physically distributed interfaces, called ports, then in distributed testing we observe a local trace at each port and we compare the set of local traces with the set of allowed behaviours (global traces). This paper investigates the oracle problem for deterministic and non-deterministic FSMs and for two alternative definitions of conformance for distributed testing. We show that the oracle problem can be solved in polynomial time for the weaker notion of conformance but is NP-hard for the stronger notion of conformance, even if the FSM is deterministic. However, when testing from a deterministic FSM with controllable input sequences the oracle problem can be solved in polynomial time and similar results hold for nondeterministic FSMs. Thus, in some cases the oracle problem can be efficiently solved when using stronger notion of conformance and where this is not the case we can use the decision procedure for weaker notion of conformance as a sound approximation

Canonical finite state machines for distributed systems

There has been much interest in testing from finite state machines (FSMs) as a result of their suitability for modelling or specifying state-based systems. Where there are multiple ports/interfaces a multi-port FSM is used and in testing, a tester is placed at each port. If the testers cannot communicate with one another directly and there is no global clock then we are testing in the distributed test architecture. It is known that the use of the distributed test architecture can affect the power of testing and recent work has characterised this in terms of local s-equivalence: in the distributed test architecture we can distinguish two FSMs, such as an implementation and a specification, if and only if they are not locally s-equivalent. However, there may be many FSMs that are locally s-equivalent to a given FSM and the nature of these FSMs has not been explored. This paper examines the set of FSMs that are locally s-equivalent to a given FSM M. It shows that there is a unique smallest FSM χmin(M) and a unique largest FSM χmax(M) that are locally s-equivalent to M. Here smallest and largest refer to the set of traces defined by an FSM and thus to its semantics. We also show that for a given FSM M the set of FSMs that are locally s-equivalent to M defines a bounded lattice. Finally, we define an FSM that, amongst all FSMs locally s-equivalent to M, has fewest states. We thus give three alternative canonical FSMs that are locally s-equivalent to an FSM M: one that defines the smallest set of traces, one that defines the largest set of traces, and one with fewest states. All three provide valuable information and the first two can be produced in time that is polynomial in terms of the number of states of M. We prove that the problem of finding an s-equivalent FSM with fewest states is NP-hard in general but can be solved in polynomial time for the special case where there are two ports

Using schedulers to test probabilistic distributed systems

This is the author's accepted manuscript. The final publication is available at Springer via http://dx.doi.org/10.1007/s00165-012-0244-5. Copyright © 2012, British Computer Society.Formal methods are one of the most important approaches to increasing the confidence in the correctness of software systems. A formal specification can be used as an oracle in testing since one can determine whether an observed behaviour is allowed by the specification. This is an important feature of formal testing: behaviours of the system observed in testing are compared with the specification and ideally this comparison is automated. In this paper we study a formal testing framework to deal with systems that interact with their environment at physically distributed interfaces, called ports, and where choices between different possibilities are probabilistically quantified. Building on previous work, we introduce two families of schedulers to resolve nondeterministic choices among different actions of the system. The first type of schedulers, which we call global schedulers, resolves nondeterministic choices by representing the environment as a single global scheduler. The second type, which we call localised schedulers, models the environment as a set of schedulers with there being one scheduler for each port. We formally define the application of schedulers to systems and provide and study different implementation relations in this setting

Fifty Shades of Grey in SOA Testing

Abstract-Testing is undisputedly a fundamental verification principle in the software landscape. Today's products require us to effectively handle and test huge, complex systems and in this context to tackle challenging traits like heterogeneity, distribution and controllability to name just a few. The advent of ServiceOriented Architectures with their inherent technological features like dynamics and heterogeneity exacerbated faced challenges, requiring us to evolve our technology. The traditional view of white or black box testing, for example, does not accommodate the multitude of shades of grey one should be able to exploit effectively for system-wide tests. Today, while there are a multitude of approaches for testing single services, there is still few work on methodological system tests for SOAs. In this paper we propose a corresponding workflow for tackling SOA testing and diagnosis, discuss SOA test case generation in more detail, and report preliminary research in that direction

Reaching and distinguishing states of distributed systems

Some systems interact with their environment at physically distributed interfaces, called ports, and in testing such a system it is normal to place a tester at each port. Each tester observes only the events at its port and it is known that this limited observational power introduces additional controllability and observability problems into testing. Given a multiport finite state machine (FSM) $M$, we consider the problems of defining strategies for the testers either to reach a given state of $M$ or to distinguish two states of $M$. These are important problems since most techniques for testing from a single-port FSM use sequences that reach and distinguish states. Both problems can be solved in low-order polynomial time for single-port FSMs but we prove that the corresponding decision problems are undecidable for multiport FSMs. However, we also show that they can be solved in low-order polynomial times for deterministic FSMs if we restrict our attention to controllable tests. These results have important ramifications for testing from a multiport FSM since they suggest that methods for testing from a single-port FSM cannot be easily adapted. In addition, two FSMs can be distinguished if and only if their initial states can be distinguished and so the results suggest that, in contrast to single-port FSMs, we cannot expect to produce general complete test generation methods for multiport FSMs

Improvements in finite state machines

Finite State Machine (FSM) based testing methods have a history of over half a century, starting in 1956 with the works on machine identi cation. This was then followed by works checking the conformance of a given implementation to a given speci cation. When it is possible to identify the states of an FSM using an appropriate input sequence, it's been long known that it is possible to generate a Fault Detection Experiment with fault coverage with respect to a certain fault model in polynomial time. In this thesis, we investigate two notions of fault detection sequences; Checking Sequence (CS), Checking Experiment (CE). Since a fault detection sequence (either a CS or a CE) is constructed once but used many times, the importance of having short fault detection sequences is obvious and hence recent works in this eld aim to generate shorter fault detection sequences. In this thesis, we rst investigate a strategy and related problems to reduce the length of a CS. A CS consists several components such as Reset Sequences and State Identi - cation Sequences. All works assume that for a given FSM, a reset sequence and a state identi cation sequence are also given together with the speci cation FSM M. Using the given reset and state identi cation sequences, a CS is formed that gives full fault coverage under certain assumptions. In other words, any faulty implementation N can be identi ed by using this test sequence. In the literature, di erent methods for CS construction take di erent approaches to put these components together, with the aim of coming up with a shorter CS incorporating all of these components. One obvious way of keeping the CS short is to keep components short. As the reset sequence and the state identi cation sequence are the biggest components, having short reset and state identi cation sequences is very important as well. It was shown in 1991 that for a given FSM M, shortest reset sequence cannot be computed in polynomial time if P 6≠NP. Recently it was shown that when the FSM has particular type (\monotonic") of transition structure, constructing one of the shortest reset word is polynomial time solvable. However there has been no work on constructing one of the shortest reset word for a monotonic partially speci ed machines. In this thesis, we showed that this problem is NP-hard. On the other hand, in 1994 it was shown that one can check if M has special type of state identi cation sequence (known as an adaptive distinguishing sequence) in polynomial time. The same work also suggests a polynomial time algorithm to construct a state identi cation sequence when one exists. However, this algorithm generates a state identi cation sequence without any particular emphasis on generating a short one. There has been no work on the generation of state identi cation sequences for complete or partial machines after this work. In this thesis, we showed that construction of short state identi cation sequences is NP-complete and NP-hard to approximate. We propose methods of generating short state identi cation sequences and experimentally validate that such state identi cation sequences can reduce the length of fault detection sequences by 29:2% on the average. Another line of research, in this thesis, devoted for reducing the cost of checking experiments. A checking experiment consist of a set of input sequences each of which aim to test di erent properties of the implementation. As in the case of CSs, a large portion of these input sequences contain state identi cation sequences. There are several kinds of state identi cation sequences that are applicable in CEs. In this work, we propose a new kind of state identi cation sequence and show that construction of such sequences are PSPACE-complete. We propose a heuristic and we perform experiments on benchmark FSMs and experimentally show that the proposed notion of state identi cation sequence can reduce the cost of CEs by 65% in the extreme case. Testing distributed architectures is another interesting eld for FSM based fault detection sequence generation. The additional challenge when such distributed architectures are considered is to generate a fault detection sequence which does not pose controllability or observability problem. Although the existing methods again assume that a state identi cation sequence is given using which a fault detection sequence is constructed, there is no work on how to generate a state identi cation sequence which do not have controllability/observability problem itself. In this thesis we investigate the computational complexities to generate such state identi cation sequences and show that no polynomial time algorithm can construct a state identi cation sequence for a given distributed FSM