Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

A unified model for protocol test suite design

This thesis is concerned with developing new algorithms for solving some basic problems of conformance testing. In particular, the following problems of conformance testing are considered: (i) generation of test cases from Language Of Temporal Ordering Specification (LOTOS) and Specification and Description Language (SDL), (ii) selection of test cases which meet certain data flow coverage criteria, and (iii) representation of test cases for Local Single-layer (LS) and Remote Single-layer (RS) architectures. The algorithms presented in this thesis can be used to solve in an efficient manner these fundamental problems of conformance testing. These algorithms rely heavily on two concepts: the Extended Finite State Machine (EFSM) chart and the Input/Output (I/O) diagram. In this thesis, we introduce a unified model (using the EFSM chart and the I/O diagram) for existing protocol specification languages. Based on the new unified model, a conceptually simple, easy to implement and computationally efficient methodology is proposed in this thesis for studying conformance testing. In this thesis, the protocol specification is mapped into an EFSM chart. The structure of input/output data is modeled by hierarchical diagrams called I/O diagrams. Test cases are generated from the EFSM chart. Furthermore, a data flow graph is constructed from the chart, and used to identify the protocol functions for testing the data flow aspects of an Implementation Under Test (IUT). The zero-one integer programming technique is used to select test cases to meet the data flow coverage requirement. The selected test cases are modeled as a dependency graph and then evaluated by taking predicate slices from the test case dependency graph. Predicate slices are used to identify infeasible test cases that must be eliminated. Redundant assignments and predicates in all the feasible test cases are removed by reducing the test cases. Reduction is achieved by using the test case dependency graph as well as the data flow graph. The reduced test case dependency graph is adapted for LS and RS architectures. The tester's behaviour in each test case is obtained by a series of transformations called representation and selection. Test case representation refers to the steps of inverting the direction of events and the generation of base and dynamic constraints on the events. These constraints are generated in the form of an I/O diagram. Test case selection refers to the steps of assigning a test purpose according to the hierarchy of test cases in a test suite and then completing the tester's behaviour by assigning verdict and parameter value information

Enabling Proactive Adaptation through Just-in-time Testing of Conversational Services

Service-based applications (SBAs) will increasingly be composed of third-party services available over the Internet. Reacting to failures of those third-party services by dynamically adapting the SBAs will become a key enabler for ensuring reliability. Determining when to adapt an SBA is especially challenging in the presence of conversational (aka. stateful) services. A conversational service might fail in the middle of an invocation sequence, in which case adapting the SBA might be costly; e.g., due to the necessary state transfer to an alternative service. In this paper we propose just-in-time testing of conversational services as a novel approach to detect potential problems and to proactively trigger adaptations, thereby preventing costly compensation activities. The approach is based on a framework for online testing and a formal test-generation method which guarantees functional correctness for conversational services. The applicability of the approach is discussed with respect to its underlying assumptions and its performance. The benefits of the approach are demonstrated using a realistic example

Reducing the cost of applying adaptive test cases

The testing of a state-based system may involve the application of a number of adaptive test cases. Where the implementation under test (IUT) is deterministic, the response of the IUT to some adaptive test case $\gamma_1$ could be capable of determining the response of the IUT to another adaptive test case $\gamma_2". Thus, the expected cost of applying a set of adaptive test cases depends upon the order in which they are applied. This paper explores properties of adaptive test cases and considers the problem of finding an order of application of the elements from some set of adaptive test cases, which minimises the expected cost of testing

Estimating the feasibility of transition paths in extended finite state machines

There has been significant interest in automating testing on the basis of an extended finite state machine (EFSM) model of the required behaviour of the implementation under test (IUT). Many test criteria require that certain parts of the EFSM are executed. For example, we may want to execute every transition of the EFSM. In order to find a test suite (set of input sequences) that achieves this we might first derive a set of paths through the EFSM that satisfy the criterion using, for example, algorithms from graph theory. We then attempt to produce input sequences that trigger these paths. Unfortunately, however, the EFSM might have infeasible paths and the problem of determining whether a path is feasible is generally undecidable. This paper describes an approach in which a fitness function is used to estimate how easy it is to find an input sequence to trigger a given path through an EFSM. Such a fitness function could be used in a search-based approach in which we search for a path with good fitness that achieves a test objective, such as executing a particular transition, and then search for an input sequence that triggers the path. If this second search fails then we search for another path with good fitness and repeat the process. We give a computationally inexpensive approach (fitness function) that estimates the feasibility of a path. In order to evaluate this fitness function we compared the fitness of a path with the ease with which an input sequence can be produced using search to trigger the path and we used random sampling in order to estimate this. The empirical evidence suggests that a reasonably good correlation (0.72 and 0.62) exists between the fitness of a path, produced using the proposed fitness function, and an estimate of the ease with which we can randomly generate an input sequence to trigger the path

Testing from a stochastic timed system with a fault model

In this paper we present a method for testing a system against a non-deterministic stochastic finite state machine. As usual, we assume that the functional behaviour of the system under test (SUT) is deterministic but we allow the timing to be non-deterministic. We extend the state counting method of deriving tests, adapting it to the presence of temporal requirements represented by means of random variables. The notion of conformance is introduced using an implementation relation considering temporal aspects and the limitations imposed by a black-box framework. We propose an algorithm for generating a test suite that determines the conformance of a deterministic SUT with respect to a non-deterministic specification. We show how previous work on testing from stochastic systems can be encoded into the framework presented in this paper as an instantiation of our parameterized implementation relation. In this setting, we use a notion of conformance up to a given confidence level

Automatic generation of test sequences form EFSM models using evolutionary algorithms

Automated test data generation through evolutionary testing (ET) is a topic of interest to the software engineering community. While there are many ET-based techniques for automatically generating test data from code, the problem of generating test data from an extended finite state machine (EFSMs) is more complex and has received little attention. In this paper, we introduce a novel approach that addresses the problem of generating input test sequences that trigger given feasible paths in an EFSM model by employing an ET-based technique. The proposed approach expresses the problem as a search for input parameters to be applied to a set of functions to be called sequentially. In order to apply ET-based technique, a new fitness function is introduced to cope with the case when a test target involves calls to a set of transitions sequentially. We evaluate our approach empirically using five sets of randomly generated paths through two EFSM case studies: INRES and class 2 transport protocols. In the experiments, we apply two search techniques: a random and an ET-based which utilizes our new fitness function. Experimental results show that the proposed approach produces input test sequences that trigger all the feasible paths used with a success rate of 100%, however, the random technique failed in most cases with a success rate of 20.8%

The history of WiMAX: a complete survey of the evolution in certification and standarization for IEEE 802.16 and WiMAX

Most researchers are familiar with the technical features of WiMAX technology but the evolution that WiMAX went through, in terms of standardization and certification, is missing and unknown to most people. Knowledge of this historical process would however aid to understand how WiMAX has become the widespread technology that it is today. Furthermore, it would give insight in the steps to undertake for anyone aiming at introducing a new wireless technology on a worldwide scale. Therefore, this article presents a survey on all relevant activities that took place within three important organizations: the 802.16 Working Group of the IEEE (Institute of Electrical and Electronics Engineers) for technology development and standardization, the WiMAX Forum for product certification and the ITU (International Telecommunication Union) for international recognition. An elaborated and comprehensive overview of all those activities is given, which reveals the importance of the willingness to innovate and to continuously incorporate new ideas in the IEEE standardization process and the importance of the WiMAX Forum certification label granting process to ensure interoperability. We also emphasize the steps that were taken in cooperating with the ITU to improve the international esteem of the technology. Finally, a WiMAX trend analysis is made. We showed how industry interest has fluctuated over time and quantified the evolution in WiMAX product certification and deployments. It is shown that most interest went to the 2.5 GHz and 3.5GHz frequencies, that most deployments are in geographic regions with a lot of developing countries and that the highest people coverage is achieved in Asia Pacific. This elaborated description of all standardization and certification activities, from the very start up to now, will make the reader comprehend how past and future steps are taken in the development process of new WiMAX features