An Optimization of the TorX Test Generation Algorithm

In this paper we will discuss the process of automatic test derivation from formal specification. The process will be illustrated in the TORX algorithm. We will present an optimization of TORX. The extension of the algorithm with explicit probabilities leads to improvements in the tests generated with respect to the chances of finding errors in the implementation

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

Model-based testing of probabilistic systems

This work presents an executable model-based testing framework for probabilistic systems with non-determinism. We provide algorithms to automatically generate, execute and evaluate test cases from a probabilistic requirements specification. The framework connects input/output conformance-theory with hypothesis testing: our algorithms handle functional correctness, while statistical methods assess, if the frequencies observed during the test process correspond to the probabilities specified in the requirements. At the core of our work lies the conformance relation for probabilistic input/output conformance, enabling us to pin down exactly when an implementation should pass a test case. We establish the correctness of our framework alongside this relation as soundness and completeness; Soundness states that a correct implementation indeed passes a test suite, while completeness states that the framework is powerful enough to discover each deviation from a specification up to arbitrary precision for a sufficiently large sample size. The underlying models are probabilistic automata that allow invisible internal progress. We incorporate divergent systems into our framework by phrasing four rules that each well-formed system needs to adhere to. This enables us to treat divergence as the absence of output, or quiescence, which is a well-studied formalism in model-based testing. Lastly, we illustrate the application of our framework on three case studies

Automated model-based testing of hybrid systems

In automated model-based input-output conformance testing, tests are automati- cally generated from a speci¯cation and automatically executed on an implemen- tation. Input is applied to the implementation and output is observed from the implementation. If the observed output is allowed according to the test, then test- ing may continue, or stop with the verdict pass. If the observed output is not allowed according to the test, then testing stops with the verdict fail. The advantages of this test method are that: ² specifications can be reused to test every product in exactly the same way, ² test environments can be controlled because the behavior of the environment is specified as the input of the implementation, ² tests can be generated that a test engineer did not think of yet, ² a huge quantity of tests can be generated and repeated endlessly, and ² the test engineer can focus on testing the parts of the system for which tests are not automated. A hybrid system is a system with both discrete-events and continuous behavior. By continuous behavior we usually mean the behavior of physical quantities over time. A thermostat that observes a chamber temperature and turns on a heater based on the observed temperature change is a system with continuous input and discrete-event output. A robot arm that moves with a certain speed on command (e.g. "GO LEFT") is a system with discrete-event input and continuous output. Within the Tangram project, a four year research project on model-based test and integration methods and their applications, one of the goals was to develop model- based testing for hybrid systems. This involves incorporating continuous behavior and discrete-event behavior into one input-output conformance relation and into a notion of hybrid test. Then, this approach to hybrid model-based testing had to be tried out in practice, in an industrial environment. In this thesis we describe the result of this research. In Chapter 2 and Chapter 3 we define the necessary preliminaries for defining our conformance relation and notion of test for hybrid systems. We use hybrid tran- sition systems to formally represent the implementation and the specification of a system. We base our conformance relation on the discrete-event input-output con- formance relation by Tretmans, and the timed input-output conformance relations by Brandan-Briones and Brinksma, and by Krichen and Tripakis. In Chapter 4 we define our input-output conformance relation for hybrid systems. In this chapter we also define a notion of test for hybrid systems that we have proven sound and exhaustive with respect to the hybrid conformance relation. Based on the notion of hybrid test, we have implemented a proof-of-concept hybrid model-based test tool. The architecture of our tool is based on the TorX test tool and the tests are generated from a hybrid specification using the hybrid  simulation tool. In Chapter 5 we describe TorX and the hybrid X language. In Chapter 6 we describe the issues involved in developing a hybrid model-based test tool in general, and our proof-of-concept tool in particular. In order to better fit theory and practice, we adapt our hybrid input-output conformance relation and notion of test to a conformance relation and notion of test for sampled behavior. We have proven that, under certain conditions, if a hybrid implementation conforms to a hybrid specification, then the implementation also conforms to the specification with sampled behavior. In Chapter 7 we describe the results of a case study that we have performed on a vacuum controller of a waferstepper machine. This controller has sampled con- tinuous input (namely samples of pressure observations) and discrete-event output (namely controlling pumps and valves). We have made a specification that models the sequences of events required for pumping down a vacuum chamber or venting a vacuum chamber. We have modeled the pressure loow in the chamber as continu- ous behavior. With the proof-of-concept tool we have been able to generate tests, stimulate the vacuum control software with sampled pressure low, observe output of the vacuum control software, and give a verdict. We have found a fault in the control software that was not found previously in the field, nor by co-simulation of the controller and a model of the hardware, nor by model checking using Uppaal. This result shows that hybrid model-based testing has added value. In chapter 8 we describe the results of this research and we present some directions for future research

Learning and testing the bounded retransmission protocol

Abstract Using a well-known industrial case study from the verification literature, the bounded retransmission protocol, we show how active learning can be used to establish the correctness of protocol implementation I relative to a given reference implementation R. Using active learning, we learn a model M R of reference implementation R, which serves as input for a model based testing tool that checks conformance of implementation I to M R . In addition, we also explore an alternative approach in which we learn a model M I of implementation I, which is compared to model M R using an equivalence checker. Our work uses a unique combination of software tools for model construction (Uppaal), active learning (LearnLib, Tomte), model-based testing (JTorX, TorXakis) and verification (CADP, MRMC). We show how these tools can be used for learning these models, analyzing the obtained results, and improving the learning performance