Formal Methods in Conformance Testing: A Probabilistic Refinement

This paper refines the framework of ‘Formal Methods in Conformance Testing’ by introducing probabilities for concepts which have a stochastic nature. Test execution is refined into test runs, where each test run is considered as a stochastic process that returns a possible observa- tion with a certain probability. This implies that not every possible observation that could be made, will actually be made. The development process of an implementation from a specifica- tion is also viewed as a stochastic process that may result in a specific implementation with a certain probability. Together with a weight assignment on implementations this introduces a valuation measure on implementations. The test run probabilities and the valuation measures are integrated in generalized definitions of soundness and exhaustiveness, which can be used to compare test suites with respect to their ability to accept correct, and to reject erroneous implementations

Testing theory in practice: a simple experiment

In this paper we discuss the experiences gained in conducting a simple testing experiment. The goal of this experiment is to apply the abstract, formal testing framework [8] in a practical setting, and to indicate the critical aspects in its application to realistic testing situations. For that purpose a formal description of the system under test (SUT) is made from which tests are systematically derived. These tests are interpreted and executed by a tester against concrete implementations, thereby giving an indication about the correctness of these implementations

Factorized Test Generation for Multi Input/Output Transition Systems

Abstract In this paper we present factorized test generation techniques that can be used to generate test cases from a specification that is modelled as a labelled transition system. The test generation techniques are able to construct a sound (and complete) test suite for correctness criterion miocoF [5] by splitting up this correctness criterion into many simpler correctness criteria, and by generating tests for these simpler correctness criteria. By isolating the relevant part of the specification that is needed to generate tests for each of these simpler correctness criteria and using this part to generate tests from, test generation can be done more efficiently. These techniques are intended to keep the generation of tests from a specification feasible and manageable

Specification Based Formal Testing: The Easy Link Case Study

Testing is, in most cases, a manual activity that is time consuming and error prone. Automation, however, can severely reduce the associated costs. In the project Cote de Resyste (COnformance TEsting of REactive SYSTEms) theory is being developed and a protoype tool is being built to support the automatic test generation and execution from formal specifications for reactive systems such as communication protocols and embedded system software [14]. Industrial case studies are carried out by industrial partners to evaluate the theory and tools in practice, to identify potential bottlenecks and to suggest improvements. In this paper we describe our experiences with one of such case studies, the EasyLink case study. In this case study Audio/Video (AV) devices are automatically tested for compliance to the EasyLink standard. The purpose of this case study is to check the viability of the theory and the tools developed in project by applying them in an industrial setting. This paper discusses how the case study led to theoretical developments and tool improvements, and in that way provided valuable feedback that was used to guide the future direction of the project