Testing refinements by refining tests

One of the potential benefits of formal methods is that they offer the possibility of reducing the costs of testing. A specification acts as both the benchmark against which any implementation is tested, and also as the means by which tests are generated. There has therefore been interest in developing test generation techniques from formal specifications, and a number of different methods have been derived for state based languages such as Z, B and VDM. However, in addition to deriving tests from a formal specification, we might wish to refine the specification further before its implementation. The purpose of this paper is to explore the relationship between testing and refinement. As our model for test generation we use a DNF partition analysis for operations written in Z, which produces a number of disjoint test cases for each operation. In this paper we discuss how the partition analysis of an operation alters upon refinement, and we develop techniques that allow us to refine abstract tests in order to generate test cases for a refinement. To do so we use (and extend existing) methods for calculating the weakest data refinement of a specification

Conformance relations for distributed testing based on CSP

Copyright @ 2011 Springer Berlin HeidelbergCSP is a well established process algebra that provides comprehensive theoretical and practical support for refinement-based design and verification of systems. Recently, a testing theory for CSP has also been presented. In this paper, we explore the problem of testing from a CSP specification when observations are made by a set of distributed testers. We build on previous work on input-output transition systems, but the use of CSP leads to significant differences, since some of its conformance (refinement) relations consider failures as well as traces. In addition, we allow events to be observed by more than one tester. We show how the CSP notions of refinement can be adapted to distributed testing. We consider two contexts: when the testers are entirely independent and when they can cooperate. Finally, we give some preliminary results on test-case generation and the use of coordination messages. © 2011 IFIP International Federation for Information Processing

Towards an I/O Conformance Testing Theory for Software Product Lines based on Modal Interface Automata

We present an adaptation of input/output conformance (ioco) testing principles to families of similar implementation variants as appearing in product line engineering. Our proposed product line testing theory relies on Modal Interface Automata (MIA) as behavioral specification formalism. MIA enrich I/O-labeled transition systems with may/must modalities to distinguish mandatory from optional behavior, thus providing a semantic notion of intrinsic behavioral variability. In particular, MIA constitute a restricted, yet fully expressive subclass of I/O-labeled modal transition systems, guaranteeing desirable refinement and compositionality properties. The resulting modal-ioco relation defined on MIA is preserved under MIA refinement, which serves as variant derivation mechanism in our product line testing theory. As a result, modal-ioco is proven correct in the sense that it coincides with traditional ioco to hold for every derivable implementation variant. Based on this result, a family-based product line conformance testing framework can be established.Comment: In Proceedings FMSPLE 2015, arXiv:1504.0301

Rigorous development process of a safety-critical system: from ASM models to Java code

The paper presents an approach for rigorous development of safety-critical systems based on the Abstract State Machine formal method. The development process starts from a high level formal view of the system and, through refinement, derives more detailed models till the desired level of specification. Along the process, different validation and verification activities are available, as simulation, model review, and model checking. Moreover, each refinement step can be proved correct using an SMT-based approach. As last step of the refinement process, a Java implementation can be developed and linked to the formal specification. The correctness of the implementation w.r.t. its formal specification can be proved by means of model-based testing and runtime verification. The process is exemplified by using a Landing Gear System as case study

Action refinement in testing with UIOCO

In model based testing test cases are derived from a specification of the implementation that we want to test. In general the specification is given on a more abstract level than the implementation. This may result in test cases that are not executable, because their actions are too abstract; the implementation does not understand them. One approach is to rewrite the specification to the required level of detail and regenerate the test cases. Rewriting a specification by hand is an error-prone and time consuming exercise that is not always favorable. Very often there is a good reason for the level of abstraction in a specification, for example to illustrate the structure of the system or to separate concerns. In this paper we present an approach to automatically obtain test cases of the required level of detail by means of action refinement. Action refinement is a way to add information to the abstract specification. The extra information relates actions from the abstract specification to concrete actions of the implementation. We will apply this approach to a simple case of action refinement, so called atomic linear input-inputs refinement. For this type of action refinement our approach enables us to automatically refine traces and transition systems. Furthermore, we present an implementation relation that relates an abstract specification with its concrete implementation and show that it is equivalent with the UIOCO implementation relation on the refined specification

Formal Modelling, Testing and Verification of HSA Memory Models using Event-B

The HSA Foundation has produced the HSA Platform System Architecture Specification that goes a long way towards addressing the need for a clear and consistent method for specifying weakly consistent memory. HSA is specified in a natural language which makes it open to multiple ambiguous interpretations and could render bugs in implementations of it in hardware and software. In this paper we present a formal model of HSA which can be used in the development and verification of both concurrent software applications as well as in the development and verification of the HSA-compliant platform itself. We use the Event-B language to build a provably correct hierarchy of models from the most abstract to a detailed refinement of HSA close to implementation level. Our memory models are general in that they represent an arbitrary number of masters, programs and instruction interleavings. We reason about such general models using refinements. Using Rodin tool we are able to model and verify an entire hierarchy of models using proofs to establish that each refinement is correct. We define an automated validation method that allows us to test baseline compliance of the model against a suite of published HSA litmus tests. Once we complete model validation we develop a coverage driven method to extract a richer set of tests from the Event-B model and a user specified coverage model. These tests are used for extensive regression testing of hardware and software systems. Our method of refinement based formal modelling, baseline compliance testing of the model and coverage driven test extraction using the single language of Event-B is a new way to address a key challenge facing the design and verification of multi-core systems.Comment: 9 pages, 10 figure

Behavior models for software architecture

Monterey Phoenix (MP) is an approach to formal software system architecture specification based on behavior models. Architecture modeling focuses not only on the activities and interactions within the system, but also on the interactions between the system and its environment, providing an abstraction for interaction specification. The behavior of the system is defined as a set of events (event trace) with two basic relations: precedence and inclusion. The structure of possible event traces is specified using event grammars and other constraints organized into schemas. The separation of the interaction description from the components behavior is an essential MP feature. The schema framework is amenable to stepwise architecture refinement, reuse, composition, visualization, and multiple view extraction. The approach yields a basis for executable architecture specification supporting early testing and verification, systematic use case generation, and performance estimates with automated tools.Consortium for Robotics and Unmanned Systems Education and Research (CRUSER)Consortium for Robotics and Unmanned Systems Education and Research (CRUSER)Approved for public release; distribution is unlimited.Approved for public release; distribution is unlimited

Comparing test sets and criteria in the presence of test hypotheses and fault domains

A number of authors have considered the problem of comparing test sets and criteria. Ideally test sets are compared using a preorder with the property that test set T1 is at least as strong as T2 if whenever T2 determines that an implementation p is faulty, T1 will also determine that p is faulty. This notion can be extended to test criteria. However, it has been noted that very few test sets and criteria are comparable under such an ordering; instead orderings are based on weaker properties such as subsumes. This paper explores an alternative approach, in which comparisons are made in the presence of a test hypothesis or fault domain. This approach allows strong statements about fault detecting ability to be made and yet for a number of test sets and criteria to be comparable. It may also drive incremental test generation