Towards an I/O Conformance Testing Theory for Software Product Lines based on Modal Interface Automata

We present an adaptation of input/output conformance (ioco) testing principles to families of similar implementation variants as appearing in product line engineering. Our proposed product line testing theory relies on Modal Interface Automata (MIA) as behavioral specification formalism. MIA enrich I/O-labeled transition systems with may/must modalities to distinguish mandatory from optional behavior, thus providing a semantic notion of intrinsic behavioral variability. In particular, MIA constitute a restricted, yet fully expressive subclass of I/O-labeled modal transition systems, guaranteeing desirable refinement and compositionality properties. The resulting modal-ioco relation defined on MIA is preserved under MIA refinement, which serves as variant derivation mechanism in our product line testing theory. As a result, modal-ioco is proven correct in the sense that it coincides with traditional ioco to hold for every derivable implementation variant. Based on this result, a family-based product line conformance testing framework can be established.Comment: In Proceedings FMSPLE 2015, arXiv:1504.0301

Inputs and outputs in CSP : a model and a testing theory

This article addresses refinement and testing based on CSP models, when we distinguish input and output events. In a testing experiment, the tester (or the environment) controls the inputs, and the system under test controls the outputs. The standard models and refinement relations of CSP, however, do not differentiate inputs and outputs and are not, therefore, entirely suitable for testing. Here, we consider an alphabet of events partitioned into inputs and outputs, and we present a novel refusal-testing model for CSP with a notion of input-output refusal-traces refinement. We compare that with the ioco relation often used in testing, and we find that it is more widely applicable and stronger. This means that mistakes found using traditional ioco testing do indicate mistakes in the development. Finally, we provide a CSP testing theory that takes into account inputs and outputs. With our theory, it becomes feasible to develop techniques and tools for automatic generation of realistic and sound tests from CSP models. Our work reconciles the normally disparate areas of refinement and (formal) testing by identifying how ioco testing can be used to inform refinement-based results and vice-versa

Input-output Conformance Testing for Channel-based Service Connectors

Service-based systems are software systems composed of autonomous components or services provided by different vendors, deployed on remote machines and accessible through the web. One of the challenges of modern software engineering is to ensure that such a system behaves as intended by its designer. The Reo coordination language is an extensible notation for formal modeling and execution of service compositions. Services that have no prior knowledge about each other communicate through advanced channel connectors which guarantee that each participant, service or client, receives the right data at the right time. Each channel is a binary relation that imposes synchronization and data constraints on input and output messages. Furthermore, channels are composed together to realize arbitrarily complex behavioral protocols. During this process, a designer may introduce errors into the connector model or the code for their execution, and thus affect the behavior of a composed service. In this paper, we present an approach for model-based testing of coordination protocols designed in Reo. Our approach is based on the input-output conformance (ioco) testing theory and exploits the mapping of automata-based semantic models for Reo to equivalent process algebra specifications

A formal abstract framework for modelling and testing complex software systems

International audienceThe contribution of this paper is twofold: first, it defines a unified framework for modeling abstract components, as well as a formalization of integration rules to combine their behaviour. This is based on a coalgebraic definition of components, which is a categorical representation allowing the unification of a large family of formalisms for specifying state-based systems. Second, it studies compositional conformance testing i.e. checking whether an implementation made of correct interacting components combined with integration operators conforms to its specification

Combining Partial Specifications using Alternating Interface Automata

To model real-world software systems, modelling paradigms should support a form of compositionality. In interface theory and model-based testing with inputs and outputs, conjunctive operators have been introduced: the behaviour allowed by composed specification s1 $\wedge$ s2 is the behaviour allowed by both partial models s1 and s2. The models at hand are non-deterministic interface automata, but the interaction between non-determinism and conjunction is not yet well understood. On the other hand, in the theory of alternating automata, conjunction and non-determinism are core aspects. Alternating automata have not been considered in the context of inputs and outputs, making them less suitable for modelling software interfaces. In this paper, we combine the two modelling paradigms to define alternating interface automata (AIA). We equip these automata with an observational, trace-based semantics, and define testers, to establish correctness of black-box interfaces with respect to an AIA specification

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

Towards a test generation approach for compositional real-time systems.

We can find many examples of Real-time Systems (RTS) in critical applications such as patient monitoring, air traffic control and others. A failure in this kind of system can be catastrophic. For example, it can harm human lives or increase project budgets. Hence, the testing of real-time systems must be accurate. Models are used to perform this task, since they contain information about how the system behaves and when actions may happen. Due to the complexity of the available systems, most RTS are composed of subsystems that interact as part of a bigger system. These subsystems are combined through operators to model their specification behavior. However, works on the testing of compositional models for RTS are practically nonexistent. Among the available approaches to perform testing for non-compositional RTS models, the tioco conformance testing theory focuses on generating test cases based on implementation and specification models. Moreover, a conformance relation defines whether success in testing means conformance between an implementation and a specification. To express specifications and to represent implementations under test, we use Timed Input Output Symbolic Transitions Systems (TIOSTS). These models store symbolic data and clock variables, avoiding the state space and region explosion problems. Regarding the testing of compositional models, some questions may arise: If two subsystem implementations are tioco conformant to their specifications, is it correct to assume that the composition of the implementations is also tioco conformant to the composition of their specifications? In this case, how can operators be defined to work with TIOSTS and tioco? To answer these questions, this thesis proposes the sequential, interruption and parallel operators for the TIOSTS model. For each operator, we study how the tioco conformance relation behaves with respect to subsystems and the composed system. We present results towards properties of compositional operators when the subsystems are composed, as well as implementing them. Besides, we show three examples where each operator can be used and illustrate the applicability of our approach in two exploratory studies. The first models components of a aircraft specification and the second presents application level interruptions in an Android system.We can nd many examples of Real-time Systems (RTS) in critical applications such as patient monitoring, air tra c control and others. A failure in this kind of system can be catastrophic. For example, it can harm human lives or increase project budgets. Hence, the testing of real-time systems must be accurate. Models are used to perform this task, since they contain information about how the system behaves and when actions may happen. Due to the complexity of the available systems, most RTS are composed of subsystems that interact as part of a bigger system. These subsystems are combined through operators to model their speci cation behavior. However, works on the testing of compositional models for RTS are practically nonexistent. Among the available approaches to perform testing for non-compositional RTS models, the tioco conformance testing theory focuses on generating test cases based on implementation and speci cation models. Moreover, a conformance relation de nes whether success in testing means conformance between an implementation and a speci cation. To express speci cations and to represent implementations under test, we use Timed Input Output Symbolic Transitions Systems (TIOSTS). These models store symbolic data and clock variables, avoiding the state space and region explosion problems. Regarding the testing of compositional models, some questions may arise: If two subsystem implementations are tioco conformant to their speci cations, is it correct to assume that the composition of the implementations is also tioco conformant to the composition of their speci cations? In this case, how can operators be de ned to work with TIOSTS and tioco? To answer these questions, this thesis proposes the sequential, interruption and parallel operators for the TIOSTS model. For each operator, we study how the tioco conformance relation behaves with respect to subsystems and the composed system. We present results towards properties of compositional operators when the subsystems are composed, as well as implementing them. Besides, we show three examples where each operator can be used and illustrate the applicability of our approach in two exploratory studies. The rst models components of a aircraft speci cation and the second presents application level interruptions in an Android system

The Best of Both Worlds: Model-Driven Engineering Meets Model-Based Testing

We study the connection between stable-failures refinement and the ioco conformance relation. Both behavioural relations underlie methodologies that have gained traction in industry: stable-failures refinement is used in several commercial Model-Driven Engineering tool suites, whereas the ioco conformance relation is used in Model-Based Testing tools. Refinement-based Model-Driven Engineering approaches promise to generate executable code from high-level models, thus guaranteeing that the code upholds specified behavioural contracts. Manual testing, however, is still required to gain confidence that the model-to-code transformation and the execution platform do not lead to unexpected contract violations. We identify conditions under which also this last step in the design methodology can be automated using the ioco conformance relation and the associated tools