Testing timed systems modeled by stream X-machines

Stream X-machines have been used to specify real systems where complex data structures. They are a variety of extended finite state machine where a shared memory is used to represent communications between the components of systems. In this paper we introduce an extension of the Stream X-machines formalism in order to specify systems that present temporal requirements. We add time in two different ways. First, we consider that (output) actions take time to be performed. Second, our formalism allows to specify timeouts. Timeouts represent the time a system can wait for the environment to react without changing its internal state. Since timeous affect the set of available actions of the system, a relation focusing on the functional behavior of systems, that is, the actions that they can perform, must explicitly take into account the possible timeouts. In this paper we also propose a formal testing methodology allowing to systematically test a system with respect to a specification. Finally, we introduce a test derivation algorithm. Given a specification, the derived test suite is sound and complete, that is, a system under test successfully passes the test suite if and only if this system conforms to the specification

The Best of Both Worlds: Model-Driven Engineering Meets Model-Based Testing

We study the connection between stable-failures refinement and the ioco conformance relation. Both behavioural relations underlie methodologies that have gained traction in industry: stable-failures refinement is used in several commercial Model-Driven Engineering tool suites, whereas the ioco conformance relation is used in Model-Based Testing tools. Refinement-based Model-Driven Engineering approaches promise to generate executable code from high-level models, thus guaranteeing that the code upholds specified behavioural contracts. Manual testing, however, is still required to gain confidence that the model-to-code transformation and the execution platform do not lead to unexpected contract violations. We identify conditions under which also this last step in the design methodology can be automated using the ioco conformance relation and the associated tools

Constraint-based oracles for timed distributed systems

© 2017, IFIP International Federation for Information Processing. This paper studies the situation in which the system under test and the system model are distributed and have the same structure; they have corresponding remote components that communicate asynchronously. In testing, a component with interface C i has its own local tester that interacts with C i and this local tester observes a local trace consisting of inputs, outputs and durations as perceived by C i . An observation made in testing is thus a multi-trace: a tuple of (timed) local traces, one for each C i . The conformance relation for such distributed systems combines a classical unitary conformance relation for localised components and the requirement that the communication policy was satisfied. By expressing the communication policy as a constraint satisfaction problem, we were able to implement the computation of test verdicts by orchestrating localised off-line testing algorithms and the verification of constraints defined by message passing between components. Lastly, we illustrate our approach on a telecommunications system

Passive Testing of Stochastic Timed Systems

In this paper we introduce a formal Methodology to perforin passive testing, based on invariants, for systems where the passing of time is represented in probabilistic terms by means of probability distributions functions. In our approach, invariants express the fact that each time the implementation under test performs a given sequence of actions, then it must exhibit a behavior according to the probability distribution functions reflected it? the invariant. We present algorithms to decide the correctness of the proposed invariants with respect to a given specification. Once we know that an invariant is correct, we check whether the execution traces observed from the implementation respect the invariant. In addition to the theoretical framework we have developed a tool., called PASTE, that helps in the automation of our passive testing approach. We have used the tool to obtain experimental results front the application of our methodology

An implementation relation for cyclic systems with refusals and discrete time

This paper explores a particular type of model, a cyclic model, in which there are sequences of observable actions separated by discrete time intervals, introduces a novel implementation relation and studies some properties of this relation. Implementation relations formalise what it means for an unknown model of the system under test (SUT) to be a correct implementation of a specification. Many implementation relations are variants of the well known ioco implementation relation, and this includes several timed versions of ioco. It transpires that the timed variants of ioco are not suitable for cyclic models. Our implementation relation encapsulates the discrete nature of time in cyclic models and takes into account not only the actions that models can perform but also the ones that they can refuse at each point of time. We prove that our implementation relation is a conservative extension of trace containment and present two alternative characterisations

Compositional schedulability analysis of real-time actor-based systems

We present an extension of the actor model with real-time, including deadlines associated with messages, and explicit application-level scheduling policies, e.g.,"earliest deadline first" which can be associated with individual actors. Schedulability analysis in this setting amounts to checking whether, given a scheduling policy for each actor, every task is processed within its designated deadline. To check schedulability, we introduce a compositional automata-theoretic approach, based on maximal use of model checking combined with testing. Behavioral interfaces define what an actor expects from the environment, and the deadlines for messages given these assumptions. We use model checking to verify that actors match their behavioral interfaces. We extend timed automata refinement with the notion of deadlines and use it to define compatibility of actor environments with the behavioral interfaces. Model checking of compatibility is computationally hard, so we propose a special testing process. We show that the analyses are decidable and automate the process using the Uppaal model checke

Implementation relations and testing for cyclic systems: adding probabilities

This paper concerns the systematic testing of robotic control software based on state-based models. We focus on cyclic systems that typically receive inputs (values from sensors), perform computations, produce outputs (sent to actuators) and possibly change state. We provide a testing theory for such cyclic systems where time can be represented and probabilities are used to quantify non-deterministic choices, making it possible to model probabilistic algorithms. In addition, refusals, the inability of a system to perform a set of actions, are taken into account. We consider several possible testing scenarios. For example, a tester might only be able to passively observe a sequence of events and so cannot check probabilities, while in another scenario a tester might be able to repeatedly apply a test case and so estimate the probabilities of sequences of events. These different testing scenarios lead to a range of implementation relations (notions of correctness). As a consequence, this paper provides formal definitions of implementation relations that can form the basis of sound automated testing in a range of testing scenarios. We also validate the implementation relations by showing how observers can be used to provide an alternative but equivalent characterisation

Automated model-based testing of hybrid systems

In automated model-based input-output conformance testing, tests are automati- cally generated from a speci¯cation and automatically executed on an implemen- tation. Input is applied to the implementation and output is observed from the implementation. If the observed output is allowed according to the test, then test- ing may continue, or stop with the verdict pass. If the observed output is not allowed according to the test, then testing stops with the verdict fail. The advantages of this test method are that: ² specifications can be reused to test every product in exactly the same way, ² test environments can be controlled because the behavior of the environment is specified as the input of the implementation, ² tests can be generated that a test engineer did not think of yet, ² a huge quantity of tests can be generated and repeated endlessly, and ² the test engineer can focus on testing the parts of the system for which tests are not automated. A hybrid system is a system with both discrete-events and continuous behavior. By continuous behavior we usually mean the behavior of physical quantities over time. A thermostat that observes a chamber temperature and turns on a heater based on the observed temperature change is a system with continuous input and discrete-event output. A robot arm that moves with a certain speed on command (e.g. "GO LEFT") is a system with discrete-event input and continuous output. Within the Tangram project, a four year research project on model-based test and integration methods and their applications, one of the goals was to develop model- based testing for hybrid systems. This involves incorporating continuous behavior and discrete-event behavior into one input-output conformance relation and into a notion of hybrid test. Then, this approach to hybrid model-based testing had to be tried out in practice, in an industrial environment. In this thesis we describe the result of this research. In Chapter 2 and Chapter 3 we define the necessary preliminaries for defining our conformance relation and notion of test for hybrid systems. We use hybrid tran- sition systems to formally represent the implementation and the specification of a system. We base our conformance relation on the discrete-event input-output con- formance relation by Tretmans, and the timed input-output conformance relations by Brandan-Briones and Brinksma, and by Krichen and Tripakis. In Chapter 4 we define our input-output conformance relation for hybrid systems. In this chapter we also define a notion of test for hybrid systems that we have proven sound and exhaustive with respect to the hybrid conformance relation. Based on the notion of hybrid test, we have implemented a proof-of-concept hybrid model-based test tool. The architecture of our tool is based on the TorX test tool and the tests are generated from a hybrid specification using the hybrid  simulation tool. In Chapter 5 we describe TorX and the hybrid X language. In Chapter 6 we describe the issues involved in developing a hybrid model-based test tool in general, and our proof-of-concept tool in particular. In order to better fit theory and practice, we adapt our hybrid input-output conformance relation and notion of test to a conformance relation and notion of test for sampled behavior. We have proven that, under certain conditions, if a hybrid implementation conforms to a hybrid specification, then the implementation also conforms to the specification with sampled behavior. In Chapter 7 we describe the results of a case study that we have performed on a vacuum controller of a waferstepper machine. This controller has sampled con- tinuous input (namely samples of pressure observations) and discrete-event output (namely controlling pumps and valves). We have made a specification that models the sequences of events required for pumping down a vacuum chamber or venting a vacuum chamber. We have modeled the pressure loow in the chamber as continu- ous behavior. With the proof-of-concept tool we have been able to generate tests, stimulate the vacuum control software with sampled pressure low, observe output of the vacuum control software, and give a verdict. We have found a fault in the control software that was not found previously in the field, nor by co-simulation of the controller and a model of the hardware, nor by model checking using Uppaal. This result shows that hybrid model-based testing has added value. In chapter 8 we describe the results of this research and we present some directions for future research