Model-based testing with chi and TorX : a case study of the ASML laser subsystem

Within the TANGRAM project, a case study on model-based testing of the ASML laser subsystem has been performed. The approach used in the case study is based on the proposed model-based testing framework, instantiated with state-of-the-art tooling from the TANGRAM project partners: chi as specification language and TorX as test tool. A chi specification model of the laser state behavior and communication interface has been developed. After verification and validation, the model has been used for automatic model-based testing with TorX. Using this approach, discrepancies between the implementation and specification of the laser subsystem have been found

The implementation of specification-based testing system for real-time reactive system in TROMLAB framework

This major report describes the analysis, design, and implementation of the "TROM-SBTS: specification-based testing system for real-time reactive system in TROMLIB framework" in Java. Specification-based testing is a new approach in black box testing for real-time reactive system developed by Mao Zheng [Mao02]. The TROM-SBTS has specification-based unit testing, pair testing, and system testing functionalities. After studying the specification-based testing algorithms reported in Mao's thesis [Mao02], the updating of the algorithms is reported. Object oriented design technology is used in the system architecture and detailed design of the TROM-SBTS. Some existing softwares are integrated into the system. Finally, the train-gate-controller problem is taken as the case study for unit testing, pair testing, and system testing. The empirical result proves the correctness of algorithms and their implementation

Scenarios-based testing of systems with distributed ports

Copyright @ 2011 John Wiley & SonsDistributed systems are usually composed of several distributed components that communicate with their environment through specific ports. When testing such a system we separately observe sequences of inputs and outputs at each port rather than a global sequence and potentially cannot reconstruct the global sequence that occurred. Typically, the users of such a system cannot synchronise their actions during use or testing. However, the use of the system might correspond to a sequence of scenarios, where each scenario involves a sequence of interactions with the system that, for example, achieves a particular objective. When this is the case there is the potential for there to be a significant delay between two scenarios and this effectively allows the users of the system to synchronise between scenarios. If we represent the specification of the global system by using a state-based notation, we say that a scenario is any sequence of events that happens between two of these operations. We can encode scenarios in two different ways. The first approach consists of marking some of the states of the specification to denote these synchronisation points. It transpires that there are two ways to interpret such models and these lead to two implementation relations. The second approach consists of adding a set of traces to the specification to represent the traces that correspond to scenarios. We show that these two approaches have similar expressive power by providing an encoding from marked states to sets of traces. In order to assess the appropriateness of our new framework, we show that it represents a conservative extension of previous implementation relations defined in the context of the distributed test architecture: if we onsider that all the states are marked then we simply obtain ioco (the classical relation for single-port systems) while if no state is marked then we obtain dioco (our previous relation for multi-port systems). Finally, we concentrate on the study of controllable test cases, that is, test cases such that each local tester knows exactly when to apply inputs. We give two notions of controllable test cases, define an implementation relation for each of these notions, and relate them. We also show how we can decide whether a test case satisfies these conditions.Research partially supported by the Spanish MEC project TESIS (TIN2009-14312-C02-01), the UK EPSRC project Testing of Probabilistic and Stochastic Systems (EP/G032572/1), and the UCM-BSCH programme to fund research groups (GR58/08 - group number 910606)

Implementing E-learning Specifications with Conformance Testing: Profiling for IMS Learning Design

Submitted for publication. Please contact authors for reference.Improving interoperability between e-learning systems and content has been one of the driving forces behind the adoption of e-learning specifications over recent years. A vital step towards achieving this goal is the widespread adoption of conformant implementations of e-learning specifications. A conformant implementation is one which fully complies with the conformance requirements of the specification. However, conformance testing is time consuming and expensive. The process of localising specifications to create so-called “Application Profiles” to meet individual community needs further complicates conformance testing efforts. To solve this problem, we developed the conformance testing approach presented in this article. This approach simplifies the development of Application Profiles, and the process of conformance testing against them. Using this approach, test suites can be generated to test software applications against both e-learning specifications and their derived Application Profiles. A case study based around the IMS Learning Design specification demonstrates this process

Integrating Functional and Security Requirements Analysis using SOFL for Software Security Assurance

Formal methods have been applied to define requirements for safety and/or security critical software systems in some industrial sectors, but the challenge is the lack of a systematic way to take security issues into account in specifying the functional behaviors. In this paper, we propose a formal approach to expressing and explicitly interweaving security and functional requirements. With this approach, the functional behaviors of the system are precisely specified using the Structured Object Oriented Formal Language (SOFL), the security rules are systematically explored, and the result is properly incorporated into the functional specification as constraints. The resultant specification then defines the system functionality that implies the conformance to the security rules. Such a specification can be used as a firm foundation for implementation and testing of the implementation. We discuss the principle of interweaving security rules with functional specifications and present a case study to demonstrate the feasibility of our approac

Automated model-based testing of hybrid systems

In automated model-based input-output conformance testing, tests are automati- cally generated from a speci¯cation and automatically executed on an implemen- tation. Input is applied to the implementation and output is observed from the implementation. If the observed output is allowed according to the test, then test- ing may continue, or stop with the verdict pass. If the observed output is not allowed according to the test, then testing stops with the verdict fail. The advantages of this test method are that: ² specifications can be reused to test every product in exactly the same way, ² test environments can be controlled because the behavior of the environment is specified as the input of the implementation, ² tests can be generated that a test engineer did not think of yet, ² a huge quantity of tests can be generated and repeated endlessly, and ² the test engineer can focus on testing the parts of the system for which tests are not automated. A hybrid system is a system with both discrete-events and continuous behavior. By continuous behavior we usually mean the behavior of physical quantities over time. A thermostat that observes a chamber temperature and turns on a heater based on the observed temperature change is a system with continuous input and discrete-event output. A robot arm that moves with a certain speed on command (e.g. "GO LEFT") is a system with discrete-event input and continuous output. Within the Tangram project, a four year research project on model-based test and integration methods and their applications, one of the goals was to develop model- based testing for hybrid systems. This involves incorporating continuous behavior and discrete-event behavior into one input-output conformance relation and into a notion of hybrid test. Then, this approach to hybrid model-based testing had to be tried out in practice, in an industrial environment. In this thesis we describe the result of this research. In Chapter 2 and Chapter 3 we define the necessary preliminaries for defining our conformance relation and notion of test for hybrid systems. We use hybrid tran- sition systems to formally represent the implementation and the specification of a system. We base our conformance relation on the discrete-event input-output con- formance relation by Tretmans, and the timed input-output conformance relations by Brandan-Briones and Brinksma, and by Krichen and Tripakis. In Chapter 4 we define our input-output conformance relation for hybrid systems. In this chapter we also define a notion of test for hybrid systems that we have proven sound and exhaustive with respect to the hybrid conformance relation. Based on the notion of hybrid test, we have implemented a proof-of-concept hybrid model-based test tool. The architecture of our tool is based on the TorX test tool and the tests are generated from a hybrid specification using the hybrid  simulation tool. In Chapter 5 we describe TorX and the hybrid X language. In Chapter 6 we describe the issues involved in developing a hybrid model-based test tool in general, and our proof-of-concept tool in particular. In order to better fit theory and practice, we adapt our hybrid input-output conformance relation and notion of test to a conformance relation and notion of test for sampled behavior. We have proven that, under certain conditions, if a hybrid implementation conforms to a hybrid specification, then the implementation also conforms to the specification with sampled behavior. In Chapter 7 we describe the results of a case study that we have performed on a vacuum controller of a waferstepper machine. This controller has sampled con- tinuous input (namely samples of pressure observations) and discrete-event output (namely controlling pumps and valves). We have made a specification that models the sequences of events required for pumping down a vacuum chamber or venting a vacuum chamber. We have modeled the pressure loow in the chamber as continu- ous behavior. With the proof-of-concept tool we have been able to generate tests, stimulate the vacuum control software with sampled pressure low, observe output of the vacuum control software, and give a verdict. We have found a fault in the control software that was not found previously in the field, nor by co-simulation of the controller and a model of the hardware, nor by model checking using Uppaal. This result shows that hybrid model-based testing has added value. In chapter 8 we describe the results of this research and we present some directions for future research

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing