A Model-based Approach for Test Cases Generation

International audienceThe analytical methods based on evaluation models of interactive systems were proposed as an alternative to user testing in the last stages of the software development due to its costs. However, the use of isolated behavioral models of the system limits the results of the analytical methods. An example of these limitations relates to the fact that they are unable to identify implementation issues that will impact on usability. With the introduction of model-based testing we are enable to test if the implemented software meets the specified model. This paper presents an model-based approach for test cases generation from the static analysis of source code

A MOP based DSL for testing Java programs using OCL.

OCL is used to specify systems by defining pre and postconditions for class operations. Typically, the conditions refer to properties and operations that are defined in a model. When the model is implemented, various implementation decisions are made regarding properties and operations that cause the OCL conditions to be inconsistent with the implementation. This paper shows how OCL conditions can be performed against a Java implementation of a model and how a meta-object protocol can be used to control the relationship between the original model and the implementation thereby retaining OCL consistenc

Finding the needles in the haystack: Generating legal test inputs for object-oriented programs

A test input for an object-oriented program typically consists of asequence of method calls that use the API defined by the programunder test. Generating legal test inputs can be challenging because,for some programs, the set of legal method sequences is much smallerthan the set of all possible sequences; without a formalspecification of legal sequences, an input generator is bound toproduce mostly illegal sequences.We propose a scalable technique that combines dynamic analysis withrandom testing to help an input generator create legal test inputswithout a formal specification, even for programs in whichmost sequences are illegal. The technique uses an example executionof the program to infer a model of legal call sequences, and usesthe model to guide a random input generator towards legal butbehaviorally-diverse sequences.We have implemented our technique for Java, in a tool calledPalulu, and evaluated its effectiveness in creating legal inputsfor real programs. Our experimental results indicate that thetechnique is effective and scalable. Our preliminary evaluationindicates that the technique can quickly generate legal sequencesfor complex inputs: in a case study, Palulu created legal testinputs in seconds for a set of complex classes, for which it took anexpert thirty minutes to generate a single legal input

Model-based user interface testing with Spec Explorer and ConcurTaskTrees

Analytic usability analysis methods have been proposed as an alternative to user testing in early phases of development due to the cost of the latter approach. By working with models of the systems, analytic models are not capable of identifying implementation related problems that might have an impact on usability. Model-based testing enables the testing of an implemented software artefact against a model of what it should be (the oracle). In the case of model-based user interface testing, the models should be expressed at an adequate level of abstraction, adequately modelling the interaction process. This paper describes an effort to develop tool support enabling the use of task models as oracles for model-based testing of user interfaces.FCT -Fuel Cell Technologies Program(POSC/EIA/56646/2004

Model-based Testing

This paper provides a comprehensive introduction to a framework for formal testing using labelled transition systems, based on an extension and reformulation of the ioco theory introduced by Tretmans. We introduce the underlying models needed to specify the requirements, and formalise the notion of test cases. We discuss conformance, and in particular the conformance relation ioco. For this relation we prove several interesting properties, and we provide algorithms to derive test cases (either in batches, or on the fly)

Generación Automática de Casos de Prueba a partir de Casos de Uso : Una Propuesta Basada en MDD/MDT

Model Driven Testing (MDT) es una nueva propuesta que se basa en la definición de tests basados en modelos y en la definición de procesos sistemáticos para la obtención de casos de prueba. Este trabajo presenta un proceso para la generación automática de casos de prueba a partir del modelo de casos de uso de UML, contando con la ventaja de definir tempranamente casos de prueba en la etapa de análisis incentivando así el comienzo del proceso de testing en el momento en que un requerimiento funcional se encuentre identificado y documentado; evitando de esta manera posponer la actividad de testeo a etapas de construcción, donde corregir errores implica importantes pérdidas de tiempo y esfuerzo.Sociedad Argentina de Informática e Investigación Operativ

Geração de Testes de Desempenho e Estresse a partir de Testes Funcionais

Apesar do Teste de Software estar relacionado à garantia de qualidade, ele demanda tempo e recursos, e por isso acaba por não ser realizada em parte dos projetos executados. No contexto das empresas que sempre o realizam, percebe-se que o teste funcional cresce em utilização e que o teste de desempenho e estresse ainda é pouco executado, mesmo diante do aumento dos sistemas acessíveis via Web, em que esses testes são considerados essenciais. Neste artigo é proposto um método e sua ferramenta de apoio para a geração de testes de desempenho e estresse de sistemas Web a partir de testes funcionais, na tentativa de popularizar o teste para esses requisitos não-funcionais. Apresenta-se também um estudo experimental que concluiu que o uso do método e da ferramenta pode gerar ganhos em termo de redução de esforço para a geração desses testes

Automated Transition Coverage in Behavioural Conformance Testing

International audienceIn the setting of ioco-based conformance testing with test purposes, we propose an automatic approach to generate a test plan (set of test purposes) with its associated test suite (set of test cases) covering all transitions of the IOLTS model of the system. The approach can also be applied to improve an existing test plan, by both, completing the coverage and eliminating redundancies. Implementing our approach on top of the CADP toolbox, we report on experiments with several examples of concurrent systems and discuss possible variants and heuristics to fine-tune the overall performance of the approach, as well as the quality of the computed test plan

Experiences with formal engineering: model-based specification, implementation and testing of a software bus at Neopost

We report on the actual industrial use of formal methods during the development of a software bus. During an internship at Neopost Inc., of 14 weeks, we developed the server component of a software bus, called the XBus, using formal methods during the design, validation and testing phase: we modeled our design of the XBus in the process algebra mCRL2, validated the design using the mCRL2-simulator, and fully automatically tested our implementation with the model-based test tool JTorX. This resulted in a well- tested software bus with a maintainable architecture. Writing the model (mdev), simulating it, and testing the implementation with JTorX only took 17% of the total development time. Moreover, the errors found with model-based testing would have been hard to find with conventional test methods. Thus, we show that formal engineering can be feasible, beneficial and cost-effective.\ud The findings above, reported earlier by us in (Sijtema et al., 2011) [1], were well- received, also in industrially oriented conferences (Ferreira and Romanenko, 2010) [2] and [3]. In this paper, we look back on the case study, and carefully analyze its merits and shortcomings. We reflect on (1) the added benefits of model checking, (2) model completeness and (3) the quality and performance of the test process.\ud Thus, in a second phase, after the internship, we model checked the XBus protocol—this was not done in [1] since the Neopost business process required a working implementation after 14 weeks. We used the CADP tool evaluator4 to check the behavioral requirements obtained during the development. Model checking did not uncover errors in model mdev, but revealed that model mdev was neither complete nor optimized: in particular, requirements to the so-called bad weather behavior (exceptions, unexpected inputs, etc.) were missing. Therefore, we created several improved models, checked that we could validate them, and used them to analyze quality and performance of the test process. Model checking was expensive: it took us approx. 4 weeks in total, compared to 3 weeks for the entire model-based testing approach during the internship.\ud In the second phase, we analyzed the quality and performance of the test process, where we looked at both code and model coverage. We found that high code coverage (almost 100%) is in most cases obtained within 1000 test steps and 2 minutes, which matches the fact that the faults in the XBus were discovered within a few minutes.\ud Summarizing, we firmly believe that the formal engineering approach is cost-effective, and produces high quality software products. Model checking does yield significantly better models, but is also costly. Thus, system developers should trade off higher model quality against higher costs

Conformance Testing based on UML State Machines: Automated Test Case Generation, Execution and Evaluation

We describe a comprehensive approach for conformance testing of reactive systems. Based on a formal specification, namely UML state machines, we automatically generate test cases and use them to test the input-output conformance of a system under test. The test cases include not only the stimuli to trigger the system under test, they also include the test oracles to automatically evaluate the test execution. In contrast to Harel Statecharts, state machines behave asynchronously, which makes automatic test case generation a particular challenge. As a prerequisite we have completely formalized a substantial subset of UML state machines that includes complex structured data. The TEAGER tool suite implements our test approach and proves its applicability