Rule-based Test Generation with Mind Maps

This paper introduces basic concepts of rule based test generation with mind maps, and reports experiences learned from industrial application of this technique in the domain of smart card testing by Giesecke & Devrient GmbH over the last years. It describes the formalization of test selection criteria used by our test generator, our test generation architecture and test generation framework.Comment: In Proceedings MBT 2012, arXiv:1202.582

Towards flexible and efficient model-based testing, utilizing domain-specific modelling

Model-Based Testing is a test automation technique that generates test cases based on a model of the system under test. Domain-specific modelling is a modelling approach where the developed system is modelled in terms of domain-specific concepts and these models are automatically transformed to other forms such as application code. In this paper, we will discuss the adoption and integration of domain-specific modelling with model-based testing tools. Since model-based testing tools utilise various modelling notations that typically diverge from a specific domain-model, we will discuss how domain specific models can be automatically transformed to become suitable models for a chosen model-based testing tool. Furthermore, by doing this in terms of a domain-specific meta-model, we will allow one to switch between various model-based testing tools

Applications Based on Generator of Test Cases

Cílem této bakalářské práce je vytvořit přehled aktuálního stavu nástrojů, které umožňují automaticky generovat testovací případy. Dále na příkladu zvoleného nástroje ukázat způsob práce s ním a jeho schopnosti v generaci spustitelných testu. Pro následující práci byl vybrán nástroj UPPAAL, který umožňuje vytvořit model zvoleného systému pomocí časovaných automatů, ověřit a simulovat jeho běh a následně vytvořit testovací případ pro daný systém. Ve výsledku nástroj vygeneruje cestu průchodu systémem, kterou je možné uložit ve formě spustitelného testovacího případu a to v libovolném programovacím jazyce. Pro testování byly zvoleny tři různé systémy: systém vypínače světla, implementovaný v jazyce Java; 2-bitová násobička, jejíž chování je popsané pomocí jazyka Verilog; a systém zjednodušeného výtahu, který je představen v jazyce C. Ve výsledku byly získány spustitelné testovací případy pro zvolené systémy, spolu s jejich vlastnostmi jako pokrytí systému, počet kroků pro dosažení cílů a kvalita vygenerovaných cest.The aim of this bachelor's thesis is to create an overview of the current state of tools that allow automatic generation of test cases and select one tool to show how it works and its ability to generate executable tests. The UPPAAL program was chosen for the following work. Tool allows to create a model of the selected system using timed automata, verify and simulate its operation and create a test case for the system. In the results, the tool generates a path through the system, which can be saved in the form of executable test cases in any programming language. Three different systems were chosen for testing: a light switch system implemented in Java; 2-bit multiplier, which behavior is described by Verilog language; and a simplified elevator system, which working process is introduced in C language. As a result, executable test cases were obtained for selected systems along with their features such as system coverage, number of steps to achieve goals, and quality of generated paths.

Test Coverage Analysis

Hoje em dia, à medida que o software tende a assumir um papel cada vez mais crítico, a necessidade de garantir a sua qualidade torna-se cada vez mais crucial. Várias abordagens surgiram para endereçar esta necessidade, ou seja, ferramentas e processos de teste de software que ajudam a aumentar os níveis de qualidade em virtualmente qualquer tipo de sistema de software. Um exemplo de tais ferramentas são as designadas de ferramentas de Testes-Baseados em Modelos (MBT - Model-Based Testing) , que geram casos de teste a partir de modelos do sistema a testar. A qualidade dos testes gerados pode ser avaliada através de diversos meios, quer como uma percentagem dos elementos ou código 'cobertos', e é de extrema importância obter essas medições de modo a determinar quando parar de testar. No entanto, muitas vezes as ferramentas de teste não providenciam esta informação como esperado. Este projecto apresenta uma ferramenta de análise de cobertura, PARADIGM-COV, avalia analisa a qualidade dos testes gerados por uma ferramenta MBT existente, PARADIGM-ME. A ferramenta proposta produz informação sobre a cobertura de elementos de um modelo gráfico (útil para avaliar a qualidade dos casos de teste, assim como para ajudar o tester a fornecer dados apropriados durante a fase de configuração), e também a percentagem do sistema sob teste que foi atingida durante a execução dos casos de teste.Nowadays, as software tends to assume increasingly critical roles, the need to assure its quality becomes ever more crucial. Several approaches emerged to help overcome this need, i.e., tools and processes of software testing that help increase levels of quality in virtually any type of software system. One example of such tools is the so called Model-Based Testing (MBT) tools, that generate test cases from system models. The generated test cases' quality can be evaluated by various means, either as a percentage of the model elements or code covered, and it is of utmost importance to obtain those measurements in order to determine when to stop testing. However, often times testing tools do not provide this information as expected.This work concerns the implementation of a coverage tool, PARADIGM-COV, that evaluates the quality of the test cases generated by an existing MBT tool, PARADIGM-ME. The proposed tool produces coverage information over the graphical model elements, useful not only to assess test case quality, but also in aiding the tester to provide appropriate data during configuration phase, providing in addition the percentage of the application under test code that was exercised during test case execution

Applying model based testing approach in electric vehicle charging system : MyEV charging system case study

One of the major aspect in embedded system is the design itself. However, it is usually being neglected through consideration even in conception phase where in later part causing discovery of fault and error to the system.This report discusses the current issue that have been arousing the embedded system testing specifically during design phase. Currently, there are few model based testing approach that have been introduced in embedded system field using model diagram. Existing technique of model based testing approach were studied and analysed to come out with the best testing approach to be implemented in electric vehicle charging system. MyEV system has been selected as case study of this approach

Experiences with formal engineering: model-based specification, implementation and testing of a software bus at Neopost

We report on the actual industrial use of formal methods during the development of a software bus. During an internship at Neopost Inc., of 14 weeks, we developed the server component of a software bus, called the XBus, using formal methods during the design, validation and testing phase: we modeled our design of the XBus in the process algebra mCRL2, validated the design using the mCRL2-simulator, and fully automatically tested our implementation with the model-based test tool JTorX. This resulted in a well- tested software bus with a maintainable architecture. Writing the model (mdev), simulating it, and testing the implementation with JTorX only took 17% of the total development time. Moreover, the errors found with model-based testing would have been hard to find with conventional test methods. Thus, we show that formal engineering can be feasible, beneficial and cost-effective.\ud The findings above, reported earlier by us in (Sijtema et al., 2011) [1], were well- received, also in industrially oriented conferences (Ferreira and Romanenko, 2010) [2] and [3]. In this paper, we look back on the case study, and carefully analyze its merits and shortcomings. We reflect on (1) the added benefits of model checking, (2) model completeness and (3) the quality and performance of the test process.\ud Thus, in a second phase, after the internship, we model checked the XBus protocol—this was not done in [1] since the Neopost business process required a working implementation after 14 weeks. We used the CADP tool evaluator4 to check the behavioral requirements obtained during the development. Model checking did not uncover errors in model mdev, but revealed that model mdev was neither complete nor optimized: in particular, requirements to the so-called bad weather behavior (exceptions, unexpected inputs, etc.) were missing. Therefore, we created several improved models, checked that we could validate them, and used them to analyze quality and performance of the test process. Model checking was expensive: it took us approx. 4 weeks in total, compared to 3 weeks for the entire model-based testing approach during the internship.\ud In the second phase, we analyzed the quality and performance of the test process, where we looked at both code and model coverage. We found that high code coverage (almost 100%) is in most cases obtained within 1000 test steps and 2 minutes, which matches the fact that the faults in the XBus were discovered within a few minutes.\ud Summarizing, we firmly believe that the formal engineering approach is cost-effective, and produces high quality software products. Model checking does yield significantly better models, but is also costly. Thus, system developers should trade off higher model quality against higher costs

Model based approach to software testing

Ohjelmistotestaus vaikeutuu päivä päivältä koska ohjelmistojen suunnittelu- ja testausvaiheet ovat selkeästi toisistaan erillisiä toimintoja, jonka johdosta testitapaukset eivät vastaa ohjelmiston oikeaa toiminnallisuutta. Eräs tapa ratkaista tämä ongelma on ottaa käyttöön malli joka kuvaa ohjelmiston toimintaa esimerkiksi käyttöliittymätasolla. Mallista voidaan mallipohjaisella testauksella tuottaa automaattisesti tarvittavat testitapaukset. Mallista voidaan tuottaa automaattisesti myös esimerkiksi lähdekoodia. Tämän työn tavoitteina oli löytää tapa, jolla mallipohjaisen lähestymistavan voisi tuoda organisaation käyttöön helposti, kehittää mallipohjainen testaustyökalu, jotta lähestymistapaa voitaisiin testata sekä todistaa että lähestymistapa tarjoaa taloudellisia hyötyjä. Työ on jaettu kahteen osaan. Ensin, kirjallisuusosassa, mallipohjainen lähestymistapa käydään läpi. Toisessa osassa esitellään ehdotus lähestymistavan tuomiseksi organisaation, käydään läpi ehdotukseen perustuvan mallipohjaisen testaustyökalun implementaatio sekä verrataan mallipohjaista testausta perinteisiin testausmenetelmiin. Tehty vertailu todisti että mallipohjainen testaus tarjoaa suuria taloudellisia hyötyjä. Tekniikkaa käyttämällä säästettiin paljon testaukseen käytettyjä miestyötunteja.Software testing is becoming more and more difficult task every day because in the current software engineering cycle the design and testing activities are separated, which leads to a situation where test cases are not in harmony with the actual application. One way to solve this problem is to take a model of the application into use, which can be, e.g., a user interface model and from which the test cases can be derived automatically. This technique is known as model based testing. The model can also be used for, e.g., automatic code generation. The main objectives of this thesis were to find a suitable way to deploy the model based approach into an organization, to develop a model based testing tool in order to test the approach and to show proof of financial benefits of the approach. The thesis is divided into two parts. First, in the literature part, the model based approach is explored thoroughly. In the second part a suggestion for the deployment of model based approach into an organization is presented, a model based testing tool implementation based on that suggestion is reviewed and a comparison between traditional testing methods and model based testing is conducted. The comparison between the testing methods showed that model based testing brings financial benefits to the organization. By using model based testing, a lot of time was saved in the test execution compared to the traditional testing methods

Model-Based Scenario Testing and Model Checking with Applications in the Railway Domain

This thesis introduces Timed Moore Automata, a specification formalism, which extends the classical Moore Automata by adding the concept of abstract timers without concrete delay time values, which can be started and reset, and which can change their state from running to elapsed. The formalism is used in real-world railway domain applications, and algorithms for the automated test data generation and explicit model checking of Timed Moore Automata models are presented. In addition, this thesis deals with test data generation for larger scale test models using standardized modeling formalisms. An existing framework for the automated test data generation is presented, and its underlying work-flow is extended and modified in order to allow user interaction and guidance within the generation process. As opposed to specifying generation constraints for entire test scenarios, the modified work flow then allows for an iterative approach to elaborating and formalizing test generation goals