Mutation Analysis for the Evaluation of AD Models

UML has became the industry standard for analysis and design modeling. Model is a key artifact in Model Driven Architect (MDA) and considered as an only concrete artifact available at earlier development stages. Error detection at earlier development stages can save enormous amount of cost and time. The article presents a novel mutation analysis technique for UML 2.0 Activity Diagram (AD). Based on the AD oriented fault types, a number of mutation operators are defined. The technique focuses on the key features of AD and enhances the confidence in design correctness by showing the absence of control-flow and concurrency related faults. It will enable the automated analysis technique of AD models and can potentially be used for service oriented applications, workflows and concurrent applications

Mutation testing from probabilistic finite state machines

Mutation testing traditionally involves mutating a program in order to produce a set of mutants and using these mutants in order to either estimate the effectiveness of a test suite or to drive test generation. Recently, however, this approach has been applied to specifications such as those written as finite state machines. This paper extends mutation testing to finite state machine models in which transitions have associated probabilities. The paper describes several ways of mutating a probabilistic finite state machine (PFSM) and shows how test sequences that distinguish between a PFSM and its mutants can be generated. Testing then involves applying each test sequence multiple times, observing the resultant output sequences and using results from statistical sampling theory in order to compare the observed frequency of each output sequence with that expected

Equivalence checking of NuSMV specifications

We present a technique for checking the equivalence of NuSMV specifications. The approach is founded on the notion of equivalence between Kripke structures. The necessity to tackle this problem arisen working on using mutation to asses the static analysis fault detection capability. Indeed, mutation, consisting into introducing simple syntactic changes -- representing typical mistakes designers often make -- into specifications, may produce equivalent mutants, namely models behaving as the original one. Equivalent mutants should be detected since they do not represent actual faults. In program mutation, detecting equivalent mutants is an undecidable problem and, when possible, is a time-consuming activity, difficult to automatize. In this work we focus on how detecting equivalence of NuSMV specifications. The novel technique we propose, consists in building a merging unique specification and proving by model checking a series of CTL properties

Mutation Testing Applied to Validate SDL Specifications

Abstract. Mutation Testing is an error-based criterion that provides mechanisms to evaluate the quality of a test set and/or to generate test sets. This criterion, originally proposed to program testing, has also been applied to specification testing. In this paper, we propose the application of Mutation Testing for testing SDL specifications. We define a mutant operator set for SDL that intends to model errors related to the behavioral aspect of the processes, the communication among processes, the structure of the specification and some intrinsic characteristics of SDL. A testing strategy to apply the mutant operators to test SDL specifications is proposed. We illustrate our approach using the Alternating-Bit protocol

Software engineering: Testing real-time embedded systems using timed automata based approaches

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Real-time Embedded Systems (RTESs) have an increasing role in controlling society infrastructures that we use on a day-to-day basis. RTES behaviour is not based solely on the interactions it might have with its surrounding environment, but also on the timing requirements it induces. As a result, ensuring that an RTES behaves correctly is non-trivial, especially after adding time as a new dimension to the complexity of the testing process. This research addresses the problem of testing RTESs from Timed Automata (TA) specification by the following. First, a new Priority-based Approach (PA) for testing RTES modelled formally as UPPAAL timed automata (TA variant) is introduced. Test cases generated according to a proposed timed adequacy criterion (clock region coverage) are divided into three sets of priorities, namely boundary, out-boundary and in-boundary. The selection of which set is most appropriate for a System Under Test (SUT) can be decided by the tester according to the system type, time specified for the testing process and its budget. Second, PA is validated in comparison with four well-known timed testing approaches based on TA using Specification Mutation Analysis (SMA). To enable the validation, a set of timed and functional mutation operators based on TA is introduced. Three case studies are used to run SMA. The effectiveness of timed testing approaches are determined and contrasted according to the mutation score which shows that our PA achieves high mutation adequacy score compared with others. Third, to enhance the applicability of PA, a new testing tool (GeTeX) that deploys PA is introduced. In its current version, GeTeX supports Control Area Network (CAN) applications. GeTeX is validated by developing a prototype for that purpose. Using GeTeX, PA is also empirically validated in comparison with some TA testing approaches using a complete industrial-strength test bed. The assessment is based on fault coverage, structural coverage, the length of generated test cases and a proposed assessment factor. The assessment is based on fault coverage, structural coverage, the length of generated test cases and a proposed assessment factor. The assessment results confirmed the superiority of PA over the other test approaches. The overall assessment factor showed that structural and fault coverage scores of PA with respect to the length of its tests were better than the others proving the applicability of PA. Finally, an Analytical Hierarchy Process (AHP) decision-making framework for our PA is developed. The framework can provide testers with a systematic approach by which they can prioritise the available PA test sets that best fulfils their testing requirements. The AHP framework developed is based on the data collected heuristically from the test bed and data collected by interviewing testing experts. The framework is then validated using two testing scenarios. The decision outcomes of the AHP framework were significantly correlated to those of testing experts which demonstrated the soundness and validity of the framework.This study is funded by Damascus University, Syri

Un lenguaje de dominio específico para mutación de modelos

Tesis doctoral inédita leída en la Universidad Autónoma de Madrid, Escuela Politécnica Superior, Departamento de Ingeniería Informática. Fecha de lectura: 14-07-2020Las técnicas de mutación de software se utilizan en campos diversos como las pruebas de mutación, la pruebas de programas, la prueba de fórmulas lógicas, los algoritmos genéticos y la generación automática de ejercicios. Las soluciones existentes suelen utilizar un enfoque a medida, construyendo desde cero una solución específica para el problema que pretenden abordar. Este enfoque conlleva un esfuerzo elevado de implementación para cada caso, es propenso a errores, supone un alto coste de mantenimiento, y tiene como resultado la creación de numerosas soluciones cada una de las cuáles sólo resuelve un problema en concreto. Con el objetivo de proporcionar un enfoque genérico que ayude a superar estos inconvenientes, esta tesis presenta un lenguaje de dominio específico para mutación de modelos llamado Wodel, y su entorno de desarrollo. Wodel es independiente del dominio y puede utilizarse con cualquier lenguaje definido por medio de un meta-modelo. Incluye primitivas de mutación para creación, modificación, borrado, retipado y clonado de objetos, y para creación, modificación y borrado de referencias. Wodel proporciona facilidades de soporte al proceso de mutación, como la validación de los mutantes, un registro de las mutaciones aplicadas y la detección de mutantes equivalentes. También da soporte a la ingeniería de operadores de mutación mediante la generación de métricas de mutación y la síntesis automática de modelos semilla que aseguran la cobertura de todas las instrucciones de un programa Wodel, facilitando de este modo su prueba y validación. Además, Wodel es extensible y permite aprovechar sus funcionalidades para la mutación de modelos en aplicaciones de post-procesado. Para ilustrar dicha extensibilidad, esta tesis presenta dos de estas extensiones a Wodel: una para la generación automática de ejercicios de auto-evaluación para estudiantes, a la que se ha llamado Wodel-Edu; y otra para facilitar la creación de herramientas de pruebas de mutación para lenguajes de programación o de modelado, a la que se ha llamado Wodel-TestSoftware mutation techniques are used in different fields such as mutation testing, software testing, logic formulas testing, genetic algorithms and the automated generation of exercises. The existing solutions are usually ad-hoc, creating from scratch a specific solution for the faced problem. This approach involves a high implementation effort, it is error-prone, entails a high maintenance cost, and results in a variety of solutions each of which only solves a particular problem. With the purpose of providing a generic approach that alleviates these inconveniences, this thesis introduces a domain-specific language for model mutation called Wodel, and its development environment. Wodel is domain independent and can be used with any arbitrary language defined by a meta-model. It includes mutation primitives to create, modify, delete, retype and clone objects, and to create, modify and delete references. Wodel provides facilities to simplify the mutation process, such as model validation, a registry of the applied mutations and the detection of equivalent mutants. It also supports the engineering of mutation operators by the generation of metrics of the mutation programs and the automated synthesis of seed models which ensure full coverage of the statements in a Wodel program, hence easing its testing and validation. Additionally, Wodel is extensible and permits taking advantage of all its model mutation functionalities in post-processing applications. To illustrate this extensibility capability, this thesis reports on two of these extensions to Wodel: the first one targeted to the automated generation of self-assessment exercises for students, called Wodel-Edu; and the second one targeted to ease the creation of mutation testing tools for programming or modelling languages, called Wodel-Tes

Model based test suite minimization using metaheuristics

Software testing is one of the most widely used methods for quality assurance and fault detection purposes. However, it is one of the most expensive, tedious and time consuming activities in software development life cycle. Code-based and specification-based testing has been going on for almost four decades. Model-based testing (MBT) is a relatively new approach to software testing where the software models as opposed to other artifacts (i.e. source code) are used as primary source of test cases. Models are simplified representation of a software system and are cheaper to execute than the original or deployed system. The main objective of the research presented in this thesis is the development of a framework for improving the efficiency and effectiveness of test suites generated from UML models. It focuses on three activities: transformation of Activity Diagram (AD) model into Colored Petri Net (CPN) model, generation and evaluation of AD based test suite and optimization of AD based test suite. Unified Modeling Language (UML) is a de facto standard for software system analysis and design. UML models can be categorized into structural and behavioral models. AD is a behavioral type of UML model and since major revision in UML version 2.x it has a new Petri Nets like semantics. It has wide application scope including embedded, workflow and web-service systems. For this reason this thesis concentrates on AD models. Informal semantics of UML generally and AD specially is a major challenge in the development of UML based verification and validation tools. One solution to this challenge is transforming a UML model into an executable formal model. In the thesis, a three step transformation methodology is proposed for resolving ambiguities in an AD model and then transforming it into a CPN representation which is a well known formal language with extensive tool support. Test case generation is one of the most critical and labor intensive activities in testing processes. The flow oriented semantic of AD suits modeling both sequential and concurrent systems. The thesis presented a novel technique to generate test cases from AD using a stochastic algorithm. In order to determine if the generated test suite is adequate, two test suite adequacy analysis techniques based on structural coverage and mutation have been proposed. In terms of structural coverage, two separate coverage criteria are also proposed to evaluate the adequacy of the test suite from both perspectives, sequential and concurrent. Mutation analysis is a fault-based technique to determine if the test suite is adequate for detecting particular types of faults. Four categories of mutation operators are defined to seed specific faults into the mutant model. Another focus of thesis is to improve the test suite efficiency without compromising its effectiveness. One way of achieving this is identifying and removing the redundant test cases. It has been shown that the test suite minimization by removing redundant test cases is a combinatorial optimization problem. An evolutionary computation based test suite minimization technique is developed to address the test suite minimization problem and its performance is empirically compared with other well known heuristic algorithms. Additionally, statistical analysis is performed to characterize the fitness landscape of test suite minimization problems. The proposed test suite minimization solution is extended to include multi-objective minimization. As the redundancy is contextual, different criteria and their combination can significantly change the solution test suite. Therefore, the last part of the thesis describes an investigation into multi-objective test suite minimization and optimization algorithms. The proposed framework is demonstrated and evaluated using prototype tools and case study models. Empirical results have shown that the techniques developed within the framework are effective in model based test suite generation and optimizatio