Potential Errors and Test Assessment in Software Product Line Engineering

Software product lines (SPL) are a method for the development of variant-rich software systems. Compared to non-variable systems, testing SPLs is extensive due to an increasingly amount of possible products. Different approaches exist for testing SPLs, but there is less research for assessing the quality of these tests by means of error detection capability. Such test assessment is based on error injection into correct version of the system under test. However to our knowledge, potential errors in SPL engineering have never been systematically identified before. This article presents an overview over existing paradigms for specifying software product lines and the errors that can occur during the respective specification processes. For assessment of test quality, we leverage mutation testing techniques to SPL engineering and implement the identified errors as mutation operators. This allows us to run existing tests against defective products for the purpose of test assessment. From the results, we draw conclusions about the error-proneness of the surveyed SPL design paradigms and how quality of SPL tests can be improved.Comment: In Proceedings MBT 2015, arXiv:1504.0192

Mutation testing from probabilistic finite state machines

Mutation testing traditionally involves mutating a program in order to produce a set of mutants and using these mutants in order to either estimate the effectiveness of a test suite or to drive test generation. Recently, however, this approach has been applied to specifications such as those written as finite state machines. This paper extends mutation testing to finite state machine models in which transitions have associated probabilities. The paper describes several ways of mutating a probabilistic finite state machine (PFSM) and shows how test sequences that distinguish between a PFSM and its mutants can be generated. Testing then involves applying each test sequence multiple times, observing the resultant output sequences and using results from statistical sampling theory in order to compare the observed frequency of each output sequence with that expected

A set of metrics for characterizing simulink model comprehension

Simulink is a powerful tool for Embedded Systems, playing a key role in dynamic systems modeling. However, far too little attention has been paid to quality of Simulink models. In addition, no research has been found linking the relationship between model complexity and its impact in the comprehension quality of Simulink models. The aim of this paper is to define a set of metrics to support the characterization of Simulink models and to investigate their relationship with the model comprehension property. For this study, we performed a controlled experiment using two versions of a robotic Simulink model — one of them was constructed through the ad hoc development approach and the other one through the re-engineered development approach. The results of the experiment show that the re-engineered model is more comprehensible than the ad hoc model. In summary, the set of metrics collected from each version of the Simulink model suggests an inverse relationship with the model comprehension, i.e., the lower the metrics, the greater the model comprehension.Facultad de Informátic

A set of metrics for characterizing simulink model comprehension

Simulink is a powerful tool for Embedded Systems, playing a key role in dynamic systems modeling. However, far too little attention has been paid to quality of Simulink models. In addition, no research has been found linking the relationship between model complexity and its impact in the comprehension quality of Simulink models. The aim of this paper is to define a set of metrics to support the characterization of Simulink models and to investigate their relationship with the model comprehension property. For this study, we performed a controlled experiment using two versions of a robotic Simulink model — one of them was constructed through the ad hoc development approach and the other one through the re-engineered development approach. The results of the experiment show that the re-engineered model is more comprehensible than the ad hoc model. In summary, the set of metrics collected from each version of the Simulink model suggests an inverse relationship with the model comprehension, i.e., the lower the metrics, the greater the model comprehension.Facultad de Informátic

A set of metrics for characterizing simulink model comprehension

Simulink is a powerful tool for Embedded Systems, playing a key role in dynamic systems modeling. However, far too little attention has been paid to quality of Simulink models. In addition, no research has been found linking the relationship between model complexity and its impact in the comprehension quality of Simulink models. The aim of this paper is to define a set of metrics to support the characterization of Simulink models and to investigate their relationship with the model comprehension property. For this study, we performed a controlled experiment using two versions of a robotic Simulink model — one of them was constructed through the ad hoc development approach and the other one through the re-engineered development approach. The results of the experiment show that the re-engineered model is more comprehensible than the ad hoc model. In summary, the set of metrics collected from each version of the Simulink model suggests an inverse relationship with the model comprehension, i.e., the lower the metrics, the greater the model comprehension.Facultad de Informátic

Testbench qualification of SystemC TLM protocols through Mutation Analysis

Transaction-level modeling (TLM) has become the de-facto reference modeling style for system-level design and verification of embedded systems. It allows designers to implement high-level communication protocols for simulations up to 1000x faster than at register-transfer level (RTL). To guarantee interoperability between TLM IP suppliers and users, designers implement the TLM communication protocols by relying on a reference standard, such as the standard OSCI for SystemC TLM. Functional correctness of such protocols as well as their compliance to the reference TLM standard are usually verified through user-defined testbenches, which high-quality and completeness play a key role for an efficient TLM design and verification flow. This article presents a methodology to apply mutation analysis, a technique applied in literature for SW testing, for measuring the testbench quality in verifying TLM protocols. In particular, the methodology aims at (i) qualifying the testbenches by considering both the TLM protocol correctness and their compliance to a defined standard (i.e., OSCI TLM), (ii) optimizing the simulation time during mutation analysis by avoiding mutation redundancies, and (iii) driving the designers in the testbench improvement. Experimental results on benchmarks of different complexity and architectural characteristics are reported to analyze the methodology applicability

Equivalence checking of NuSMV specifications

We present a technique for checking the equivalence of NuSMV specifications. The approach is founded on the notion of equivalence between Kripke structures. The necessity to tackle this problem arisen working on using mutation to asses the static analysis fault detection capability. Indeed, mutation, consisting into introducing simple syntactic changes -- representing typical mistakes designers often make -- into specifications, may produce equivalent mutants, namely models behaving as the original one. Equivalent mutants should be detected since they do not represent actual faults. In program mutation, detecting equivalent mutants is an undecidable problem and, when possible, is a time-consuming activity, difficult to automatize. In this work we focus on how detecting equivalence of NuSMV specifications. The novel technique we propose, consists in building a merging unique specification and proving by model checking a series of CTL properties

Mutation Testing Applied to Validate SDL Specifications

Abstract. Mutation Testing is an error-based criterion that provides mechanisms to evaluate the quality of a test set and/or to generate test sets. This criterion, originally proposed to program testing, has also been applied to specification testing. In this paper, we propose the application of Mutation Testing for testing SDL specifications. We define a mutant operator set for SDL that intends to model errors related to the behavioral aspect of the processes, the communication among processes, the structure of the specification and some intrinsic characteristics of SDL. A testing strategy to apply the mutant operators to test SDL specifications is proposed. We illustrate our approach using the Alternating-Bit protocol

Search-based Similarity-driven Behavioural SPL Testing

International audienceDissimilar test cases have been proven to be effective to reveal faults in software systems. In the Software Product Line (SPL) context, this criterion has been applied successfully to mimic combinatorial interaction testing in an efficient and scalable manner by selecting and prioritising most dissimilar configurations of feature models using evolutionary algorithms. In this paper, we extend dissimilarity to behavioural SPL models (FTS) in a search-based approach, and evaluate its effectiveness in terms of product and fault coverage. We investigate different distances as well as as single-objective algorithms, (dissimilarity on actions, random , all-actions). Our results on four case studies show the relevance of dissimilarity-based test generation for be-havioural SPL models, especially on the largest case-study where no other approach can match it

Web services choreography testing using semantic service description

Web services have become popular due to their ability to integrate with and to interoperate heterogeneous applications. Several web services can be combined into a single application to meet the needs of users. In the course of web services selection, a web candidate service needs to conform to the behaviour of its client, and one way of ensuring this conformity is by testing the interaction between the web service and its user. The existing web services test approaches mainly focus on syntax-based web services description, whilst the semantic-based solutions mostly address composite process flow testing. The aim of this research is to provide an automated testing approach to support service selection during automatic web services composition using Web Service Modeling Ontology (WSMO). The research work began with understanding and analysing the existing test generation approaches for web services. Second, the weaknesses of the existing approaches were identified and addressed by utilizing the choreography transition rules of WSMO in an effort to generate a Finite State Machine (FSM). The FSM was then used to generate the working test cases. Third, a technique to generate an FSM from Abstract State Machine (ASM) was adapted to be used with WSMO. This thesis finally proposed a new testing model called the Choreography to Finite State Machine (C2FSM) to support the service selection of an automatic web service composition. It proposed new algorithms to automatically generate the test cases from the semantic description (WSMO choreography description). The proposed approach was then evaluated using the Amazon E-Commerce Web Service WSMO description. The quality of the test cases generated using the proposed approach was measured by assessing their mutation adequacy score. A total of 115 mutants were created based on 7 mutant operators. A mutation adequacy score of 0.713 was obtained. The experimental validation demonstrated a significant result in the sense that C2FSM provided an efficient and feasible solution. The result of this research could assist the service consumer agents in verifying the behaviour of the Web service in selecting appropriate services for web service composition