Experimental evaluation of SDL and One-Op mutation for C

Mutation analysis modifies a program by applying\ud syntactic rules, called mutation operators, systematically to create\ud many versions of the program (mutants) that differ in small\ud ways. Testers then design tests to cause the mutants to behave\ud differently from the original program. Mutation testing is widely\ud considered to result in very effective tests, however, it is also quite\ud costly. Cost comes from the many mutants that are created, the\ud number of tests that are needed to kill the mutants, and the\ud difficulty of deciding whether mutants behave equivalently to\ud the original program. One-op mutation theorizes that cost can be\ud reduced by using a single, very powerful, mutation operator that\ud leads to tests that are almost as effective as if all operators are\ud used. Previous research proposed the statement deletion operator\ud (SDL) and found promising results. This paper investigates the\ud use of SDL-mutation in a new context, the language C, and poses\ud additional empirical questions, including whether other operators\ud can be used. We carried out a controlled experiment in which\ud cost and effectiveness of each individual C mutation operator\ud were collected for 39 different subject programs. Experimental\ud data are used to define a cost-effectiveness metric to choose the\ud best single operator for one-op mutation.FAPESP (número processo 2012/16950-5

Potential Errors and Test Assessment in Software Product Line Engineering

Software product lines (SPL) are a method for the development of variant-rich software systems. Compared to non-variable systems, testing SPLs is extensive due to an increasingly amount of possible products. Different approaches exist for testing SPLs, but there is less research for assessing the quality of these tests by means of error detection capability. Such test assessment is based on error injection into correct version of the system under test. However to our knowledge, potential errors in SPL engineering have never been systematically identified before. This article presents an overview over existing paradigms for specifying software product lines and the errors that can occur during the respective specification processes. For assessment of test quality, we leverage mutation testing techniques to SPL engineering and implement the identified errors as mutation operators. This allows us to run existing tests against defective products for the purpose of test assessment. From the results, we draw conclusions about the error-proneness of the surveyed SPL design paradigms and how quality of SPL tests can be improved.Comment: In Proceedings MBT 2015, arXiv:1504.0192

An integrated search-based approach for automatic testing from extended finite state machine (EFSM) models

This is the post-print version of the Article - Copyright @ 2011 ElsevierThe extended finite state machine (EFSM) is a modelling approach that has been used to represent a wide range of systems. When testing from an EFSM, it is normal to use a test criterion such as transition coverage. Such test criteria are often expressed in terms of transition paths (TPs) through an EFSM. Despite the popularity of EFSMs, testing from an EFSM is difficult for two main reasons: path feasibility and path input sequence generation. The path feasibility problem concerns generating paths that are feasible whereas the path input sequence generation problem is to find an input sequence that can traverse a feasible path. While search-based approaches have been used in test automation, there has been relatively little work that uses them when testing from an EFSM. In this paper, we propose an integrated search-based approach to automate testing from an EFSM. The approach has two phases, the aim of the first phase being to produce a feasible TP (FTP) while the second phase searches for an input sequence to trigger this TP. The first phase uses a Genetic Algorithm whose fitness function is a TP feasibility metric based on dataflow dependence. The second phase uses a Genetic Algorithm whose fitness function is based on a combination of a branch distance function and approach level. Experimental results using five EFSMs found the first phase to be effective in generating FTPs with a success rate of approximately 96.6%. Furthermore, the proposed input sequence generator could trigger all the generated feasible TPs (success rate = 100%). The results derived from the experiment demonstrate that the proposed approach is effective in automating testing from an EFSM

Semantic mutation testing

This is the Pre-print version of the Article. The official published version can be obtained from the link below - Copyright @ 2011 ElsevierMutation testing is a powerful and flexible test technique. Traditional mutation testing makes a small change to the syntax of a description (usually a program) in order to create a mutant. A test suite is considered to be good if it distinguishes between the original description and all of the (functionally non-equivalent) mutants. These mutants can be seen as representing potential small slips and thus mutation testing aims to produce a test suite that is good at finding such slips. It has also been argued that a test suite that finds such small changes is likely to find larger changes. This paper describes a new approach to mutation testing, called semantic mutation testing. Rather than mutate the description, semantic mutation testing mutates the semantics of the language in which the description is written. The mutations of the semantics of the language represent possible misunderstandings of the description language and thus capture a different class of faults. Since the likely misunderstandings are highly context dependent, this context should be used to determine which semantic mutants should be produced. The approach is illustrated through examples with statecharts and C code. The paper also describes a semantic mutation testing tool for C and the results of experiments that investigated the nature of some semantic mutation operators for C

A path-aware approach to mutant reduction in mutation testing

Context: Mutation testing, which systematically generates a set of mutants by seeding various faults into the base program under test, is a popular technique for evaluating the effectiveness of a testing method. However, it normally requires the execution of a large amount of mutants and thus incurs a high cost. Objective: A common way to decrease the cost of mutation testing is mutant reduction, which selects a subset of representative mutants. In this paper, we propose a new mutant reduction approach from the perspective of program structure. Method: Our approach attempts to explore path information of the program under test, and select mutants that are as diverse as possible with respect to the paths they cover. We define two path-aware heuristic rules, namely module-depth and loop-depth rules, and combine them with statement- and operator-based mutation selection to develop four mutant reduction strategies. Results: We evaluated the cost-effectiveness of our mutant reduction strategies against random mutant selection on 11 real-life C programs with varying sizes and sampling ratios. Our empirical studies show that two of our mutant reduction strategies, which primarily rely on the path-aware heuristic rules, are more effective and systematic than pure random mutant selection strategy in terms of selecting more representative mutants. In addition, among all four strategies, the one giving loop-depth the highest priority has the highest effectiveness. Conclusion: In general, our path-aware approach can reduce the number of mutants without jeopardizing its effectiveness, and thus significantly enhance the overall cost-effectiveness of mutation testing. Our approach is particularly useful for the mutation testing on large-scale complex programs that normally involve a huge amount of mutants with diverse fault characteristics

Designing deletion mutation operators

Mutation analysis modifies a program by applying\ud syntactic rules, called mutation operators, systematically to create\ud many versions of the program (mutants) that differ in small\ud ways. Testers then design tests to cause the mutants to behave\ud differently from the original program. Mutation testing is widely\ud considered to result in very effective tests, however, it is also quite\ud costly. Cost comes from the many mutants that are created, the\ud number of tests that are needed to kill the mutants, and the\ud difficulty of deciding whether mutants behave equivalently to\ud the original program. One-op mutation theorizes that cost can be\ud reduced by using a single, very powerful, mutation operator that\ud leads to tests that are almost as effective as if all operators are\ud used. Previous research proposed the statement deletion operator\ud (SDL) and found promising results. This paper investigates the\ud use of SDL-mutation in a new context, the language C, and poses\ud additional empirical questions, including whether other operators\ud can be used. We carried out a controlled experiment in which\ud cost and effectiveness of each individual C mutation operator\ud were collected for 39 different subject programs. Experimental\ud data are used to define a cost-effectiveness metric to choose the\ud best single operator for one-op mutation.FAPESP (número processo 2012/16950-5

A search-based technique for testing from extended finite state machine model

Extended finite state machines (EFSMs), and languages such as state-charts that are similar to EFSMs, are widely used to model state-based systems. When testing from an EFSM M it is common to aim to produce a set of test sequences (input sequences) that satisfies a test criterion that relates to the transition paths (TPs) of M that are executed by the test sequences. For example, we might require that the set of TPs triggered includes all of the transitions of M. One approach to generating such a set of test sequences is to split the problem into two stages: choosing a set of TPs that achieves the test criterion and then producing test sequences to trigger these TPs. However, the EFSM may contain infeasible TPs and the problem of generating a test sequence to trigger a given feasible TP (FTP) is generally uncomputable. In this paper we present a search-based approach that uses two techniques: (1) A TP fitness metric based on our previous work that estimates the feasibility of a given transition path; and (2) A fitness function to guide the search for a test sequence to trigger a given FTP. We evaluated our approach on five EFSMs: A simple in-flight safety system; a class II transport protocol; a lift system; an ATM; and the Inres initiator. In the experiments the proposed approach successfully tested approximately 96.75 % of the transitions and the proposed test sequence generation technique triggered all of the generated FTPs