Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?

Mutation testing is a means to assess the effectiveness of a test suite and its outcome is considered more meaningful than code coverage metrics. However, despite several optimizations, mutation testing requires a significant computational effort and has not been widely adopted in industry. Therefore, we study in this paper whether test effectiveness can be approximated using a more light-weight approach. We hypothesize that a test case is more likely to detect faults in methods that are close to the test case on the call stack than in methods that the test case accesses indirectly through many other methods. Based on this hypothesis, we propose the minimal stack distance between test case and method as a new test measure, which expresses how close any test case comes to a given method, and study its correlation with test effectiveness. We conducted an empirical study with 21 open-source projects, which comprise in total 1.8 million LOC, and show that a correlation exists between stack distance and test effectiveness. The correlation reaches a strength up to 0.58. We further show that a classifier using the minimal stack distance along with additional easily computable measures can predict the mutation testing result of a method with 92.9% precision and 93.4% recall. Hence, such a classifier can be taken into consideration as a light-weight alternative to mutation testing or as a preceding, less costly step to that.Comment: EASE 201

Exploring a Better Search–based Implementation on Second–Order Mutant Generation

Pengujian perangkat lunak adalah bagian dari proses pengembangan perangkat lunak, dengan tujuan utama untuk mengurangi/menghilangkan kesalahan pada perangkat lunak, hal ini umumnya dilakukan dengan menjalankan kasus-uji. Salah satu teknik untuk mengukur dan meningkatkan kualitas dari kasus uji adalah pengujian mutasi, tetapi walaupun sudah terbukti keefektifannya, teknik ini masih memiliki suatu kendala besar, yaitu tidak praktis untuk digunakan karena melibatkan pembangkitan dan eksekusi dari jumlah mutan yang besar. Belakangan ini penggunaan optimisasi berbasis-pencarian pada permasalahan pengujian perangkat lunak sedang popular. Pada penelitian ini, dilakukan eksplorasi penggunaan optimasi berbasis-pencarian pada pembangkitan mutan (variasi dari program), dengan tujuan untuk menghasilkan mutan yang tidak dapat dideteksi oleh kasus-uji, karena mutan jenis ini memiliki dapat kekurangan dari kasus-uji. Metode usulan dibandingkan dengan algoritma pembangkitan second-order mutant yang umum digunakan, dan juga dibandingkan dengan pendekatan berbasis pencarian lainnya. Hasil menunjukkan bahwa metode usulan dapat membangkitkan lebih banyak mutan tidak-terdeteksi (undetected-mutant) daripada dengan metode pembangkitan mutan yang umum. Metode usulan memiliki performansi yang lebih rendah daripada metode pembangkitan berbasis-pencarian benchmark, tetapi performansinya dapat ditingkatkan dengan melakukan perubahan pada representasi solusi, dan dengan adopsi parameter optimasi yang digunakan oleh metode pembanding. Software testing is a part of a software development process with a major concern is to reduce/eliminate fault in the software, and mainly done by executing a test case. One of the techniques for measuring and improving test case quality is mutation testing, but despite it is good effectiveness, this technique has a major problem that is impractical because it involves generation and execution of huge amount of mutant. This trend also happens in software testing, with the main focus on optimizing the test case generation. In this research, we explore the used of search-based optimization to the mutant (program variant) generation, with a goal to generate mutants that can escape test case detection, because these mutants have a probability to show test case deficiency. In this research, the proposed method is compared with a general second-order mutant generation algorithm and with other search-based mutant generation. The result shows that the proposed method can generate more undetected-mutant than a general second-order mutant generation. The proposed method performs worse than the benchmark search-based mutant generation, but this performance improved by altering it is solution representation and by the adoption of an optimization parameter

A Model to Estimate First-Order Mutation Coverage from Higher-Order Mutation Coverage

The test suite is essential for fault detection during software development. First-order mutation coverage is an accurate metric to quantify the quality of the test suite. However, it is computationally expensive. Hence, the adoption of this metric is limited. In this study, we address this issue by proposing a realistic model able to estimate first-order mutation coverage using only higher-order mutation coverage. Our study shows how the estimation evolves along with the order of mutation. We validate the model with an empirical study based on 17 open-source projects.Comment: 2016 IEEE International Conference on Software Quality, Reliability, and Security. 9 page

A New Mutant Generation Algorithm based on Path Coverage for Mutant Reduction

Mutation testing is a fault-based white-box testing technique that can be applied to evaluate the adequacy of a given test suite, but its application is very time-consuming due to the necessity of generating and executing a great number of mutants. How to reduce the cost still remains a challenge for research. In our research, we present a new mutant generation algorithm based on the basic path coverage to reduce mutants. The algorithm is characterized by implementing a basic path segment identification criterion for determining appropriate program points at which faults are inserted and a mutant generation priority criterion for selecting proper mutant operators to make a fault for insertion. We discuss the algorithm by analyzing how the two criteria are realized based on analyzing the control flow graph (CFG) of the program under test and applying effective mutation operators on the appropriate statements in the relevant path segments. We also present an automated mutation testing tool that supports the proposed approach, and a small experiment to evaluate our tool by comparing it with a traditional mutation testing method on six programs. The result of the experiment suggests that using the method of this paper, the high mutation score can be maintained while reducing the number of mutants

A Study of Equivalent and Stubborn Mutation Operators using Human Analysis of Equivalence

Though mutation testing has been widely studied for more than thirty years, the prevalence and properties of equivalent mutants remain largely unknown. We report on the causes and prevalence of equivalent mutants and their relationship to stubborn mutants (those that remain undetected by a high quality test suite, yet are non-equivalent). Our results, based on manual analysis of 1,230 mutants from 18 programs, reveal a highly uneven distribution of equivalence and stubbornness. For example, the ABS class and half UOI class generate many equivalent and almost no stubborn mutants, while the LCR class generates many stubborn and few equivalent mutants. We conclude that previous test effectiveness studies based on fault seeding could be skewed, while developers of mutation testing tools should prioritise those operators that we found generate disproportionately many stubborn (and few equivalent) mutants

Mutant reduction based on dominance relation for weak mutation testing

Context: As a fault-based testing technique, mutation testing is effective at evaluating the quality of existing test suites. However, a large number of mutants result in the high computational cost in mutation testing. As a result, mutant reduction is of great importance to improve the efficiency of mutation testing. Objective: We aim to reduce mutants for weak mutation testing based on the dominance relation between mutant branches. Method: In our method, a new program is formed by inserting mutant branches into the original program. By analyzing the dominance relation between mutant branches in the new program, the non-dominated one is obtained, and the mutant corresponding to the non-dominated mutant branch is the mutant after reduction. Results: The proposed method is applied to test ten benchmark programs and six classes from open-source projects. The experimental results show that our method reduces over 80% mutants on average, which greatly improves the efficiency of mutation testing. Conclusion: We conclude that dominance relation between mutant branches is very important and useful in reducing mutants for mutation testing

Mutant subsumption graphs

Mutation testing researchers have long known that\ud many generated mutants are not needed. This paper develops a\ud graph model to describe redundancy among mutations. We define\ud “true” subsumption, a relation that practicing test engineers\ud would like to have, but cannot due to issues of computability.\ud We also define dynamic subsumption and static subsumption as\ud approximations of “true” subsumption. We explore the properties\ud of the approximate subsumption relations in the context of a small\ud example. We suggest possible uses for subsumption graphs.FAPESP (número processo 2012/16950-5