Dynamic Mutant Subsumption Analysis using LittleDarwin

Many academic studies in the field of software testing rely on mutation testing to use as their comparison criteria. However, recent studies have shown that redundant mutants have a significant effect on the accuracy of their results. One solution to this problem is to use mutant subsumption to detect redundant mutants. Therefore, in order to facilitate research in this field, a mutation testing tool that is capable of detecting redundant mutants is needed. In this paper, we describe how we improved our tool, LittleDarwin, to fulfill this requirement

Jumble Java Byte Code to Measure the Effectiveness of Unit Tests

Jumble is a byte code level mutation testing tool for Java which inter-operates with JUnit. It has been designed to operate in an industrial setting with large projects. Heuristics have been included to speed the checking of mutations, for example, noting which test fails for each mutation and running this first in subsequent mutation checks. Significant effort has been put into ensuring that it can test code which uses custom class loading and reflection. This requires careful attention to class path handling and coexistence with foreign class-loaders. Jumble is currently used on a continuous basis within an agile programming environment with approximately 370,000 lines of Java code under source control. This checks out project code every fifteen minutes and runs an incremental set of unit tests and mutation tests for modified classes. Jumble is being made available as open source

Fault Localization in Multi-Threaded C Programs using Bounded Model Checking (extended version)

Software debugging is a very time-consuming process, which is even worse for multi-threaded programs, due to the non-deterministic behavior of thread-scheduling algorithms. However, the debugging time may be greatly reduced, if automatic methods are used for localizing faults. In this study, a new method for fault localization, in multi-threaded C programs, is proposed. It transforms a multi-threaded program into a corresponding sequential one and then uses a fault-diagnosis method suitable for this type of program, in order to localize faults. The code transformation is implemented with rules and context switch information from counterexamples, which are typically generated by bounded model checkers. Experimental results show that the proposed method is effective, in such a way that sequential fault-localization methods can be extended to multi-threaded programs.Comment: extended version of paper published at SBESC'1

Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?

Mutation testing is a means to assess the effectiveness of a test suite and its outcome is considered more meaningful than code coverage metrics. However, despite several optimizations, mutation testing requires a significant computational effort and has not been widely adopted in industry. Therefore, we study in this paper whether test effectiveness can be approximated using a more light-weight approach. We hypothesize that a test case is more likely to detect faults in methods that are close to the test case on the call stack than in methods that the test case accesses indirectly through many other methods. Based on this hypothesis, we propose the minimal stack distance between test case and method as a new test measure, which expresses how close any test case comes to a given method, and study its correlation with test effectiveness. We conducted an empirical study with 21 open-source projects, which comprise in total 1.8 million LOC, and show that a correlation exists between stack distance and test effectiveness. The correlation reaches a strength up to 0.58. We further show that a classifier using the minimal stack distance along with additional easily computable measures can predict the mutation testing result of a method with 92.9% precision and 93.4% recall. Hence, such a classifier can be taken into consideration as a light-weight alternative to mutation testing or as a preceding, less costly step to that.Comment: EASE 201

Evaluating Random Mutant Selection at Class-Level in Projects with Non-Adequate Test Suites

Mutation testing is a standard technique to evaluate the quality of a test suite. Due to its computationally intensive nature, many approaches have been proposed to make this technique feasible in real case scenarios. Among these approaches, uniform random mutant selection has been demonstrated to be simple and promising. However, works on this area analyze mutant samples at project level mainly on projects with adequate test suites. In this paper, we fill this lack of empirical validation by analyzing random mutant selection at class level on projects with non-adequate test suites. First, we show that uniform random mutant selection underachieves the expected results. Then, we propose a new approach named weighted random mutant selection which generates more representative mutant samples. Finally, we show that representative mutant samples are larger for projects with high test adequacy.Comment: EASE 2016, Article 11 , 10 page

Mutation testing on an object-oriented framework: An experience report

This is the preprint version of the article - Copyright @ 2011 ElsevierContext The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners. Objective This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective. Method In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT. Results Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field. Conclusion We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.This work has been partially supported by the European Commission (FEDER) and Spanish Government under CICYT Project SETI (TIN2009-07366) and the Andalusian Government Projects ISABEL (TIC-2533) and THEOS (TIC-5906)

Exploring a Better Search–based Implementation on Second–Order Mutant Generation

Pengujian perangkat lunak adalah bagian dari proses pengembangan perangkat lunak, dengan tujuan utama untuk mengurangi/menghilangkan kesalahan pada perangkat lunak, hal ini umumnya dilakukan dengan menjalankan kasus-uji. Salah satu teknik untuk mengukur dan meningkatkan kualitas dari kasus uji adalah pengujian mutasi, tetapi walaupun sudah terbukti keefektifannya, teknik ini masih memiliki suatu kendala besar, yaitu tidak praktis untuk digunakan karena melibatkan pembangkitan dan eksekusi dari jumlah mutan yang besar. Belakangan ini penggunaan optimisasi berbasis-pencarian pada permasalahan pengujian perangkat lunak sedang popular. Pada penelitian ini, dilakukan eksplorasi penggunaan optimasi berbasis-pencarian pada pembangkitan mutan (variasi dari program), dengan tujuan untuk menghasilkan mutan yang tidak dapat dideteksi oleh kasus-uji, karena mutan jenis ini memiliki dapat kekurangan dari kasus-uji. Metode usulan dibandingkan dengan algoritma pembangkitan second-order mutant yang umum digunakan, dan juga dibandingkan dengan pendekatan berbasis pencarian lainnya. Hasil menunjukkan bahwa metode usulan dapat membangkitkan lebih banyak mutan tidak-terdeteksi (undetected-mutant) daripada dengan metode pembangkitan mutan yang umum. Metode usulan memiliki performansi yang lebih rendah daripada metode pembangkitan berbasis-pencarian benchmark, tetapi performansinya dapat ditingkatkan dengan melakukan perubahan pada representasi solusi, dan dengan adopsi parameter optimasi yang digunakan oleh metode pembanding. Software testing is a part of a software development process with a major concern is to reduce/eliminate fault in the software, and mainly done by executing a test case. One of the techniques for measuring and improving test case quality is mutation testing, but despite it is good effectiveness, this technique has a major problem that is impractical because it involves generation and execution of huge amount of mutant. This trend also happens in software testing, with the main focus on optimizing the test case generation. In this research, we explore the used of search-based optimization to the mutant (program variant) generation, with a goal to generate mutants that can escape test case detection, because these mutants have a probability to show test case deficiency. In this research, the proposed method is compared with a general second-order mutant generation algorithm and with other search-based mutant generation. The result shows that the proposed method can generate more undetected-mutant than a general second-order mutant generation. The proposed method performs worse than the benchmark search-based mutant generation, but this performance improved by altering it is solution representation and by the adoption of an optimization parameter