Code coverage and postrelease defects: A large-scale study on open source projects

International audienceTesting is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in post-release bugs? Do files that have higher test coverage actually have fewer bug reports? The direct relationship between code coverage and actual bug reports has not yet been analysed via a comprehensive empirical study on real bugs. Past studies only involve a few software systems or artificially injected bugs (mutants).In this empirical study, we examine these questions in the context of open-source software projects based on their actual reported bugs. We analyze 100 large open-source Java projects and measure the code coverage of the test cases that come along with these projects. We collect real bugs logged in the issue tracking system after the release of the software and analyse the correlations between code coverage and these bugs. We also collect other metrics such as cyclomatic complexity and lines of code, which are used to normalize the number of bugs and coverage to correlate with other metrics as well as use these metrics in regression analysis. Our results show that coverage has an insignificant correlation with the number of bugs that are found after the release of the software at the project level, and no such correlation at the file level

Evaluating Random Mutant Selection at Class-Level in Projects with Non-Adequate Test Suites

Mutation testing is a standard technique to evaluate the quality of a test suite. Due to its computationally intensive nature, many approaches have been proposed to make this technique feasible in real case scenarios. Among these approaches, uniform random mutant selection has been demonstrated to be simple and promising. However, works on this area analyze mutant samples at project level mainly on projects with adequate test suites. In this paper, we fill this lack of empirical validation by analyzing random mutant selection at class level on projects with non-adequate test suites. First, we show that uniform random mutant selection underachieves the expected results. Then, we propose a new approach named weighted random mutant selection which generates more representative mutant samples. Finally, we show that representative mutant samples are larger for projects with high test adequacy.Comment: EASE 2016, Article 11 , 10 page

A Model to Estimate First-Order Mutation Coverage from Higher-Order Mutation Coverage

The test suite is essential for fault detection during software development. First-order mutation coverage is an accurate metric to quantify the quality of the test suite. However, it is computationally expensive. Hence, the adoption of this metric is limited. In this study, we address this issue by proposing a realistic model able to estimate first-order mutation coverage using only higher-order mutation coverage. Our study shows how the estimation evolves along with the order of mutation. We validate the model with an empirical study based on 17 open-source projects.Comment: 2016 IEEE International Conference on Software Quality, Reliability, and Security. 9 page

Contextual Sensitivity in Grounded Theory: The Role of Pilot Studies

Grounded Theory is an established methodological approach for context specific inductive theory building. The grounded nature of the methodology refers to these specific contexts from which emergent propositions are drawn. Thus, any grounded theory study requires not only theoretical sensitivity, but also a good insight on how to design the research in the human activity systems to be studied. The lack of this insight may result in inefficient theoretical sampling or even erroneous purposeful sampling. These problems would not necessarily be critical, as it could be argued that through the elliptical process that characterizes grounded theory, remedial loops would always bring the researcher to the core of the theory. However, these elliptical remedial processes can take very long periods of time and result in catastrophic delays in research projects. As a strategy, this paper discusses, contrasts and compares the use of pilot studies in four different grounded theory projects. Each pilot brought different insights about the context, resulting in changes of focus, guidance to improve data collection instruments and informing theoretical sampling. Additionally, as all four projects were undertaken by researchers with little experience of inductive approaches in general and grounded theory in particular, the pilot studies also served the purpose of training in interviewing, relating to interviewees, memoing, constant comparison and coding. This last outcome of the pilot study was actually not planned initially, but revealed itself to be a crucial success factor in the running of the projects. The paper concludes with a theoretical proposition for the concept of contextual sensitivity and for the inclusion of the pilot study in grounded theory research designs

Mutation Testing as a Safety Net for Test Code Refactoring

Refactoring is an activity that improves the internal structure of the code without altering its external behavior. When performed on the production code, the tests can be used to verify that the external behavior of the production code is preserved. However, when the refactoring is performed on test code, there is no safety net that assures that the external behavior of the test code is preserved. In this paper, we propose to adopt mutation testing as a means to verify if the behavior of the test code is preserved after refactoring. Moreover, we also show how this approach can be used to identify the part of the test code which is improperly refactored