1 research outputs found
Assessing Test Case Prioritization on Real Faults and Mutants
Test Case Prioritization (TCP) is an important component of regression
testing, allowing for earlier detection of faults or helping to reduce testing
time and cost. While several TCP approaches exist in the research literature, a
growing number of studies have evaluated them against synthetic software
defects, called mutants. Hence, it is currently unclear to what extent TCP
performance on mutants would be representative of the performance achieved on
real faults. To answer this fundamental question, we conduct the first
empirical study comparing the performance of TCP techniques applied to both
real-world and mutation faults. The context of our study includes eight
well-studied TCP approaches, 35k+ mutation faults, and 357 real-world faults
from five Java systems in the Defects4J dataset. Our results indicate that the
relative performance of the studied TCP techniques on mutants may not strongly
correlate with performance on real faults, depending upon attributes of the
subject programs. This suggests that, in certain contexts, the best performing
technique on a set of mutants may not be the best technique in practice when
applied to real faults. We also illustrate that these correlations vary for
mutants generated by different operators depending on whether chosen operators
reflect typical faults of a subject program. This highlights the importance,
particularly for TCP, of developing mutation operators tailored for specific
program domains.Comment: Accepted to the 34th International Conference on Software Maintenance
and Evolution (ICSME'18