Mutation testing is an effective approach to evaluate and strengthen software
test suites, but its adoption is currently limited by the mutants' execution
computational cost. Several strategies have been proposed to reduce this cost
(a.k.a. mutation cost reduction strategies), however none of them has proven to
be effective for all scenarios since they often need an ad-hoc manual selection
and configuration depending on the software under test (SUT). In this paper, we
propose a novel multi-objective evolutionary hyper-heuristic approach, dubbed
Sentinel, to automate the generation of optimal cost reduction strategies for
every new SUT. We evaluate Sentinel by carrying out a thorough empirical study
involving 40 releases of 10 open-source real-world software systems and both
baseline and state-of-the-art strategies as a benchmark. We execute a total of
4,800 experiments, and evaluate their results with both quality indicators and
statistical significance tests, following the most recent best practice in the
literature. The results show that strategies generated by Sentinel outperform
the baseline strategies in 95% of the cases always with large effect sizes.
They also obtain statistically significantly better results than
state-of-the-art strategies in 88% of the cases, with large effect sizes for
95% of them. Also, our study reveals that the mutation strategies generated by
Sentinel for a given software version can be used without any loss in quality
for subsequently developed versions in 95% of the cases. These results show
that Sentinel is able to automatically generate mutation strategies that reduce
mutation testing cost without affecting its testing effectiveness (i.e.
mutation score), thus taking off from the tester's shoulders the burden of
manually selecting and configuring strategies for each SUT.Comment: in IEEE Transactions on Software Engineerin