Test prioritization based on the coverage of recently modified source code: An industrial case study

Abstract

Regression tests are re-executed to ensure quality and lack of side-effects after software changes to incorporate new/improved functionalities and/or bug fixes. Prioritizing these tests for detecting faults earlier can increase productivity especially when the testing duration increases. We conduct an industrial case study in the consumer electronics domain, where regression tests take several weeks to complete. We evaluate the effectiveness of a test prioritization approach in terms of the rate of early fault detection. We analyze test cases individually but apply prioritization at a higher granularity level, where we prioritize weekly test plans rather than individual test cases. Our approach gives higher priority to those test cases that cover the recently modified parts of the source code. We use 3 Digital TV projects as subject systems. We compare the effectiveness of the original execution order of test cases with the alternative ordering as suggested by our approach. Results show that the alternative ordering is more effective in finding faults earlier for all the 3 subject systems, where the rate of early fault detection can be increased by up to 38%