Search algorithms for regression test case prioritization

Regression testing is an expensive, but important, process. Unfortunately, there may be insufficient resources to allow for the re-execution of all test cases during regression testing. In this situation, test case prioritisation techniques aim to improve the effectiveness of regression testing, by ordering the test cases so that the most beneficial are executed first. Previous work on regression test case prioritisation has focused on Greedy Algorithms. However, it is known that these algorithms may produce sub-optimal results, because they may construct results that denote only local minima within the search space. By contrast, meta-heuristic and evolutionary search algorithms aim to avoid such problems. This paper presents results from an empirical study of the application of several greedy, meta-heuristic and evolutionary search algorithms to six programs, ranging from 374 to 11,148 lines of code for 3 choices of fitness metric. The paper addresses the problems of choice of fitness metric, characterisation of landscape modality and determination of the most suitable search technique to apply. The empirical results replicate previous results concerning Greedy Algorithms. They shed light on the nature of the regression testing search space, indicating that it is multi-modal. The results also show that Genetic Algorithms perform well, although Greedy approaches are surprisingly effective, given the multi-modal nature of the landscape

Dispersity-Based Test Case Prioritization

With real-world projects, existing test case prioritization (TCP) techniques have limitations when applied to them, because these techniques require certain information to be made available before they can be applied. For example, the family of input-based TCP techniques are based on test case values or test script strings; other techniques use test coverage, test history, program structure, or requirements information. Existing techniques also cannot guarantee to always be more effective than random prioritization (RP) that does not have any precondition. As a result, RP remains the most applicable and most fundamental TCP technique. In this thesis, we propose a new TCP technique, and mainly aim at studying the Effectiveness, Actual execution time for failure detection, Efficiency and Applicability of the new approach

Adaptive Test-Case Prioritization Guided by Output Inspection

Test-case prioritization is to schedule the execution order of test cases so as to maximize some objective (e.g., revealing faults early). The existing test-case prioritization approaches separate the process of test-case prioritization and the process of test-case execution by presenting the execution order of all test cases before programmers start running test cases. As the execution information of the modified program is not available for the existing test-case prioritization approaches, these approaches mainly rely on only the execution information of the previous program before modification. To address this problem, we present an adaptive test-case prioritization approach, which determines the execution order of test cases simultaneously during the execution of test cases. In particular, the adaptive approach selects test cases based on their fault-detection capability, which is calculated based on the output of selected test cases. As soon as a test case is selected and runs, the fault-detection capability of each unselected test case is modified according to the output of the latest selected test case. To evaluate the effectiveness of the proposed adaptive approach, we conducted an experimental study on eight C programs and four Java programs. The experimental results show that the adaptive approach is usually significantly better than the total test-case prioritization approach and competitive to the additional test-case prioritization approach. Moreover, the adaptive approach is better than the additional approach on some subjects (e.g, replace and schedule).http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000331216500026&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=8e1609b174ce4e31116a60747a720701Computer Science, Software EngineeringComputer Science, Theory & MethodsEICPCI-S(ISTP)

Reinforcement Learning for Test Case Prioritization

Continuous Integration (CI) significantly reduces integration problems, speeds up development time, and shortens release time. However, it also introduces new challenges for quality assurance activities, including regression testing, which is the focus of this work. Though various approaches for test case prioritization have shown to be very promising in the context of regression testing, specific techniques must be designed to deal with the dynamic nature and timing constraints of CI. Recently, Reinforcement Learning (RL) has shown great potential in various challenging scenarios that require continuous adaptation, such as game playing, real-time ads bidding, and recommender systems. Inspired by this line of work and building on initial efforts in supporting test case prioritization with RL techniques, we perform here a comprehensive investigation of RL-based test case prioritization in a CI context. To this end, taking test case prioritization as a ranking problem, we model the sequential interactions between the CI environment and a test case prioritization agent as an RL problem, using three alternative ranking models. We then rely on carefully selected and tailored state-of-the-art RL techniques to automatically and continuously learn a test case prioritization strategy, whose objective is to be as close as possible to the optimal one. Our extensive experimental analysis shows that the best RL solutions provide a significant accuracy improvement over previous RL-based work, with prioritization strategies getting close to being optimal, thus paving the way for using RL to prioritize test cases in a CI context

Test Case Prioritization for Software Product Line: A Systematic Mapping Study

Combinatorial explosion remains a common issue in testing. Due to the vast number of product variants, the number of test cases required for comprehensive coverage has significantly increased. One of the techniques to efficiently tackle this problem is prioritizing the test suites using a regression testing method. However, there is a lack of comprehensive reviews focusing on test case prioritization in SPLs. To address this research gap, this paper proposed a systematic mapping study to observe the extent of test case prioritization usage in Software Product Line Testing. The study aims to classify various aspects of SPL-TCP (Software Product Line – Test Case Prioritization), including methods, criteria, measurements, constraints, empirical studies, and domains. Over the last ten years, a thorough investigation uncovered twenty-four primary studies, consisting of 12 journal articles and 12 conference papers, all related to Test Case Prioritization for SPLs. This systematic mapping study presents a comprehensive classification of the different approaches to test case prioritization for Software Product Lines. This classification can be valuable in identifying the most suitable strategies to address specific challenges and serves as a guide for future research works. In conclusion, this mapping study systematically classifies different approaches to test case prioritization in Software Product Lines. The results of this study can serve as a valuable resource for addressing challenges in SPL testing and provide insights for future research

Test Case Prioritization Based on Artificial Immune Algorithm

Regression testing is an essential and critical part of smart terminal program development. The test case suite is usually preprocessed by test case prioritization technology to improve the efficiency of regression testing. To address the problems of traditional genetic algorithm in solving the test case prioritization problem, this paper proposed a test case prioritization algorithm for intelligent terminal based on artificial immune algorithm. Firstly, different sequences of test case sets were used as the encoding of antibodies to initialize the antibody population; secondly, the Hemming distance was introduced as the concentration index of antibodies to calculate the incentive degree; finally, the antibodies were immunized to find the optimal test case set sequence. The experimental results showed that the algorithm based on the artificial immune algorithm was more capable of global search and less likely to fall into local optimum than the genetic algorithm, which indicated that the artificial immune algorithm was more stable and could better solve the test case prioritization problem