On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite’s rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite’s rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general

Infrastructure Support for Controlled Experimentation with Software Testing and Regression Testing Techniques

Where the development, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support rigorous controlled experimentation with testing techniques, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of testing techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with software testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have on controlled experimentation with testing techniques

A Controlled Experiment Assessing Test Case Prioritization Techniques via Mutation Faults

Regression testing is an important part of software maintenance, but it can also be very expensive. To reduce this expense, software testers may prioritize their test cases so that those that are more important are run earlier in the regression testing process. Previous work has shown that prioritization can improve a test suite’s rate of fault detection, but the assessment of prioritization techniques has been limited to hand-seeded faults, primarily due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults. We have therefore designed and performed a controlled experiment to assess the ability of prioritization techniques to improve the rate of fault detection techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. We also compare our results to those collected earlier with respect to the relationship between hand-seeded faults and mutation faults, and the implications this has for researchers performing empirical studies of prioritization

Accounting for Context and Lifetime Factors: A New Approach for Evaluating Regression Testing Techniques

Regression testing is an expensive testing process performed on modified software to provide confidence that the software behaves correctly, and that modifications have not impaired its quality. Regression testing is widely used in industry; however, it is often performed inadequately. As a result, software quality and reliability can decrease over the software\u27s lifetime. To address this problem, researchers have spent a great deal of effort creating and studying various methodologies for improving the cost-effectiveness of regression testing. To compare and assess such methodologies, researchers relied initially on analytical approaches. More recently, however, focus has shifted to empirical studies. Empirical studies of regression testing techniques have greatly expanded our understanding of techniques and the factors that affect them, but to date, these studies have also suffered from several limitations which limit the extent to which their results may generalize to practice. (1) Most studies have considered only a few context factors (characteristics of the environment or engineering processes that may affect technique performance). (2) Prior studies have calculated costs and benefits using a snapshot view in which results are considered strictly per system version; this approach, however, ignores the fact that methodologies may exhibit different cost-benefit tradeoffs when assessed across entire system lifetimes than when assessed relative to individual versions. (3) Previous studies have largely ignored cost-benefit tradeoffs, relying on comparisons strictly in terms of simple benefit and cost factors, using cost-benefit models that are naive in their handling of important revenue and cost components, or using metrics that render comparisons across specific types of techniques impossible. Limitations such as these make it difficult or impossible to accurately compare and assess regression testing methodologies relative to practical software engineering contexts. Moreover, they can lead researchers and practitioners to inaccurate conclusions about the relative cost-effectiveness of techniques in practice, or the suitability of particular techniques to particular engineering processes. This dissertation addresses these limitations. First, we surveyed the state of the art of empirical studies of regression testing techniques and identified problems with evaluation methods and processes, and problems related to infrastructure required for empirical studies. Second, we developed infrastructure to support empirical studies of regression testing considering a wide variety of software artifacts. Third, using the infrastructure developed in the second step, we conducted several initial empirical studies on regression testing techniques. Fourth, we developed a cost-benefit model to assess the cost-effectiveness of regression testing techniques considering system lifetime and context factors. Finally we conducted an empirical study, assessing regression testing techniques using these cost-benefit models. Through our work, we provide several important advantages for practitioners and researchers. For practitioners, we provide new practical understanding of regression test techniques. For researchers, we provide a new cost-benefit model that can be used to compare and empirically evaluate regression testing techniques, and that accounts for testing context and system lifetime factors. We identify problems involving infrastructure, and provide infrastructure that can help researchers conduct various controlled experiments considering a wide variety of software artifacts. Finally, we provide better understanding of empirical methodologies that can be used by other researchers to make further progress in this area

Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite’s rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites. In particular, Java and the JUnit testing framework are being used extensively in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm

Infrastructure Support for Controlled Experimentation with Software Testing and Regression Testing Techniques

Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have

On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Abstract—Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite’s rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite’s rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques

Using sensitivity analysis to create simplified economic models for regression testing

Software engineering methodologies are subject to complex costbenefit tradeoffs. Economic models can help practitioners and researchers assess methodologies relative to these tradeoffs. Effective economic models, however, can be established only through an iterative process of refinement involving analytical and empirical methods. Sensitivity analysis provides one such method. By identifying the factors that are most important to models, sensitivity analysis can help simplify those models; it can also identify factors that must be measured with care, leading to guidelines for better test strategy definition and application. In prior work we presented the first comprehensive economic model for the regression testing process, that captures both cost and benefit factors relevant to that process while supporting evaluation of these processes across entire system lifetimes. In this work we use sensitivity analysis to examine our model analytically and assess the factors that are most important to the model. Based on the results of that analysis, we propose two new models of increasing simplicity. We assess these models empirically on data obtained by using regression testing techniques on several non-trivial software systems. Our results show that one of the simplified models assesses the relationships between techniques in the same way as the full model