Exploring regression testing and software product line testing - research and state of practice

In large software organizations with a product line development approach a selective testing of product variants is necessary in order to keep pace with the decreased development time for new products, enabled by the systematic reuse. The close relationship between products in product line indicates an option to reduce the testing effort due to redundancy. In many cases test selection is performed manually, based on test leaders’ expertise. This makes the cost and quality of the testing highly dependent on the skills and experience of the test leaders. There is a need in industry for systematic approaches to test selection. The goal of our research is to improve the control of the testing and reduce the amount of redundant testing in the product line context by applying regression test selection strategies. In this thesis, the state of art of regression testing and software product line testing are explored. Two extensive systematic reviews are conducted as well as an industrial survey of regression testing state of practice and an industrial evaluation of a pragmatic regression test selection strategy. Regression testing is not an isolated one-off activity, but rather an activity of varying scope and preconditions, strongly dependent on the context in which it is applied. Several techniques for regression test selection are proposed and evaluated empirically but in many cases the context is too specific for a technique to be easily applied directly by software developers. In order to improve the possibility for generalizing empirical results on regression test selection, guidelines for reporting the testing context are discussed in this thesis. Software product line testing is a relatively new research area. The understanding about challenges is well established but when looking for solutions to these challenges, we mostly find proposals, and empirical evaluations are sparse. Regression test selection strategies proposed in literature are not easily applicable in the product line context. Instead, control may be increased by increased visibility of the effects of testing and proper measurements of software quality. Focus of our future work will be on how to guide the planning and assessment of regression testing activities in large, complex reuse based systems, by visualizing the quality achieved in different parts of the system and evaluating the effects of different selection strategies when applied in various regression testing situations

A systematic review on regression test selection techniques

Regression testing is verifying that previously functioning software remains after a change. With the goal of finding a basis for further research in a joint industry-academia research project, we conducted a systematic review of empirical evaluations of regression test selection techniques. We identified 27 papers reporting 36 empirical studies, 21 experiments and 15 case studies. In total 28 techniques for regression test selection are evaluated. We present a qualitative analysis of the findings, an overview of techniques for regression test selection and related empirical evidence. No technique was found clearly superior since the results depend on many varying factors. We identified a need for empirical studies where concepts are evaluated rather than small variations in technical implementations

A case study of the class firewall regression test selection technique on a large scale distributed software system

Regression testing is expensive and may consume much of organizations’ software development budgets. Thus, it is of interest to reduce the total time devoted to test execution by using test selection techniques. Many techniques have been proposed but few have been evaluated on real-world, large scale systems. In this paper we report on an empirical evaluation of using the class firewall regression test selection technique, in combination with scenario testing, on a large scale industrial software system using the Java byte code in the analysis. The study was performed on a large complex distributed software system in one of Sweden’s largest banks. Effects of using scenario testing together with regression test selection are reported. The results are that not all test cases were selected by the class firewall selection technique. Using scenario testing, where test cases are dependent, affects the number of test cases selected, as do the location and number of changes in the system

A case study of the class firewall regression test selection technique on a large scale distributed software system

Regression testing is expensive and may consume much of organizations ’ software development budgets. Thus, it is of interest to reduce the total time devoted to test execution by using test selection techniques. Many techniques have been proposed but few have been evaluated on real-world, large scale systems. In this paper we report on an empirical evaluation of using the class firewall regression test selection technique, in combination with scenario testing, on a large scale industrial software system using the Java byte code in the analysis. The study was performed on a large complex distributed software system in one of Sweden’s largest banks. Effects of using scenario testing together with regression test selection are reported. The results are that not all test cases were selected by the class firewall selection technique. Using scenario testing, where test cases are dependent, affects the number of test cases selected, as do the location and number of changes in the system.

An Infrastructure to Support Interoperability in Reverse Engineering

An infrastructure that supports interoperability among reverse engineering tools and other software tools is described. The three major components of the infrastructure are: (1) a hierarchy of schemas for low- and middle-level program representation graphs, (2) g4re, a tool chain for reverse engineering C++ programs, and (3) a repository of reverse engineering artifacts, including the previous two components, a test suite, and tools, GXL instances, and XSLT transformations for graphs at each level of the hierarchy. The results of two case studies that investigated the space and time costs incurred by the infrastructure are provided. The results of two empirical evaluations that were performed using the api module of g4re, and were focused on computation of object-oriented metrics and three-dimensional visualization of class template diagrams, respectively, are also provided

Estimating and Improving the Performance of Prediction Models for Regression Test Selection

Researchers have proposed models to predict the percentage of the selected test cases when a Regression Test Selection (RTS) technique is used. One of the most successful and best performing RTS predictors is the Rosenblum and Weyuker (RW) coverage-based prediction model. However, previous evaluation results on RW predictor show that although it performs well on some subject programs, it deviates from actual percentage significantly on others. To understand the reason impacting RW predictor's performance, this research work presents a set of experiments on four factors that can potentially impact the RTS prediction performance. We setup two different set of experiments on several Java open-source test subjects and three RTS techniques. Our study on the effect of each factor on the RW performance reveals that large amount of code changes and significant code coverage overlaps between test cases are the two factors contributing to RW predictor's prediction error. Based on the experimental results and through regression analysis of the impacting factors, we propose a RW error estimator that can help testers and developers to gain a better understanding of RW predictor's confidence level and get insight into the applicability of the RW predictor to different organizations products and processes. To further improve RW predictor's performance, we propose an improved RW prediction model utilizing the error estimator to compensate prediction error. We also design a specific RTS improvement technique while presenting Harrold et al's improvement which also incorporates change history. Our experiments on these improved RW predictors demonstrate that they can reduce RW prediction error and improve performance