Comparative evaluation of genetic algorithm-based test case optimization

Software testing is a crucial phase in software development process although it consumes more time and cost of software development. Researchers have proposed several approaches focusing on helping software testers to reduce the execution time and cost of the testing process. Test case optimization is a multi-objective approach that has become one of the best solutions to overcome these problems. Test case optimization focusing on reducing the number of test cases in the test suite that may reduce the overall testing time, cost and effort of software testers especially in regression testing. This paper presents the comparative evaluation between test case optimization techniques that are based on Genetic Algorithm (GA). The evaluation is based on five criteria i.e. technique objectives, applied fitness function, contributions, the percentage of the reduced test cases, fault detection capability, and technique limitations. The evaluation results able identify the gaps in the existing GAbased test case optimization approaches and provide insight in determining the potential research directions in this area.Keywords: Test case optimization, regression testing, multi-objectives, genetic algorithm, software testin

Enhanced non-dominated sorting genetic algorithm for test case optimization

Due to inevitable software changes, regression testing has become a crucial phase in software development process. Many software testers and researchers agreed that regression testing process consumes more time and cost during software development. Test case optimization has become one of the best solutions to overcome problems in regression testing. Test case optimization is focusing on reducing number of test cases in the test suite that may reduce the overall testing time, cost and effort of software testers. It considers multiple objectives and provides several numbers of optimal solution based on objectives of the testing. Therefore, this research aims at developing an alternative solution of test case optimization technique using NSGA II with fitness scaling as an additional function. Fitness scaling function is applied in NSGA II to eliminate pre-mature convergence among set of solution in the evolution of offspring in NSGA II which may produce more efficient fitness value. This research focuses on regression testing optimization by implementing weight of test cases and fault detection rate per test case as its objective function for optimization purposes. The proposed technique is applied to the GUI-based testing case study. The result shows that Pareto front produced by enhanced NSGA II give more wider set of solution that contains more alternatives and provide better trade-off among solutions. The evaluation shows that enhanced NSGA II perform better compared to conventional NSGA II by increasing the percentage of the reduced test cases with 25% and yield lower fault detection loss with 1.64% which indicating that set of reduced test cases using enhanced NSGA II is able to maintain the fault detection capability in the system under test

Multiobjective approaches for the minimization of test suites in Software Product Lines.

Currently many developments are guided by customers, and therefore, most companies focus on the needs of their potential customers by creating a software product line -a portfolio of products closely related to variations in features and functions- rather than just a single product. The tools and techniques for the common development of software tend to focus individual products and development, of such multiple and interrelated products, is complex. The main objective of this project is develop an optimization strategy to dealt with the previous problem and it allows us to reduce the number of test cases to apply in a reasonable time, but maintaining the quality of the resulting software products. Finally, we compare results using several different algorithms (monoobjective and multi-objectives approaches)

A dissimilarity with dice-jaro-winkler test case prioritization approach for model- based testing in software product line

The effectiveness of testing in Model-based Testing (MBT) for Software Product Line (SPL) can be achieved by considering fault detection in test case. The lack of fault consideration caused test case in test suite to be listed randomly. Test Case Prioritization (TCP) is one of regression techniques that is adaptively capable to detect faults as early as possible by reordering test cases based on fault detection rate. However, there is a lack of studies that measured faults in MBT for SPL. This paper proposes a Test Case Prioritization (TCP) approach based on dissimilarity and string based distance called Last Minimal for Local Maximal Distance (LM-LMD) with Dice-Jaro-Winkler Dissimilarity. LM-LMD with Dice-Jaro-Winkler Dissimilarity adopts Local Maximum Distance as the prioritization algorithm and Dice-Jaro-Winkler similarity measure to evaluate distance among test cases. This work is based on the test case generated from statechart in Software Product Line (SPL) domain context. Our results are promising as LM-LMD with Dice-Jaro-Winkler Dissimilarity outperformed the original Local Maximum Distance, Global Maximum Distance and Enhanced All-yes Configuration algorithm in terms of Average Fault Detection Rate (APFD) and average prioritization time

Uncertainty-wise Test Case Generation and Minimization for Cyber-Physical Systems

Cyber-Physical Systems (CPSs) typically operate in highly indeterminateenvironmental conditions, which require the development of testing methods that must explicitly consider uncertainty in test design, test generation, and test optimization. Towards this direction, we propose a set of uncertainty-wise test case generation and test case minimizationstrategies that rely on test ready models explicitly specifying subjective uncertainty. We propose two test case generation strategies and four test case minimizationstrategies based on the Uncertainty Theory and multi-objectivesearch. These strategies include a novel methodology for designing and introducing indeterminacy sources in the environment during test execution and a novel set of uncertainty-wise test verdicts. We performed an extensive empirical study to select the bestalgorithm out of eight commonly used multi-objective search algorithms, for each of the four minimizationstrategies, with five use cases of two industrial CPS case studies. The minimizedset of test cases obtained with the best algorithm for each minimizationstrategy were executedon the two real CPSs. The results showed that our best test strategy managed to observe 51% more uncertainties due to unknown indeterminate behaviorsof the physical environmentsof the CPSs as compared to the other test strategies. Also, the same test strategy managed to observe 118% more unknown uncertainties as compared to the unique number of known uncertainties.submittedVersio

Cost-effective model-based test case generation and prioritization for software product line

In Software Product Line (SPL), testing is used to manage core assets that comprised variability and commonality in effective ways due to large sizes of products that continue to be developed. SPL testing requires a technique that is capable to manage SPL core assets. Model-based Testing (MBT) is a promising technique that offers automation and reusability in test cases generation. However, there are difficulties to ensure testing in MBT can achieve good test cases generation results based on cost (size of test suite, total execution time) and effectiveness (coverage criteria, fault detection rate) measures. This is due to lack of trade-off between cost and effectiveness in test cases generated in MBT for SPL. This study aims to increase quality of test cases based on cost and effectiveness by using generation and prioritization approaches for MBT in SPL. This study focuses on three parts to enhance quality of test cases. First, test model development based on traceability link. In order to improve test cases quality, this study focused on implementation of hybrid-based and hyper-heuristic based techniques to generate test cases. This is followed by Test Cases Prioritization (TCP) technique that is based on dissimilarity-based technique with string distance. These test cases generation and prioritization approaches are evaluated by using two benchmarks - one test object and one real object. The results are compared with other prominent approaches. The mapping approach showed 10.27% and 32.39% f-measure improvement against existing approach on e-shop object, respectively. For test cases generation using hybrid-based approach, the proposed approach outperformed existing approaches with 11.66% coverage, 17.78% average execution time, and 45.98% average size of test suite on vending machine object. The hyper-heuristic based approach NSGA-II-LHH outperformed other proposed low-level heuristic approaches with 12.00% improvement on coverage, 46.66% average execution time and 42.54% average size of test suite. Furthermore, evaluation of TCP approaches showed fault detection improvement of 21.60%, 10.40% and 12.20% and total execution time improvement of 48.00%, 22.70% and 31.80% in comparison with three existing approaches. The results revealed that proposed model transformations, test cases generation and prioritization approaches significantly improve cost and effectiveness measure in MBT for SPL

Some Seeds are Strong : Seeding Strategies for Search-based Test Case Selection

The time it takes software systems to be tested is usually long. Search-based test selection has been a widely investigated technique to optimize the testing process. In this paper, we propose a set of seeding strategies for the test case selection problem that generate the initial population of pareto-based multi-objective algorithms, with the goals of (1) helping to find an overall better set of solutions and (2) enhancing the convergence of the algorithms. The seeding strategies were integrated with four state-of-the-art multi-objective search algorithms and applied into two contexts where regression-testing is paramount: (1) Simulation-based testing of Cyber-Physical Systems and (2) Continuous Integration. For the first context, we evaluated our approach by using six fitness function combinations and six independent case studies, whereas in the second context we derived a total of six fitness function combinations and employed four case studies. Our evaluation suggests that some of the proposed seeding strategies are indeed helpful for solving the multi-objective test case selection problem. Specifically, the proposed seeding strategies provided a higher convergence of the algorithms towards optimal solutions in 96% of the studied scenarios and an overall cost-effectiveness with a standard search budget in 85% of the studied scenarios