Evolutionary Computation Based Test Optimization of Large-Scale Systems

Modern large-scale software systems are highly configurable. Thus, they require a large number of test cases to be implemented and revised for testing a variety of system configurations. This makes software testing of large-scale software systems very expensive and time-consuming. However, companies have limited time and resources to test software systems. They need to deliver highquality software products while facing different constraints (e.g., time), which raises the need for cost-effective testing. Driven by the needs of our industrial partner, Cisco Systems Norway for testing of video conferencing systems, this thesis applies a set of methods based on evolutionary computation for cost-effective testing. Specifically, cost-effective testing in Cisco can be formulated into four main problems: 1) test case prioritization (TCP) to cost-effectively prioritize the existing test cases, 2) test case selection (TCS) to cost-effectively select a subset of test cases from the test suite with maximum effectiveness and minimum cost, 3) test case implantation (TCI) to modify existing test cases to cost-effectively test the untested configurations, and 4) dynamic test case prioritization (DTP) to dynamically prioritize test cases based on runtime execution results of the test cases in order to further improve the results of test case prioritization. To address the above-mentioned challenges, this thesis proposes a set of methods based on evolutionary computation. It includes a: 1) Search-based prioritization approach based on incremental unique coverage and positional impact, STIPI to address TCP (Paper A); 2) Searchbased test case selection approach which can incorporate user preference for difference objectives to select test cases within a time budget to address TCS (Paper B); 3) Search-based test case implantation approach to automatically analyze and implant existing test cases, SBI to address TCI (Paper D); and 4) Test case prioritization approach, REMAP that uses rule mining and multiobjective search to dynamically prioritize test cases to address DTP (Paper E). In addition, this thesis proposes two new cluster-based genetic algorithms to address the shortcomings of the current state-of-the-art search algorithms in multi-objective test optimization (Paper C). All the proposed methods have been extensively evaluated by comparing against the state-of-theart approaches by employing different case studies (e.g., industrial, real-world, open source) and widely used evaluation metrics (e.g., average percentage of fault detected). The results showed that the proposed methods could significantly improve the performance for all the identified challenges, and thus, help in cost-effective testing