Improving regression testing efficiency and reliability via test-suite transformations

As software becomes more important and ubiquitous, high quality software also becomes crucial. Developers constantly make changes to improve software, and they rely on regression testing—the process of running tests after every change—to ensure that changes do not break existing functionality. Regression testing is widely used both in industry and in open source, but it suffers from two main challenges. (1) Regression testing is costly. Developers run a large number of tests in the test suite after every change, and changes happen very frequently. The cost is both in the time developers spend waiting for the tests to finish running so that developers know whether the changes break existing functionality, and in the monetary cost of running the tests on machines. (2) Regression test suites contain flaky tests, which nondeterministically pass or fail when run on the same version of code, regardless of any changes. Flaky test failures can mislead developers into believing that their changes break existing functionality, even though those tests can fail without any changes. Developers will therefore waste time trying to debug non existent faults in their changes. This dissertation proposes three lines of work that address these challenges of regression testing through test-suite transformations that modify test suites to make them more efficient or more reliable. Specifically, two lines of work explore how to reduce the cost of regression testing and one line of work explores how to fix existing flaky tests. First, this dissertation investigates the effectiveness of test-suite reduction (TSR), a traditional test-suite transformation that removes tests deemed redundant with respect to other tests in the test suite based on heuristics. TSR outputs a smaller, reduced test suite to be run in the future. However, TSR risks removing tests that can potentially detect faults in future changes. While TSR was proposed over two decades ago, it was always evaluated using program versions with seeded faults. Such evaluations do not precisely predict the effectiveness of the reduced test suite on the future changes. This dissertation evaluates TSR in a real-world setting using real software evolution with real test failures. The results show that TSR techniques proposed in the past are not as effective as suggested by traditional TSR metrics, and those same metrics do not predict how effective a reduced test suite is in the future. Researchers need to either propose new TSR techniques that produce more effective reduced test suites or better metrics for predicting the effectiveness of reduced test suites. Second, this dissertation proposes a new transformation to improve regression testing cost when using a modern build system by optimizing the placement of tests, implemented in a technique called TestOptimizer. Modern build systems treat a software project as a group of inter-dependent modules, including test modules that contain only tests. As such, when developers make a change, the build system can use a developer-specified dependency graph among modules to determine which test modules are affected by any changed modules and to run only tests in the affected test modules. However, wasteful test executions are a problem when using build systems this way. Suboptimal placements of tests, where developers may place some tests in a module that has more dependencies than the test actually needs, lead to running more tests than necessary after a change. TestOptimizer analyzes a project and proposes moving tests to reduce the number of test executions that are triggered over time due to developer changes. Evaluation of TestOptimizer on five large proprietary projects at Microsoft shows that the suggested test movements can reduce 21.7 million test executions (17.1%) across all evaluation projects. Developers accepted and intend to implement 84.4% of the reported suggestions. Third, to make regression testing more reliable, this dissertation proposes iFixFlakies, a framework for fixing a prominent kind of flaky tests: order dependent tests. Order-dependent tests pass or fail depending on the order in which the tests are run. Intuitively, order-dependent tests fail either because they need another test to set up the state for them to pass, or because some other test pollutes the state before they are run, and the polluted state makes them fail. The key insight behind iFixFlakies is that test suites often already have tests, which we call helpers, that contain the logic for setting/resetting the state needed for order-dependent tests to pass. iFixFlakies searches a test suite for these helpers and then recommends patches for order-dependent tests using code from the helpers. Evaluation of iFixFlakies on 137 truly order-dependent tests from a public dataset shows that 81 of them have helpers, and iFixFlakies can fix all 81. Furthermore, among our GitHub pull requests for 78 of these order dependent tests (3 of 81 had been already fixed), developers accepted 38; the remaining ones are still pending, and none are rejected so far

Sharing the Load: Considering Fairness in De-energization Scheduling to Mitigate Wildfire Ignition Risk using Rolling Optimization

Wildfires are a threat to public safety and have increased in both frequency and severity due to climate change. To mitigate wildfire ignition risks, electric power system operators proactively de-energize high-risk power lines during "public safety power shut-off" (PSPS) events. Line de-energizations can cause communities to lose power, which may result in negative economic, health, and safety impacts. Furthermore, the same communities may repeatedly experience power shutoffs over the course of a wildfire season, which compounds these negative impacts. However, there may be many combinations of power lines whose de-energization will result in about the same reduction of system-wide wildfire risk, but the associated power outages affect different communities. Therefore, one may raise concerns regarding the fairness of de-energization decisions. Accordingly, this paper proposes a framework to select lines to de-energize in order to balance wildfire risk reduction, total load shedding, and fairness considerations. The goal of the framework is to prevent a small fraction of communities from disproportionally being impacted by PSPS events, and to instead more equally share the burden of power outages. For a geolocated test case in the southwestern United States, we use actual California demand data as well as real wildfire risk forecasts to simulate PSPS events during the 2021 wildfire season and compare the performance of various methods for promoting fairness. Our results demonstrate that the proposed formulation can provide significantly more fair outcomes with limited impacts on system-wide performance.Comment: 8 pages, 4 figure

Integration and test plans for complex manufacturing systems

The integration and test phases that are part of the development and manufacturing of complex manufacturing systems are costly and time consuming. As time-to-market is becoming increasingly important, it is crucial to keep these phases as short as possible, whilemaintaining system quality. This is especially true for the time-to-market driven semiconductor industry and for companies providing manufacturing systems to this industry such as ASML, a provider of lithographic systems. The Tangram research project has the goal, to shorten integration and test time by a model-based integration and test approach. The Ph.D. project described in this thesis is part of the Tangram project. To achieve integration and test time reduction, we developed three methods that each solve one of the following three integration and test problems: • Construction of an optimal test plan with respect to time, cost and/or quality. • Construction of an optimal integration plan with respect to time, cost and/or quality. • Construction of an optimal integration and test plan with respect to time, cost and/or quality. The test plan optimization method consists of two steps. The first step is the definition of a model of the test problem. This model consists of tests that can be performed with associated cost and duration, possible faults that can reside in the system with associated fault probability and impact (importance), and the relation between the tests and the possible faults, also denoted as the test coverage for each possible fault. The second step consists of calculating the optimal test plan based on this test model given an objective function and possible constraints on time, cost and/or risk, which is a parameter for the quality of the system. By constructing an AND/OR graph of the problem, where AND nodes denote tests and OR nodes denote system states represented by the ambiguous faults, all possible test sequences of this problem are obtained. An algorithm selects the best solution from this AND/OR graph. This solution is a set of test sequences, where the test sequence that is followed depends on the outcome (pass/fail) of the previous tests. The integration plan optimization method consists of the same two steps as the test plan optimization method. The integration model consists of modules with their development times, interfaces that denote which modules can be integrated with each other, and test phases with their durations. Furthermore, the model consists of the relation between test phases and modules indicating which modules should be integrated before the test phase may start. Also for this problem, an AND/OR graph is constructed. The AND nodes denote integration actions and the OR nodes denote system states represented by the modules that are integrated. An algorithm selects the optimal solution from this AND/OR graph. The optimal solution has the shortest possible integration time. The solution is a tree of integration actions and test phases indicating, for each module, the sequence of integration actions and test phases. The integration and test planning method is a combination of the two previously mentioned methods and also consists of two steps. The integration and test model is a combination of the test model and the integration model, with additional relations between modules and possible faults describing in which modules these possible faults are inserted. During the construction of the integration AND/OR graph, a test AND/OR graph is constructed for each integration AND node. This test AND/OR graph represents the test phase that is performed after that integration action. The start and stop moments of these test phases are determined by the test phase positioning strategy. We developed several test phase positioning strategies according to which test phases are started, for example periodically or when a certain risk level is reached. We applied the methods developed to industrial case studies in ASML to investigate the benefits of these methods. From a case study performed in the manufacturing of lithographic machines, we learned that the duration of a test phase may be reduced by approximately 20% when using the test plan optimization method instead of creating a test plan manually. From a case study performed in the integration phase of a new prototype system, we learned that using the integration planning method may reduce integration time by almost 10% compared to a manually created integration plan. From a case study performed in the integration and test phase of a software system, we learned that the final test phase durationmay be reduced by approximately 40% when applying a risk-based test phase positioning strategy instead of the currently used periodic test phase positioning strategy. We conclude that the methods developed can be used to construct optimal integration and test plans. These optimal integration and test plans are often more efficient than manually created plans, which reduces the time-to-market of a complex system while maintaining the same final system quality. Future research should indicate how to incorporate the methods developed in the complete integration and test process, and how to obtain the information needed to create the integration and test models

Reactive approach for automating exploration and exploitation in ant colony optimization

Ant colony optimization (ACO) algorithms can be used to solve nondeterministic polynomial hard problems. Exploration and exploitation are the main mechanisms in controlling search within the ACO. Reactive search is an alternative technique to maintain the dynamism of the mechanics. However, ACO-based reactive search technique has three (3) problems. First, the memory model to record previous search regions did not completely transfer the neighborhood structures to the next iteration which leads to arbitrary restart and premature local search. Secondly, the exploration indicator is not robust due to the difference of magnitudes in distance matrices for the current population. Thirdly, the parameter control techniques that utilize exploration indicators in their feedback process do not consider the problem of indicator robustness. A reactive ant colony optimization (RACO) algorithm has been proposed to overcome the limitations of the reactive search. RACO consists of three main components. The first component is a reactive max-min ant system algorithm for recording the neighborhood structures. The second component is a statistical machine learning mechanism named ACOustic to produce a robust exploration indicator. The third component is the ACO-based adaptive parameter selection algorithm to solve the parameterization problem which relies on quality, exploration and unified criteria in assigning rewards to promising parameters. The performance of RACO is evaluated on traveling salesman and quadratic assignment problems and compared with eight metaheuristics techniques in terms of success rate, Wilcoxon signed-rank, Chi-square and relative percentage deviation. Experimental results showed that the performance of RACO is superior than the eight (8) metaheuristics techniques which confirmed that RACO can be used as a new direction for solving optimization problems. RACO can be used in providing a dynamic exploration and exploitation mechanism, setting a parameter value which allows an efficient search, describing the amount of exploration an ACO algorithm performs and detecting stagnation situations