Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration

Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if traceability links between code and tests are not available. This paper introduces Retecs, a new method for automatically learning test case selection and prioritization in CI with the goal to minimize the round-trip time between code commits and developer feedback on failed test cases. The Retecs method uses reinforcement learning to select and prioritize test cases according to their duration, previous last execution and failure history. In a constantly changing environment, where new test cases are created and obsolete test cases are deleted, the Retecs method learns to prioritize error-prone test cases higher under guidance of a reward function and by observing previous CI cycles. By applying Retecs on data extracted from three industrial case studies, we show for the first time that reinforcement learning enables fruitful automatic adaptive test case selection and prioritization in CI and regression testing.Comment: Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration. In Proceedings of 26th International Symposium on Software Testing and Analysis (ISSTA'17) (pp. 12--22). AC

Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review

Regression testing is an essential activity to assure that software code changes do not adversely affect existing functionalities. With the wide adoption of Continuous Integration (CI) in software projects, which increases the frequency of running software builds, running all tests can be time-consuming and resource-intensive. To alleviate that problem, Test case Selection and Prioritization (TSP) techniques have been proposed to improve regression testing by selecting and prioritizing test cases in order to provide early feedback to developers. In recent years, researchers have relied on Machine Learning (ML) techniques to achieve effective TSP (ML-based TSP). Such techniques help combine information about test cases, from partial and imperfect sources, into accurate prediction models. This work conducts a systematic literature review focused on ML-based TSP techniques, aiming to perform an in-depth analysis of the state of the art, thus gaining insights regarding future avenues of research. To that end, we analyze 29 primary studies published from 2006 to 2020, which have been identified through a systematic and documented process. This paper addresses five research questions addressing variations in ML-based TSP techniques and feature sets for training and testing ML models, alternative metrics used for evaluating the techniques, the performance of techniques, and the reproducibility of the published studies

The effectiveness of different test case prioritization techniques in continuous integration

Abstract. Regression testing is often performed in software development to increase the quality of the software. Especially continuous integration (CI) environments face challenges related to the efficiency of the regression testing feedback loop. It is important to get feedback about the changes made as soon as possible so that work can be started on fixing the potential issues caused by the changes. Regression testing is optimized by prioritizing test cases into an order that increases the early fault detection rate. However, different test case prioritization (TCP) approaches and techniques set different requirements, such as access to source code, for the environment. As the CI environments set high time and resource constraints for the TCP, not all techniques are applicable and some care should be put into selecting a suitable TCP technique. This thesis consists of a literature review that aims to consider CI environments’s special requirements, map out the most common TCP techniques as well as evaluate their applicability to these CI environments. Based on the benefits and drawbacks noted for the approaches, it seems likely that history-based TCP approaches could be the best fit for these resource intensive CI environments.Tiivistelmä. Regressiotestausta käytetään ohjelmistokehityksessä ohjelmiston laadun parantamiseen. Regressiotestauksen palautesilmukan (engl. feedback loop) tehokkuus tuottaa haasteita varsinkin jatkuvan integraation kehitysympäristöissä. On tärkeää, että kehittäjät saavat palautetta tekemiensä muutosten mahdollisista epätoivotuista vaikutuksista, jotta tarvittavat korjaustyöt voidaan aloittaa mahdollisimman pikaisesti. Regressiotestausta voidaan optimoida laittamalla testitapaukset sellaiseen tärkeysjärjestykseen, joka kasvattaa aikaista virheiden havaitsemistahtia (engl. early fault detection rate). Eri priorisointitekniikat asettavat kuitenkin erilaisia vaatimuksia suoritusympäristölle. Esimerkiksi jotkin tekniikat tarvitsevat näkyvyyden lähdekoodiin. Testitapausten priorisointitekniikan valinta kannattaa tehdä huolella jatkuvan integraation kehitysympäristöissä, sillä kyseisillä ympäristöillä on korkeita aika- ja resurssienkäyttövaatimuksia. Tämä tutkielma koostuu kirjallisuuskatsauksesta, jonka tarkoituksena on tarkastella jatkuvan integraation kehitysympäristöjen erikoisvaatimuksia, kartoittaa yleisimmät testitapausten priorisointitekniikat sekä arvioida näiden tekniikoiden soveltuvuutta jatkuvan integraation kehitysympäristöihin. Tunnistettujen testitapausten priorisoinnin lähestymistapojen etujen ja haittojen perusteella vaikuttaisi todennäköiseltä, että historiapohjaiset lähestymistavat voisivat soveltua parhaiten jatkuvan integraation resurssi-intensiivisiin kehitysympäristöihin

Search based software engineering: Trends, techniques and applications

© ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version is available from the link below.In the past five years there has been a dramatic increase in work on Search-Based Software Engineering (SBSE), an approach to Software Engineering (SE) in which Search-Based Optimization (SBO) algorithms are used to address problems in SE. SBSE has been applied to problems throughout the SE lifecycle, from requirements and project planning to maintenance and reengineering. The approach is attractive because it offers a suite of adaptive automated and semiautomated solutions in situations typified by large complex problem spaces with multiple competing and conflicting objectives. This article provides a review and classification of literature on SBSE. The work identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.EPSRC and E

Mining Fix Patterns for FindBugs Violations

In this paper, we first collect and track a large number of fixed and unfixed violations across revisions of software. The empirical analyses reveal that there are discrepancies in the distributions of violations that are detected and those that are fixed, in terms of occurrences, spread and categories, which can provide insights into prioritizing violations. To automatically identify patterns in violations and their fixes, we propose an approach that utilizes convolutional neural networks to learn features and clustering to regroup similar instances. We then evaluate the usefulness of the identified fix patterns by applying them to unfixed violations. The results show that developers will accept and merge a majority (69/116) of fixes generated from the inferred fix patterns. It is also noteworthy that the yielded patterns are applicable to four real bugs in the Defects4J major benchmark for software testing and automated repair.Comment: Accepted for IEEE Transactions on Software Engineerin

FixMiner: Mining Relevant Fix Patterns for Automated Program Repair

Patching is a common activity in software development. It is generally performed on a source code base to address bugs or add new functionalities. In this context, given the recurrence of bugs across projects, the associated similar patches can be leveraged to extract generic fix actions. While the literature includes various approaches leveraging similarity among patches to guide program repair, these approaches often do not yield fix patterns that are tractable and reusable as actionable input to APR systems. In this paper, we propose a systematic and automated approach to mining relevant and actionable fix patterns based on an iterative clustering strategy applied to atomic changes within patches. The goal of FixMiner is thus to infer separate and reusable fix patterns that can be leveraged in other patch generation systems. Our technique, FixMiner, leverages Rich Edit Script which is a specialized tree structure of the edit scripts that captures the AST-level context of the code changes. FixMiner uses different tree representations of Rich Edit Scripts for each round of clustering to identify similar changes. These are abstract syntax trees, edit actions trees, and code context trees. We have evaluated FixMiner on thousands of software patches collected from open source projects. Preliminary results show that we are able to mine accurate patterns, efficiently exploiting change information in Rich Edit Scripts. We further integrated the mined patterns to an automated program repair prototype, PARFixMiner, with which we are able to correctly fix 26 bugs of the Defects4J benchmark. Beyond this quantitative performance, we show that the mined fix patterns are sufficiently relevant to produce patches with a high probability of correctness: 81% of PARFixMiner's generated plausible patches are correct.Comment: 31 pages, 11 figure