18,455 research outputs found
Reinforcement Learning for Test Case Prioritization
Continuous Integration (CI) significantly reduces integration problems,
speeds up development time, and shortens release time. However, it also
introduces new challenges for quality assurance activities, including
regression testing, which is the focus of this work. Though various approaches
for test case prioritization have shown to be very promising in the context of
regression testing, specific techniques must be designed to deal with the
dynamic nature and timing constraints of CI.
Recently, Reinforcement Learning (RL) has shown great potential in various
challenging scenarios that require continuous adaptation, such as game playing,
real-time ads bidding, and recommender systems. Inspired by this line of work
and building on initial efforts in supporting test case prioritization with RL
techniques, we perform here a comprehensive investigation of RL-based test case
prioritization in a CI context. To this end, taking test case prioritization as
a ranking problem, we model the sequential interactions between the CI
environment and a test case prioritization agent as an RL problem, using three
alternative ranking models. We then rely on carefully selected and tailored
state-of-the-art RL techniques to automatically and continuously learn a test
case prioritization strategy, whose objective is to be as close as possible to
the optimal one. Our extensive experimental analysis shows that the best RL
solutions provide a significant accuracy improvement over previous RL-based
work, with prioritization strategies getting close to being optimal, thus
paving the way for using RL to prioritize test cases in a CI context
Hybrid and dynamic static criteria models for test case prioritization of web application regression testing
In software testing domain, different techniques and approaches are used to support the process of regression testing in an effective way. The main approaches include test case minimization, test case selection, and test case prioritization. Test case prioritization techniques improve the performance of regression testing by arranging test cases in such a way that maximize fault detection could be achieved in a shorter time. However, the problems for web testing are the timing for executing test cases and the number of fault detected. The aim of this study is to increase the effectiveness of test case prioritization by proposing an approach that could detect faults earlier at a shorter execution time. This research proposed an approach comprising two models: Hybrid Static Criteria Model (HSCM) and Dynamic Weighting Static Criteria Model (DWSCM). Each model applied three criteria: most common HTTP requests in pages, length of HTTP request chains, and dependency of HTTP requests. These criteria are used to prioritize test cases for web application regression testing. The proposed HSCM utilized clustering technique to group test cases. A hybridized technique was proposed to prioritize test cases by relying on assigned test case priorities from the combination of aforementioned criteria. A dynamic weighting scheme of criteria for prioritizing test cases was used to increase fault detection rate. The findings revealed that, the models comprising enhanced of Average Percentage Fault Detection (APFD), yielded the highest APFD of 98% in DWSCM and 87% in HSCM, which have led to improve effectiveness prioritization models. The findings confirmed the ability of the proposed techniques in improving web application regression testing
Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
Testing in Continuous Integration (CI) involves test case prioritization,
selection, and execution at each cycle. Selecting the most promising test cases
to detect bugs is hard if there are uncertainties on the impact of committed
code changes or, if traceability links between code and tests are not
available. This paper introduces Retecs, a new method for automatically
learning test case selection and prioritization in CI with the goal to minimize
the round-trip time between code commits and developer feedback on failed test
cases. The Retecs method uses reinforcement learning to select and prioritize
test cases according to their duration, previous last execution and failure
history. In a constantly changing environment, where new test cases are created
and obsolete test cases are deleted, the Retecs method learns to prioritize
error-prone test cases higher under guidance of a reward function and by
observing previous CI cycles. By applying Retecs on data extracted from three
industrial case studies, we show for the first time that reinforcement learning
enables fruitful automatic adaptive test case selection and prioritization in
CI and regression testing.Comment: Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017).
Reinforcement Learning for Automatic Test Case Prioritization and Selection
in Continuous Integration. In Proceedings of 26th International Symposium on
Software Testing and Analysis (ISSTA'17) (pp. 12--22). AC
Time-Space Efficient Regression Testing for Configurable Systems
Configurable systems are those that can be adapted from a set of options.
They are prevalent and testing them is important and challenging. Existing
approaches for testing configurable systems are either unsound (i.e., they can
miss fault-revealing configurations) or do not scale. This paper proposes
EvoSPLat, a regression testing technique for configurable systems. EvoSPLat
builds on our previously-developed technique, SPLat, which explores all
dynamically reachable configurations from a test. EvoSPLat is tuned for two
scenarios of use in regression testing: Regression Configuration Selection
(RCS) and Regression Test Selection (RTS). EvoSPLat for RCS prunes
configurations (not tests) that are not impacted by changes whereas EvoSPLat
for RTS prunes tests (not configurations) which are not impacted by changes.
Handling both scenarios in the context of evolution is important. Experimental
results show that EvoSPLat is promising. We observed a substantial reduction in
time (22%) and in the number of configurations (45%) for configurable Java
programs. In a case study on a large real-world configurable system (GCC),
EvoSPLat reduced 35% of the running time. Comparing EvoSPLat with sampling
techniques, 2-wise was the most efficient technique, but it missed two bugs
whereas EvoSPLat detected all bugs four times faster than 6-wise, on average.Comment: 14 page
Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World
This report documents the program and the outcomes of GI-Dagstuhl Seminar
16394 "Software Performance Engineering in the DevOps World".
The seminar addressed the problem of performance-aware DevOps. Both, DevOps
and performance engineering have been growing trends over the past one to two
years, in no small part due to the rise in importance of identifying
performance anomalies in the operations (Ops) of cloud and big data systems and
feeding these back to the development (Dev). However, so far, the research
community has treated software engineering, performance engineering, and cloud
computing mostly as individual research areas. We aimed to identify
cross-community collaboration, and to set the path for long-lasting
collaborations towards performance-aware DevOps.
The main goal of the seminar was to bring together young researchers (PhD
students in a later stage of their PhD, as well as PostDocs or Junior
Professors) in the areas of (i) software engineering, (ii) performance
engineering, and (iii) cloud computing and big data to present their current
research projects, to exchange experience and expertise, to discuss research
challenges, and to develop ideas for future collaborations
- …