16,514 research outputs found
Moving forward with combinatorial interaction testing
Combinatorial interaction testing (CIT) is an efficient and effective method of detecting failures that are caused by the interactions of various system input parameters. In this paper, we discuss CIT, point out some of the difficulties of applying it in practice, and highlight some recent advances that have improved CIT’s applicability to modern systems. We also provide a roadmap for future research and directions; one that we hope will lead to new CIT research and to higher quality testing of industrial systems
Feedback driven adaptive combinatorial testing
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects – failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them
Time-Space Efficient Regression Testing for Configurable Systems
Configurable systems are those that can be adapted from a set of options.
They are prevalent and testing them is important and challenging. Existing
approaches for testing configurable systems are either unsound (i.e., they can
miss fault-revealing configurations) or do not scale. This paper proposes
EvoSPLat, a regression testing technique for configurable systems. EvoSPLat
builds on our previously-developed technique, SPLat, which explores all
dynamically reachable configurations from a test. EvoSPLat is tuned for two
scenarios of use in regression testing: Regression Configuration Selection
(RCS) and Regression Test Selection (RTS). EvoSPLat for RCS prunes
configurations (not tests) that are not impacted by changes whereas EvoSPLat
for RTS prunes tests (not configurations) which are not impacted by changes.
Handling both scenarios in the context of evolution is important. Experimental
results show that EvoSPLat is promising. We observed a substantial reduction in
time (22%) and in the number of configurations (45%) for configurable Java
programs. In a case study on a large real-world configurable system (GCC),
EvoSPLat reduced 35% of the running time. Comparing EvoSPLat with sampling
techniques, 2-wise was the most efficient technique, but it missed two bugs
whereas EvoSPLat detected all bugs four times faster than 6-wise, on average.Comment: 14 page
Test-aware combinatorial interaction testing
Combinatorial interaction testing (CIT) approaches system- atically sample a given configuration space and select a set of configurations, in which each valid t-way option setting combination appears at least once. A battery of test cases are then executed in the selected configurations. Exist- ing CIT approaches, however, do not provide a system- atic way of handling test-specific inter-option constraints. Improper handling of such constraints, on the other hand, causes masking effects, which in turn causes testers to de- velop false confidence in their test processes, believing them have tested certain option setting combinations, when they in fact have not. In this work, to avoid the harmful conse- quences of masking effects caused by improper handling of test-specific constraints, we compute t-way test-aware cov- ering arrays. A t-way test-aware covering array is not just a set of configurations as is the case in traditional covering arrays, but a set of configurations, each of which is asso- ciated with a set of test cases. We furthermore present a set of empirical studies conducted by using two widely-used highly-configurable software systems as our subject applica- tions, demonstrating that test-specific constraints are likely to occur in practice and the proposed approach is a promis- ing and effective way of handling them
Fuzzy Adaptive Tuning of a Particle Swarm Optimization Algorithm for Variable-Strength Combinatorial Test Suite Generation
Combinatorial interaction testing is an important software testing technique
that has seen lots of recent interest. It can reduce the number of test cases
needed by considering interactions between combinations of input parameters.
Empirical evidence shows that it effectively detects faults, in particular, for
highly configurable software systems. In real-world software testing, the input
variables may vary in how strongly they interact, variable strength
combinatorial interaction testing (VS-CIT) can exploit this for higher
effectiveness. The generation of variable strength test suites is a
non-deterministic polynomial-time (NP) hard computational problem
\cite{BestounKamalFuzzy2017}. Research has shown that stochastic
population-based algorithms such as particle swarm optimization (PSO) can be
efficient compared to alternatives for VS-CIT problems. Nevertheless, they
require detailed control for the exploitation and exploration trade-off to
avoid premature convergence (i.e. being trapped in local optima) as well as to
enhance the solution diversity. Here, we present a new variant of PSO based on
Mamdani fuzzy inference system
\cite{Camastra2015,TSAKIRIDIS2017257,KHOSRAVANIAN2016280}, to permit adaptive
selection of its global and local search operations. We detail the design of
this combined algorithm and evaluate it through experiments on multiple
synthetic and benchmark problems. We conclude that fuzzy adaptive selection of
global and local search operations is, at least, feasible as it performs only
second-best to a discrete variant of PSO, called DPSO. Concerning obtaining the
best mean test suite size, the fuzzy adaptation even outperforms DPSO
occasionally. We discuss the reasons behind this performance and outline
relevant areas of future work.Comment: 21 page
- …