7 research outputs found
Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection
Combinatorial interaction testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. Simulated Annealing (SA) and Greedy Algorithms have been widely used to find CIT test suites. From the literature, there is a widely-held belief that SA is slower, but produces more effective tests suites than Greedy and that SA cannot scale to higher strength coverage. We evaluated both algorithms on seven real-world subjects for the well-studied two-way up to the rarely-studied six-way interaction strengths. Our findings present evidence to challenge this current orthodoxy: real-world constraints allow SA to achieve higher strengths. Furthermore, there was no evidence that Greedy was less effective (in terms of time to fault revelation) compared to SA; the results for the greedy algorithm are actually slightly superior. However, the results are critically dependent on the approach adopted to constraint handling. Moreover, we have also evaluated a genetic algorithm for constrained CIT test suite generation. This is the first time strengths higher than 3 and constraint handling have been used to evaluate GA. Our results show that GA is competitive only for pairwise testing for subjects with a small number of constraints
A Survey of Constrained Combinatorial Testing
Combinatorial Testing (CT) is a potentially powerful testing technique,
whereas its failure revealing ability might be dramatically reduced if it fails
to handle constraints in an adequate and efficient manner. To ensure the wider
applicability of CT in the presence of constrained problem domains, large and
diverse efforts have been invested towards the techniques and applications of
constrained combinatorial testing. In this paper, we provide a comprehensive
survey of representations, influences, and techniques that pertain to
constraints in CT, covering 129 papers published between 1987 and 2018. This
survey not only categorises the various constraint handling techniques, but
also reviews comparatively less well-studied, yet potentially important,
constraint identification and maintenance techniques. Since real-world programs
are usually constrained, this survey can be of interest to researchers and
practitioners who are looking to use and study constrained combinatorial
testing techniques