Automated pairwise testing approach based on classification tree modeling and negative selection algorithm

Generating the test cases for analysis is an important activity in software testing to increase the trust level of users. The traditional way to generate test cases is called exhaustive testing. It is infeasible and time consuming because it generates too many numbers of test cases. A combinatorial testing was used to solve the exhaustive testing problem. The popular technique in combinatorial testing is called pairwise testing that involves the interaction of two parameters. Although pairwise testing can cover the exhaustive testing problems, there are several issues that should be considered. First issue is related to modeling of the system under test (SUT) as a preprocess for test case generation as it has yet to be implemented in automated proposed approaches. The second issue is different approaches generate different number of test cases for different covering arrays. These issues showed that there is no one efficient way to find the optimal solution in pairwise testing that would consider the invalid combination or constraint. Therefore, a combination of Classification Tree Method and Negative Selection Algorithm (CTM-NSA) was developed in this research. The CTM approach was revised and enhanced to be used as the automated modeling and NSA approach was developed to optimize the pairwise testing by generate the low number of test cases. The findings showed that the CTM-NSA outperformed the other modeling method in terms of easing the tester and generating a low number of test cases in the small SUT size. Furthermore, it is comparable to the efficient approaches as compared to many of the test case generation approaches in large SUT size as it has good characteristic in detecting the self and non-self-sample. This characteristic occurs during the detection stage of NSA by covering the best combination of values for all parameters and considers the invalid combinations or constraints in order to achieve a hundred percent pairwise testing coverage. In addition, validation of the approach was performed using Statistical Wilcoxon Signed-Rank Test. Based on these findings, CTM-NSA had been shown to be able perform modeling in an automated way and achieve the minimum or a low number of test cases in small SUT size

A Survey of Constrained Combinatorial Testing

Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an adequate and efficient manner. To ensure the wider applicability of CT in the presence of constrained problem domains, large and diverse efforts have been invested towards the techniques and applications of constrained combinatorial testing. In this paper, we provide a comprehensive survey of representations, influences, and techniques that pertain to constraints in CT, covering 129 papers published between 1987 and 2018. This survey not only categorises the various constraint handling techniques, but also reviews comparatively less well-studied, yet potentially important, constraint identification and maintenance techniques. Since real-world programs are usually constrained, this survey can be of interest to researchers and practitioners who are looking to use and study constrained combinatorial testing techniques

Greedy combinatorial test case generation using unsatisfiable cores

Combinatorial testing aims at covering the interactions of parameters in a system under test, while some combinations may be forbidden by given constraints (forbidden tuples). In this paper, we illustrate that such forbidden tuples correspond to unsatisfiable cores, a widely understood notion in the SAT solving community. Based on this observation, we propose a technique to detect forbidden tuples lazily during a greedy test case generation, which significantly reduces the number of required SAT solving calls. We further reduce the amount of time spent in SAT solving by essentially ignoring constraints while constructing each test case, but then "amending" it to obtain a test case that satisfies the constraints, again using unsatisfiable cores. Finally, to complement a disturbance due to ignoring constraints, we implement an efficient approximative SAT checking function in the SAT solver Lingeling. Through experiments we verify that our approach significantly improves the efficiency of constraint handling in our greedy combinatorial testing algorithm.QC 20170117</p

Proceedings of SAT Competition 2021 : Solver and Benchmark Descriptions

Non peer reviewe

SUT models and t-way tests for Flex, Grep, and Make

* TestModel: SUT (System Under Test) models for projects flex, grep, and make.<br><br>.tsl: Test specification files in SIR repository. <br>.sut: Parameters and values without constraints, made from .tsl files.<br>.calot: Input models in Calot[1] format, made from .tsl files. Single and error parameter values are omitted.<br>.ts: exhaustive test cases in SIR repository.<br>.frame: executable test cases in SIR repository.<br><br>* CombCS: t-way test cases.<br><br>1-, 2-, 3-, & 4-way test cases by PICT, ACTS, and Calot, generated using t-way testing tool Calot.<br><br>* For experiments in [2], we translated t-way test cases in CombCS to corresponding (executable) test cases in .frame format (the omitted parameter values are revived), and investigated code coverage of test cases. <br><br>[1] Akihisa Yamada, Armin Biere, Cyrille Artho, Takashi Kitamura, and Eun-Hye Choi, "Greedy Combinatorial Test Case Generation Using Unsatisfiable Cores," In Proc. of 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016), pp. 614-624, September 2016.<br><br>[2] Eun-Hye Choi, Osamu Mizuno, and Yifan Hu, "Code Coverage Analysis of Combinatorial Testing," In Proc. of 4th International Workshop on Quantitative Approaches to Software Quality（QUASoQ 2016), in conjunction with APSEC 2016, pp. 34--40 December 2016.<br><br