A Survey of Binary Covering Arrays

Binary covering arrays of strength t are 0–1 matrices having the property that for each t columns and each of the possible 2[superscript t] sequences of t 0's and 1's, there exists a row having that sequence in that set of t columns. Covering arrays are an important tool in certain applications, for example, in software testing. In these applications, the number of columns of the matrix is dictated by the application, and it is desirable to have a covering array with a small number of rows. Here we survey some of what is known about the existence of binary covering arrays and methods of producing them, including both explicit constructions and search techniques

Fuzzy Adaptive Tuning of a Particle Swarm Optimization Algorithm for Variable-Strength Combinatorial Test Suite Generation

Combinatorial interaction testing is an important software testing technique that has seen lots of recent interest. It can reduce the number of test cases needed by considering interactions between combinations of input parameters. Empirical evidence shows that it effectively detects faults, in particular, for highly configurable software systems. In real-world software testing, the input variables may vary in how strongly they interact, variable strength combinatorial interaction testing (VS-CIT) can exploit this for higher effectiveness. The generation of variable strength test suites is a non-deterministic polynomial-time (NP) hard computational problem \cite{BestounKamalFuzzy2017}. Research has shown that stochastic population-based algorithms such as particle swarm optimization (PSO) can be efficient compared to alternatives for VS-CIT problems. Nevertheless, they require detailed control for the exploitation and exploration trade-off to avoid premature convergence (i.e. being trapped in local optima) as well as to enhance the solution diversity. Here, we present a new variant of PSO based on Mamdani fuzzy inference system \cite{Camastra2015,TSAKIRIDIS2017257,KHOSRAVANIAN2016280}, to permit adaptive selection of its global and local search operations. We detail the design of this combined algorithm and evaluate it through experiments on multiple synthetic and benchmark problems. We conclude that fuzzy adaptive selection of global and local search operations is, at least, feasible as it performs only second-best to a discrete variant of PSO, called DPSO. Concerning obtaining the best mean test suite size, the fuzzy adaptation even outperforms DPSO occasionally. We discuss the reasons behind this performance and outline relevant areas of future work.Comment: 21 page

GPU-based parallel computing methods for constructing covering arrays

As software systems becomes more complex, demand for e cient approaches to test these kind of systems with a lower cost is increased highly, too. One example of such applications can be given highly configurable software systems such as web servers (e.g. Apache) and databases (e.g. MySQL). They have many configurable options which interact each other and these option interactions lead having exponential growth of option configurations. Hence, these software systems become more prone to bugs which are caused by the interaction of options. A solution to this problem can be combinatorial interaction testing which systematically samples the configuration space and tests each of these samples, individually. Combinatorial interaction testing computes a small set of option configurations to be used as test suites, called covering arrays. A t-way covering array aims to cover all t-length option interactions of system under test with a minimum number of configurations where t is a small number in practical cases. Applications of covering arrays are especially encouraged after many researches empirically pointed out that substantial number of faults are caused by smaller value of option interaction. Nevertheless, computing covering arrays with a minimal number of configurations in a reasonable time is not easy task, especially when the configuration space is large and system has inter-option constraints that invalidate some configurations. Therefore, this study field attracts various researchers. Although most of approaches su er in scalability issue, many successful attempts have been also done to construct covering arrays. However, as the configuration scape gets larger, most of the approaches start to su er. Combinatorial problems e.g., in our case constructing covering arrays, are mainly solved by using e cient counting techniques. Based on this assumption, we conjecture that covering arrays can be computed using parallel algorithms e ciently since counting is an easy task which can be carried out with parallel programming strategies. Although di erent architectures are e ective in di erent researches, we choose to use GPU-based parallel computing techniques since GPUs have hundreds even sometimes thousands of cores however with small arithmetic logic units. Despite the fact that these cores are exceptionally constrained and limited, they serve our purpose very well since all we need to do is basic counting, repeatedly. We apply this idea in order to decrease the computation time on a meta-heuristic, search method simulated annealing, which is well studied in construction of covering arrays and, in general, gives the smallest size results in previous studies. Moreover, we present a technique to generate multiple neighbour states in each step of simulated annealing in parallel. Finally, we propose a novel hybrid approach using SAT solver with parallel computing techniques to decrease the negative e ect of pure random search and decrease the covering array size further. Our results prove our assumption that parallel computing is an e ective and e cient way to compute combinatorial objects

New bounds for ternary covering arrays using a parallel simulated annealing

A covering array (CA) is a combinatorial structure specified as a matrix of N rows and k columns over an alphabet on v symbols such that for each set of t columns every t-tuple of symbols is covered at least once. Given the values of t, k, and v, the optimal covering array construction problem (CAC) consists in constructing a CA (N; t, k, v) with the minimum possible value of N. There are several reported methods to attend the CAC problem, among them are direct methods, recursive methods, greedy methods, and metaheuristics methods. In this paper, There are three parallel approaches for simulated annealing: the independent, semi-independent, and cooperative searches are applied to the CAC problem. The empirical evidence supported by statistical analysis indicates that cooperative approach offers the best execution times and the same bounds as the independent and semi-independent approaches. Extensive experimentation was carried out, using 182 well-known benchmark instances of ternary covering arrays, for assessing its performance with respect to the best-known bounds reported previously. The results show that cooperative approach attains 134 new bounds and equals the solutions for other 29 instances. © 2012 Himer Avila-George et al.The authors thankfully acknowledge the computer resources and assistance provided by Spanish Supercomputing Network (TIRANT-UV). This research work was partially funded by the following projects: CONACyT 58554; Calculo de Covering Arrays; 51623-Fondo Mixto CONACyT; Gobierno del Estado de Tamaulipas.Avila-George, H.; Torres-Jimenez, J.; Hernández García, V. (2012). New bounds for ternary covering arrays using a parallel simulated annealing. Mathematical Problems in Engineering. 2012:1-19. doi:10.1155/2012/897027S119201

An orchestrated survey of available algorithms and tools for Combinatorial Testing

For functional testing based on the input domain of a functionality, parameters and their values are identified and a test suite is generated using a criterion exercising combinations of those parameters and values. Since software systems are large, resulting in large numbers of parameters and values, a technique based on combinatorics called Combinatorial Testing (CT) is used to automate the process of creating those combinations. CT is typically performed with the help of combinatorial objects called Covering Arrays. The goal of the present work is to determine available algorithms/tools for generating a combinatorial test suite. We tried to be as complete as possible by using a precise protocol for selecting papers describing those algorithms/tools. The 75 algorithms/tools we identified are then categorized on the basis of different comparison criteria, including: the test suite generation technique, the support for selection (combination) criteria, mixed covering array, the strength of coverage, and the support for constraints between parameters. Results can be of interest to researchers or software companies who are looking for a CT algorithm/tool suitable for their needs