Test Set Diameter: Quantifying the Diversity of Sets of Test Cases

A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. A key advantage is that TSDm is a universal measure of diversity and so can be applied to any test set regardless of data type of the test inputs (and, moreover, to other test-related data such as execution traces). But this universality comes at the cost of greater computational effort compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it

The optimisation of stochastic grammars to enable cost-effective probabilistic structural testing

Abstract The effectiveness of statistical testing, a probabilistic structural testing strategy, depends on the characteristics of the probability distribution from which test inputs are sampled. Metaheuristic search has been shown to be a practical method of optimising the characteristics of such distributions. However, the applicability of the existing search-based algorithm is limited by the requirement that the software’s inputs must be a fixed number of ordinal values. In this paper we propose a new algorithm that relaxes this limitation and so permits the derivation of probability distributions for a much wider range of software. The representation used by the new algorithm is based on a stochastic grammar supplemented with two novel features: conditional production weights and the dynamic partitioning of ordinal ranges. We demonstrate empirically that a search algorithm using this representation can optimise probability distributions over complex input domains and thereby enable cost-effective statistical testing, and that the use of both conditional production weights and dynamic partitioning can be beneficial to the search process

Emergence of a Novel Avian Pox Disease in British Tit Species

Avian pox is a viral disease with a wide host range. In Great Britain, avian pox in birds of the Paridae family was first diagnosed in a great tit (Parus major) from south-east England in 2006. An increasing number of avian pox incidents in Paridae have been reported each year since, indicative of an emergent infection. Here, we utilise a database of opportunistic reports of garden bird mortality and morbidity to analyse spatial and temporal patterns of suspected avian pox throughout Great Britain, 2006–2010. Reports of affected Paridae (211 incidents) outnumbered reports in non-Paridae (91 incidents). The majority (90%) of Paridae incidents involved great tits. Paridae pox incidents were more likely to involve multiple individuals (77.3%) than were incidents in non-Paridae hosts (31.9%). Unlike the small wart-like lesions usually seen in non-Paridae with avian pox in Great Britain, lesions in Paridae were frequently large, often with an ulcerated surface and caseous core. Spatial analyses revealed strong clustering of suspected avian pox incidents involving Paridae hosts, but only weak, inconsistent clustering of incidents involving non-Paridae hosts. There was no spatial association between Paridae and non-Paridae incidents. We documented significant spatial spread of Paridae pox from an origin in south-east England; no spatial spread was evident for non-Paridae pox. For both host clades, there was an annual peak of reports in August/September. Sequencing of the avian poxvirus 4b core protein produced an identical viral sequence from each of 20 great tits tested from Great Britain. This sequence was identical to that from great tits from central Europe and Scandinavia. In contrast, sequence variation was evident amongst virus tested from 17 non-Paridae hosts of 5 species. Our findings show Paridae pox to be an emerging infectious disease in wild birds in Great Britain, apparently originating from viral incursion from central Europe or Scandinavia

Finding test data with specific properties via metaheuristic search

For software testing to be effective the test data should cover a large and diverse range of the possible input domain. Boltzmann samplers were recently introduced as a systematic method to randomly generate data with a range of sizes from combinatorial classes, and there are a number of automated testing frameworks that serve a similar purpose. However, size is only one of many possible properties that data generated for software testing should exhibit. For the testing of realistic software systems we also need to trade off between multiple different properties or search for specific instances of data that combine several properties. In this paper we propose a general search-based framework for finding test data with specific properties. In particular, we use a metaheuristic, differential evolution, to search for stochastic models for the data generator. Evaluation of the framework demonstrates that it is more general and flexible than existing solutions based on random sampling

Generating valid grammar-based test inputs by means of genetic programming and annotated grammars

Automated generation of system level tests for grammar based systems requires the generation of complex and highly structured inputs, which must typically satisfy some formal grammar. In our previous work, we showed that genetic programming combined with probabilities learned from corpora gives significantly better results over the baseline (random) strategy. In this work, we extend our previous work by introducing grammar annotations as an alternative to learned probabilities, to be used when finding and preparing the corpus required for learning is not affordable. Experimental results carried out on six grammar based systems of varying levels of complexity show that grammar annotations produce a higher number of valid sentences and achieve similar levels of coverage and fault detection as learned probabilities