Comparing test sets and criteria in the presence of test hypotheses and fault domains

A number of authors have considered the problem of comparing test sets and criteria. Ideally test sets are compared using a preorder with the property that test set T1 is at least as strong as T2 if whenever T2 determines that an implementation p is faulty, T1 will also determine that p is faulty. This notion can be extended to test criteria. However, it has been noted that very few test sets and criteria are comparable under such an ordering; instead orderings are based on weaker properties such as subsumes. This paper explores an alternative approach, in which comparisons are made in the presence of a test hypothesis or fault domain. This approach allows strong statements about fault detecting ability to be made and yet for a number of test sets and criteria to be comparable. It may also drive incremental test generation

Liouville property, Wiener's test and unavoidable sets for Hunt processes

Let $(X,\mathcal W)$ be a balayage space, $1\in \mathcal W$, or - equivalently - let $\mathcal W$ be the set of excessive functions of a Hunt process on a locally compact space $X$ with countable base such that $\mathcal W$ separates points, every function in $\mathcal W$ is the supremum of its continuous minorants and there exist strictly positive continuous $u,v\in \mathcal W$ such that $u/v\to 0$ at infinity. We suppose that there is a Green function $G>0$ for $X$, a metric $\rho$ on $X$ and a decreasing function $g\colon[0,\infty)\to (0,\infty]$ having the doubling property such that $G\approx g\circ\rho$. Assuming that the constant function $1$ is harmonic and balls are relatively compact, is is shown that every positive harmonic function is constant (Liouville property) and that Wiener's test at infinity shows, if a given set $A$ in $X$ is unavoidable, that is, if the process hits $A$ with probability one, wherever it starts. An application yields that locally finite unions of pairwise disjoint balls $B(z,r_z)$, $z\in Z$, which have a certain separation property with respect to a suitable measure $\lambda$ on $X$ are unavoidable if and only if, for some/any point $x_0\in X$, the series $\sum_{z\in Z} g(\rho(x_0,z))/g(r_z)$ diverges. The results generalize and, exploiting a zero-one law for hitting probabilities, simplify recent work by S. Gardiner and M. Ghergu, A. Mimica and Z. Vondra\v cek, and the author

Test Set Diameter: Quantifying the Diversity of Sets of Test Cases

A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. An advantage is that TSDm can be applied regardless of data type and on any test-related information, not only the test inputs. A downside is the increased computational time compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.Comment: In submissio

Reducing regression test size by exclusion.

Operational software is constantly evolving. Regression testing is used to identify the unintended consequences of evolutionary changes. As most changes affect only a small proportion of the system, the challenge is to ensure that the regression test set is both safe (all relevant tests are used) and unclusive (only relevant tests are used). Previous approaches to reducing test sets struggle to find safe and inclusive tests by looking only at the changed code. We use decomposition program slicing to safely reduce the size of regression test sets by identifying those parts of a system that could not have been affected by a change; this information will then direct the selection of regression tests by eliminating tests that are not relevant to the change. The technique properly accounts for additions and deletions of code. We extend and use Rothermel and Harrold’s framework for measuring the safety of regression test sets and introduce new safety and precision measures that do not require a priori knowledge of the exact number of modification-revealing tests. We then analytically evaluate and compare our techniques for producing reduced regression test sets

Finding Multiple Solutions in Nonlinear Integer Programming with Algebraic Test-Sets

We explain how to compute all the solutions of a nonlinear integer problem using the algebraic test-sets associated to a suitable linear subproblem. These test-sets are obtained using Gröbner bases. The main advantage of this method, compared to other available alternatives, is its exactness within a quite good efficiency.Ministerio de Economía y Competitividad MTM2016-75024-PMinisterio de Economía y Competitividad MTM2016-74983-C2- 1-RJunta de Andalucía P12-FQM-269

On the Structure and Complexity of Rational Sets of Regular Languages

In a recent thread of papers, we have introduced FQL, a precise specification language for test coverage, and developed the test case generation engine FShell for ANSI C. In essence, an FQL test specification amounts to a set of regular languages, each of which has to be matched by at least one test execution. To describe such sets of regular languages, the FQL semantics uses an automata-theoretic concept known as rational sets of regular languages (RSRLs). RSRLs are automata whose alphabet consists of regular expressions. Thus, the language accepted by the automaton is a set of regular expressions. In this paper, we study RSRLs from a theoretic point of view. More specifically, we analyze RSRL closure properties under common set theoretic operations, and the complexity of membership checking, i.e., whether a regular language is an element of a RSRL. For all questions we investigate both the general case and the case of finite sets of regular languages. Although a few properties are left as open problems, the paper provides a systematic semantic foundation for the test specification language FQL