77,593 research outputs found
Distribution metric driven adaptive random testing
Adaptive Random Testing (ART) was developed to enhance the failure detection capability of Random Testing. The basic principle of ART is to enforce random test cases evenly spread inside the input domain. Various distribution metrics have been used to measure different aspects of the evenness of test case distribution. As expected, it has been observed that the failure detection capability of an ART algorithm is related to how evenly test cases are distributed. Motivated by such an observation, we propose a new family of ART algorithms, namely distribution metric driven ART, in which, distribution metrics are key drivers for evenly spreading test cases inside ART. Out study uncovers several interesting results and shows that the new algorithms can spread test cases more evenly, and also have better failure detection capabilities
Optimal Calibration for Multiple Testing against Local Inhomogeneity in Higher Dimension
Based on two independent samples X_1,...,X_m and X_{m+1},...,X_n drawn from
multivariate distributions with unknown Lebesgue densities p and q
respectively, we propose an exact multiple test in order to identify
simultaneously regions of significant deviations between p and q. The
construction is built from randomized nearest-neighbor statistics. It does not
require any preliminary information about the multivariate densities such as
compact support, strict positivity or smoothness and shape properties. The
properly adjusted multiple testing procedure is shown to be sharp-optimal for
typical arrangements of the observation values which appear with probability
close to one. The proof relies on a new coupling Bernstein type exponential
inequality, reflecting the non-subgaussian tail behavior of a combinatorial
process. For power investigation of the proposed method a reparametrized
minimax set-up is introduced, reducing the composite hypothesis "p=q" to a
simple one with the multivariate mixed density (m/n)p+(1-m/n)q as infinite
dimensional nuisance parameter. Within this framework, the test is shown to be
spatially and sharply asymptotically adaptive with respect to uniform loss on
isotropic H\"older classes. The exact minimax risk asymptotics are obtained in
terms of solutions of the optimal recovery
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
The optimization of algorithm (hyper-)parameters is crucial for achieving
peak performance across a wide range of domains, ranging from deep neural
networks to solvers for hard combinatorial problems. The resulting algorithm
configuration (AC) problem has attracted much attention from the machine
learning community. However, the proper evaluation of new AC procedures is
hindered by two key hurdles. First, AC benchmarks are hard to set up. Second
and even more significantly, they are computationally expensive: a single run
of an AC procedure involves many costly runs of the target algorithm whose
performance is to be optimized in a given AC benchmark scenario. One common
workaround is to optimize cheap-to-evaluate artificial benchmark functions
(e.g., Branin) instead of actual algorithms; however, these have different
properties than realistic AC problems. Here, we propose an alternative
benchmarking approach that is similarly cheap to evaluate but much closer to
the original AC problem: replacing expensive benchmarks by surrogate benchmarks
constructed from AC benchmarks. These surrogate benchmarks approximate the
response surface corresponding to true target algorithm performance using a
regression model, and the original and surrogate benchmark share the same
(hyper-)parameter space. In our experiments, we construct and evaluate
surrogate benchmarks for hyperparameter optimization as well as for AC problems
that involve performance optimization of solvers for hard combinatorial
problems, drawing training data from the runs of existing AC procedures. We
show that our surrogate benchmarks capture overall important characteristics of
the AC scenarios, such as high- and low-performing regions, from which they
were derived, while being much easier to use and orders of magnitude cheaper to
evaluate
Active Sampling-based Binary Verification of Dynamical Systems
Nonlinear, adaptive, or otherwise complex control techniques are increasingly
relied upon to ensure the safety of systems operating in uncertain
environments. However, the nonlinearity of the resulting closed-loop system
complicates verification that the system does in fact satisfy those
requirements at all possible operating conditions. While analytical proof-based
techniques and finite abstractions can be used to provably verify the
closed-loop system's response at different operating conditions, they often
produce conservative approximations due to restrictive assumptions and are
difficult to construct in many applications. In contrast, popular statistical
verification techniques relax the restrictions and instead rely upon
simulations to construct statistical or probabilistic guarantees. This work
presents a data-driven statistical verification procedure that instead
constructs statistical learning models from simulated training data to separate
the set of possible perturbations into "safe" and "unsafe" subsets. Binary
evaluations of closed-loop system requirement satisfaction at various
realizations of the uncertainties are obtained through temporal logic
robustness metrics, which are then used to construct predictive models of
requirement satisfaction over the full set of possible uncertainties. As the
accuracy of these predictive statistical models is inherently coupled to the
quality of the training data, an active learning algorithm selects additional
sample points in order to maximize the expected change in the data-driven model
and thus, indirectly, minimize the prediction error. Various case studies
demonstrate the closed-loop verification procedure and highlight improvements
in prediction error over both existing analytical and statistical verification
techniques.Comment: 23 page
Data-driven rate-optimal specification testing in regression models
We propose new data-driven smooth tests for a parametric regression function.
The smoothing parameter is selected through a new criterion that favors a large
smoothing parameter under the null hypothesis. The resulting test is adaptive
rate-optimal and consistent against Pitman local alternatives approaching the
parametric model at a rate arbitrarily close to 1/\sqrtn. Asymptotic critical
values come from the standard normal distribution and the bootstrap can be used
in small samples. A general formalization allows one to consider a large class
of linear smoothing methods, which can be tailored for detection of additive
alternatives.Comment: Published at http://dx.doi.org/10.1214/009053604000001200 in the
Annals of Statistics (http://www.imstat.org/aos/) by the Institute of
Mathematical Statistics (http://www.imstat.org
- …