14,828 research outputs found
Moving forward with combinatorial interaction testing
Combinatorial interaction testing (CIT) is an efficient and effective method of detecting failures that are caused by the interactions of various system input parameters. In this paper, we discuss CIT, point out some of the difficulties of applying it in practice, and highlight some recent advances that have improved CITâs applicability to modern systems. We also provide a roadmap for future research and directions; one that we hope will lead to new CIT research and to higher quality testing of industrial systems
Is One Hyperparameter Optimizer Enough?
Hyperparameter tuning is the black art of automatically finding a good
combination of control parameters for a data miner. While widely applied in
empirical Software Engineering, there has not been much discussion on which
hyperparameter tuner is best for software analytics. To address this gap in the
literature, this paper applied a range of hyperparameter optimizers (grid
search, random search, differential evolution, and Bayesian optimization) to
defect prediction problem. Surprisingly, no hyperparameter optimizer was
observed to be `best' and, for one of the two evaluation measures studied here
(F-measure), hyperparameter optimization, in 50\% cases, was no better than
using default configurations.
We conclude that hyperparameter optimization is more nuanced than previously
believed. While such optimization can certainly lead to large improvements in
the performance of classifiers used in software analytics, it remains to be
seen which specific optimizers should be applied to a new dataset.Comment: 7 pages, 2 columns, accepted for SWAN1
ASlib: A Benchmark Library for Algorithm Selection
The task of algorithm selection involves choosing an algorithm from a set of
algorithms on a per-instance basis in order to exploit the varying performance
of algorithms over a set of instances. The algorithm selection problem is
attracting increasing attention from researchers and practitioners in AI. Years
of fruitful applications in a number of domains have resulted in a large amount
of data, but the community lacks a standard format or repository for this data.
This situation makes it difficult to share and compare different approaches
effectively, as is done in other, more established fields. It also
unnecessarily hinders new researchers who want to work in this area. To address
this problem, we introduce a standardized format for representing algorithm
selection scenarios and a repository that contains a growing number of data
sets from the literature. Our format has been designed to be able to express a
wide variety of different scenarios. Demonstrating the breadth and power of our
platform, we describe a set of example experiments that build and evaluate
algorithm selection models through a common interface. The results display the
potential of algorithm selection to achieve significant performance
improvements across a broad range of problems and algorithms.Comment: Accepted to be published in Artificial Intelligence Journa
Simple Local Computation Algorithms for the General Lovasz Local Lemma
We consider the task of designing Local Computation Algorithms (LCA) for
applications of the Lov\'{a}sz Local Lemma (LLL). LCA is a class of sublinear
algorithms proposed by Rubinfeld et al.~\cite{Ronitt} that have received a lot
of attention in recent years. The LLL is an existential, sufficient condition
for a collection of sets to have non-empty intersection (in applications,
often, each set comprises all objects having a certain property). The
ground-breaking algorithm of Moser and Tardos~\cite{MT} made the LLL fully
constructive, following earlier results by Beck~\cite{beck_lll} and
Alon~\cite{alon_lll} giving algorithms under significantly stronger LLL-like
conditions. LCAs under those stronger conditions were given in~\cite{Ronitt},
where it was asked if the Moser-Tardos algorithm can be used to design LCAs
under the standard LLL condition. The main contribution of this paper is to
answer this question affirmatively. In fact, our techniques yield LCAs for
settings beyond the standard LLL condition
Sparse experimental design : an effective an efficient way discovering better genetic algorithm structures
The focus of this paper is the demonstration that sparse experimental design is a useful strategy for developing Genetic Algorithms. It is increasingly apparent from a number of reports and papers within a variety of different problem domains that the 'best' structure for a GA may be dependent upon the application. The GA structure is defined as both the types of operators and the parameters settings used during operation. The differences observed may be linked to the nature of the problem, the type of fitness function, or the depth or breadth of the problem under investigation. This paper demonstrates that advanced experimental design may be adopted to increase the understanding of the relationships between the GA structure and the problem domain, facilitating the selection of improved structures with a minimum of effort
- âŠ