4,278 research outputs found
Recommended from our members
Automated generation of computationally hard feature models using evolutionary algorithms
This is the post-print version of the final paper published in Expert Systems with Applications. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2014 Elsevier B.V.A feature model is a compact representation of the products of a software product line. The automated extraction of information from feature models is a thriving topic involving numerous analysis operations, techniques and tools. Performance evaluations in this domain mainly rely on the use of random feature models. However, these only provide a rough idea of the behaviour of the tools with average problems and are not sufficient to reveal their real strengths and weaknesses. In this article, we propose to model the problem of finding computationally hard feature models as an optimization problem and we solve it using a novel evolutionary algorithm for optimized feature models (ETHOM). Given a tool and an analysis operation, ETHOM generates input models of a predefined size maximizing aspects such as the execution time or the memory consumption of the tool when performing the operation over the model. This allows users and developers to know the performance of tools in pessimistic cases providing a better idea of their real power and revealing performance bugs. Experiments using ETHOM on a number of analyses and tools have successfully identified models producing much longer executions times and higher memory consumption than those obtained with random models of identical or even larger size.European Commission (FEDER), the Spanish Government and
the Andalusian Government
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
The optimization of algorithm (hyper-)parameters is crucial for achieving
peak performance across a wide range of domains, ranging from deep neural
networks to solvers for hard combinatorial problems. The resulting algorithm
configuration (AC) problem has attracted much attention from the machine
learning community. However, the proper evaluation of new AC procedures is
hindered by two key hurdles. First, AC benchmarks are hard to set up. Second
and even more significantly, they are computationally expensive: a single run
of an AC procedure involves many costly runs of the target algorithm whose
performance is to be optimized in a given AC benchmark scenario. One common
workaround is to optimize cheap-to-evaluate artificial benchmark functions
(e.g., Branin) instead of actual algorithms; however, these have different
properties than realistic AC problems. Here, we propose an alternative
benchmarking approach that is similarly cheap to evaluate but much closer to
the original AC problem: replacing expensive benchmarks by surrogate benchmarks
constructed from AC benchmarks. These surrogate benchmarks approximate the
response surface corresponding to true target algorithm performance using a
regression model, and the original and surrogate benchmark share the same
(hyper-)parameter space. In our experiments, we construct and evaluate
surrogate benchmarks for hyperparameter optimization as well as for AC problems
that involve performance optimization of solvers for hard combinatorial
problems, drawing training data from the runs of existing AC procedures. We
show that our surrogate benchmarks capture overall important characteristics of
the AC scenarios, such as high- and low-performing regions, from which they
were derived, while being much easier to use and orders of magnitude cheaper to
evaluate
The Configurable SAT Solver Challenge (CSSC)
It is well known that different solution strategies work well for different
types of instances of hard combinatorial problems. As a consequence, most
solvers for the propositional satisfiability problem (SAT) expose parameters
that allow them to be customized to a particular family of instances. In the
international SAT competition series, these parameters are ignored: solvers are
run using a single default parameter setting (supplied by the authors) for all
benchmark instances in a given track. While this competition format rewards
solvers with robust default settings, it does not reflect the situation faced
by a practitioner who only cares about performance on one particular
application and can invest some time into tuning solver parameters for this
application. The new Configurable SAT Solver Competition (CSSC) compares
solvers in this latter setting, scoring each solver by the performance it
achieved after a fully automated configuration step. This article describes the
CSSC in more detail, and reports the results obtained in its two instantiations
so far, CSSC 2013 and 2014
ASlib: A Benchmark Library for Algorithm Selection
The task of algorithm selection involves choosing an algorithm from a set of
algorithms on a per-instance basis in order to exploit the varying performance
of algorithms over a set of instances. The algorithm selection problem is
attracting increasing attention from researchers and practitioners in AI. Years
of fruitful applications in a number of domains have resulted in a large amount
of data, but the community lacks a standard format or repository for this data.
This situation makes it difficult to share and compare different approaches
effectively, as is done in other, more established fields. It also
unnecessarily hinders new researchers who want to work in this area. To address
this problem, we introduce a standardized format for representing algorithm
selection scenarios and a repository that contains a growing number of data
sets from the literature. Our format has been designed to be able to express a
wide variety of different scenarios. Demonstrating the breadth and power of our
platform, we describe a set of example experiments that build and evaluate
algorithm selection models through a common interface. The results display the
potential of algorithm selection to achieve significant performance
improvements across a broad range of problems and algorithms.Comment: Accepted to be published in Artificial Intelligence Journa
On Optimization Modulo Theories, MaxSMT and Sorting Networks
Optimization Modulo Theories (OMT) is an extension of SMT which allows for
finding models that optimize given objectives. (Partial weighted) MaxSMT --or
equivalently OMT with Pseudo-Boolean objective functions, OMT+PB-- is a
very-relevant strict subcase of OMT. We classify existing approaches for MaxSMT
or OMT+PB in two groups: MaxSAT-based approaches exploit the efficiency of
state-of-the-art MAXSAT solvers, but they are specific-purpose and not always
applicable; OMT-based approaches are general-purpose, but they suffer from
intrinsic inefficiencies on MaxSMT/OMT+PB problems.
We identify a major source of such inefficiencies, and we address it by
enhancing OMT by means of bidirectional sorting networks. We implemented this
idea on top of the OptiMathSAT OMT solver. We run an extensive empirical
evaluation on a variety of problems, comparing MaxSAT-based and OMT-based
techniques, with and without sorting networks, implemented on top of
OptiMathSAT and {\nu}Z. The results support the effectiveness of this idea, and
provide interesting insights about the different approaches.Comment: 17 pages, submitted at Tacas 1
Optimization Modulo Theories with Linear Rational Costs
In the contexts of automated reasoning (AR) and formal verification (FV),
important decision problems are effectively encoded into Satisfiability Modulo
Theories (SMT). In the last decade efficient SMT solvers have been developed
for several theories of practical interest (e.g., linear arithmetic, arrays,
bit-vectors). Surprisingly, little work has been done to extend SMT to deal
with optimization problems; in particular, we are not aware of any previous
work on SMT solvers able to produce solutions which minimize cost functions
over arithmetical variables. This is unfortunate, since some problems of
interest require this functionality.
In the work described in this paper we start filling this gap. We present and
discuss two general procedures for leveraging SMT to handle the minimization of
linear rational cost functions, combining SMT with standard minimization
techniques. We have implemented the procedures within the MathSAT SMT solver.
Due to the absence of competitors in the AR, FV and SMT domains, we have
experimentally evaluated our implementation against state-of-the-art tools for
the domain of linear generalized disjunctive programming (LGDP), which is
closest in spirit to our domain, on sets of problems which have been previously
proposed as benchmarks for the latter tools. The results show that our tool is
very competitive with, and often outperforms, these tools on these problems,
clearly demonstrating the potential of the approach.Comment: Submitted on january 2014 to ACM Transactions on Computational Logic,
currently under revision. arXiv admin note: text overlap with arXiv:1202.140
- …