2,759 research outputs found
Utilitarian Algorithm Configuration
We present the first nontrivial procedure for configuring heuristic
algorithms to maximize the utility provided to their end users while also
offering theoretical guarantees about performance. Existing procedures seek
configurations that minimize expected runtime. However, very recent theoretical
work argues that expected runtime minimization fails to capture algorithm
designers' preferences. Here we show that the utilitarian objective also
confers significant algorithmic benefits. Intuitively, this is because mean
runtime is dominated by extremely long runs even when they are incredibly rare;
indeed, even when an algorithm never gives rise to such long runs,
configuration procedures that provably minimize mean runtime must perform a
huge number of experiments to demonstrate this fact. In contrast, utility is
bounded and monotonically decreasing in runtime, allowing for meaningful
empirical bounds on a configuration's performance. This paper builds on this
idea to describe effective and theoretically sound configuration procedures. We
prove upper bounds on the runtime of these procedures that are similar to
theoretical lower bounds, while also demonstrating their performance
empirically
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
The optimization of algorithm (hyper-)parameters is crucial for achieving
peak performance across a wide range of domains, ranging from deep neural
networks to solvers for hard combinatorial problems. The resulting algorithm
configuration (AC) problem has attracted much attention from the machine
learning community. However, the proper evaluation of new AC procedures is
hindered by two key hurdles. First, AC benchmarks are hard to set up. Second
and even more significantly, they are computationally expensive: a single run
of an AC procedure involves many costly runs of the target algorithm whose
performance is to be optimized in a given AC benchmark scenario. One common
workaround is to optimize cheap-to-evaluate artificial benchmark functions
(e.g., Branin) instead of actual algorithms; however, these have different
properties than realistic AC problems. Here, we propose an alternative
benchmarking approach that is similarly cheap to evaluate but much closer to
the original AC problem: replacing expensive benchmarks by surrogate benchmarks
constructed from AC benchmarks. These surrogate benchmarks approximate the
response surface corresponding to true target algorithm performance using a
regression model, and the original and surrogate benchmark share the same
(hyper-)parameter space. In our experiments, we construct and evaluate
surrogate benchmarks for hyperparameter optimization as well as for AC problems
that involve performance optimization of solvers for hard combinatorial
problems, drawing training data from the runs of existing AC procedures. We
show that our surrogate benchmarks capture overall important characteristics of
the AC scenarios, such as high- and low-performing regions, from which they
were derived, while being much easier to use and orders of magnitude cheaper to
evaluate
Learning Heuristic Selection with Dynamic Algorithm Configuration
A key challenge in satisficing planning is to use multiple heuristics within
one heuristic search. An aggregation of multiple heuristic estimates, for
example by taking the maximum, has the disadvantage that bad estimates of a
single heuristic can negatively affect the whole search. Since the performance
of a heuristic varies from instance to instance, approaches such as algorithm
selection can be successfully applied. In addition, alternating between
multiple heuristics during the search makes it possible to use all heuristics
equally and improve performance. However, all these approaches ignore the
internal search dynamics of a planning system, which can help to select the
most useful heuristics for the current expansion step. We show that dynamic
algorithm configuration can be used for dynamic heuristic selection which takes
into account the internal search dynamics of a planning system. Furthermore, we
prove that this approach generalizes over existing approaches and that it can
exponentially improve the performance of the heuristic search. To learn dynamic
heuristic selection, we propose an approach based on reinforcement learning and
show empirically that domain-wise learned policies, which take the internal
search dynamics of a planning system into account, can exceed existing
approaches.Comment: Long version of the paper at the International Conference on
Automated Planning and Scheduling (ICAPS) 202
MO-ParamILS: A Multi-objective Automatic Algorithm Configuration Framework
International audienceAutomated algorithm configuration procedures play an increasingly important role in the development and application of algorithms for a wide range of computationally challenging problems. Until very recently, these configuration procedures were limited to optimising a single performance objective, such as the running time or solution quality achieved by the algorithm being configured. However, in many applications there is more than one performance objective of interest. This gives rise to the multi-objective automatic algorithm configuration problem, which involves finding a Pareto set of configurations of a given target algorithm that characterises trade-offs between multiple performance objectives. In this work, we introduce MO-ParamILS, a multi-objective extension of the state-of-the-art single-objective algorithm configuration framework ParamILS, and demonstrate that it produces good results on several challenging bi-objective algorithm configuration scenarios compared to a base-line obtained from using a state-of-the-art single-objective algorithm configurator
- …