51 research outputs found
Analysis of Different Types of Regret in Continuous Noisy Optimization
The performance measure of an algorithm is a crucial part of its analysis.
The performance can be determined by the study on the convergence rate of the
algorithm in question. It is necessary to study some (hopefully convergent)
sequence that will measure how "good" is the approximated optimum compared to
the real optimum. The concept of Regret is widely used in the bandit literature
for assessing the performance of an algorithm. The same concept is also used in
the framework of optimization algorithms, sometimes under other names or
without a specific name. And the numerical evaluation of convergence rate of
noisy algorithms often involves approximations of regrets. We discuss here two
types of approximations of Simple Regret used in practice for the evaluation of
algorithms for noisy optimization. We use specific algorithms of different
nature and the noisy sphere function to show the following results. The
approximation of Simple Regret, termed here Approximate Simple Regret, used in
some optimization testbeds, fails to estimate the Simple Regret convergence
rate. We also discuss a recent new approximation of Simple Regret, that we term
Robust Simple Regret, and show its advantages and disadvantages.Comment: Genetic and Evolutionary Computation Conference 2016, Jul 2016,
Denver, United States. 201
COCO: The Experimental Procedure
We present a budget-free experimental setup and procedure for benchmarking
numericaloptimization algorithms in a black-box scenario. This procedure can be
applied with the COCO benchmarking platform. We describe initialization of and
input to the algorithm and touch upon therelevance of termination and restarts.Comment: ArXiv e-prints, arXiv:1603.0877
Comparing Results of 31 Algorithms from the Black-Box Optimization Benchmarking BBOB-2009
pp. 1689-1696This paper presents results of the BBOB-2009 benchmark- ing of 31 search algorithms on 24 noiseless functions in a black-box optimization scenario in continuous domain. The runtime of the algorithms, measured in number of function evaluations, is investigated and a connection between a sin- gle convergence graph and the runtime distribution is uncov- ered. Performance is investigated for different dimensions up to 40-D, for different target precision values, and in dif- ferent subgroups of functions. Searching in larger dimension and multi-modal functions appears to be more difficult. The choice of the best algorithm also depends remarkably on the available budget of function evaluations
Sequential vs. Integrated Algorithm Selection and Configuration: A Case Study for the Modular CMA-ES
When faced with a specific optimization problem, choosing which algorithm to
use is always a tough task. Not only is there a vast variety of algorithms to
select from, but these algorithms often are controlled by many hyperparameters,
which need to be tuned in order to achieve the best performance possible.
Usually, this problem is separated into two parts: algorithm selection and
algorithm configuration. With the significant advances made in Machine
Learning, however, these problems can be integrated into a combined algorithm
selection and hyperparameter optimization task, commonly known as the CASH
problem. In this work we compare sequential and integrated algorithm selection
and configuration approaches for the case of selecting and tuning the best out
of 4608 variants of the Covariance Matrix Adaptation Evolution Strategy
(CMA-ES) tested on the Black Box Optimization Benchmark (BBOB) suite. We first
show that the ranking of the modular CMA-ES variants depends to a large extent
on the quality of the hyperparameters. This implies that even a sequential
approach based on complete enumeration of the algorithm space will likely
result in sub-optimal solutions. In fact, we show that the integrated approach
manages to provide competitive results at a much smaller computational cost. We
also compare two different mixed-integer algorithm configuration techniques,
called irace and Mixed-Integer Parallel Efficient Global Optimization
(MIP-EGO). While we show that the two methods differ significantly in their
treatment of the exploration-exploitation balance, their overall performances
are very similar
Benchmarking the Pure Random Search on the Bi-objective BBOB-2016 Testbed
International audienceThe Comparing Continuous Optimizers platform COCO has become a standard for benchmarking numerical (single-objective) optimization algorithms effortlessly. In 2016, COCO has been extended towards multi-objective optimization by providing a first bi-objective test suite. To provide a baseline, we benchmark a pure random search on this bi-objective bbob-biobj test suite of the COCO platform. For each combination of function, dimension n, and instance of the test suite, candidate solutions are sampled uniformly within the sampling box
- …