191 research outputs found
COCO: Performance Assessment
We present an any-time performance assessment for benchmarking numerical
optimization algorithms in a black-box scenario, applied within the COCO
benchmarking platform. The performance assessment is based on runtimes measured
in number of objective function evaluations to reach one or several quality
indicator target values. We argue that runtime is the only available measure
with a generic, meaningful, and quantitative interpretation. We discuss the
choice of the target values, runlength-based targets, and the aggregation of
results by using simulated restarts, averages, and empirical distribution
functions
Black-box optimization benchmarking of IPOP-saACM-ES on the BBOB-2012 noisy testbed
In this paper, we study the performance of IPOP-saACM-ES, recently proposed
self-adaptive surrogate-assisted Covariance Matrix Adaptation Evolution
Strategy. The algorithm was tested using restarts till a total number of
function evaluations of was reached, where is the dimension of the
function search space. The experiments show that the surrogate model control
allows IPOP-saACM-ES to be as robust as the original IPOP-aCMA-ES and
outperforms the latter by a factor from 2 to 3 on 6 benchmark problems with
moderate noise. On 15 out of 30 benchmark problems in dimension 20,
IPOP-saACM-ES exceeds the records observed during BBOB-2009 and BBOB-2010.Comment: Genetic and Evolutionary Computation Conference (GECCO 2012) (2012
Analysis of Different Types of Regret in Continuous Noisy Optimization
The performance measure of an algorithm is a crucial part of its analysis.
The performance can be determined by the study on the convergence rate of the
algorithm in question. It is necessary to study some (hopefully convergent)
sequence that will measure how "good" is the approximated optimum compared to
the real optimum. The concept of Regret is widely used in the bandit literature
for assessing the performance of an algorithm. The same concept is also used in
the framework of optimization algorithms, sometimes under other names or
without a specific name. And the numerical evaluation of convergence rate of
noisy algorithms often involves approximations of regrets. We discuss here two
types of approximations of Simple Regret used in practice for the evaluation of
algorithms for noisy optimization. We use specific algorithms of different
nature and the noisy sphere function to show the following results. The
approximation of Simple Regret, termed here Approximate Simple Regret, used in
some optimization testbeds, fails to estimate the Simple Regret convergence
rate. We also discuss a recent new approximation of Simple Regret, that we term
Robust Simple Regret, and show its advantages and disadvantages.Comment: Genetic and Evolutionary Computation Conference 2016, Jul 2016,
Denver, United States. 201
Self-Adaptive Surrogate-Assisted Covariance Matrix Adaptation Evolution Strategy
This paper presents a novel mechanism to adapt surrogate-assisted
population-based algorithms. This mechanism is applied to ACM-ES, a recently
proposed surrogate-assisted variant of CMA-ES. The resulting algorithm,
saACM-ES, adjusts online the lifelength of the current surrogate model (the
number of CMA-ES generations before learning a new surrogate) and the surrogate
hyper-parameters. Both heuristics significantly improve the quality of the
surrogate model, yielding a significant speed-up of saACM-ES compared to the
ACM-ES and CMA-ES baselines. The empirical validation of saACM-ES on the
BBOB-2012 noiseless testbed demonstrates the efficiency and the scalability
w.r.t the problem dimension and the population size of the proposed approach,
that reaches new best results on some of the benchmark problems.Comment: Genetic and Evolutionary Computation Conference (GECCO 2012) (2012
Benchmarking Evolutionary Algorithms For Single Objective Real-valued Constrained Optimization - A Critical Review
Benchmarking plays an important role in the development of novel search
algorithms as well as for the assessment and comparison of contemporary
algorithmic ideas. This paper presents common principles that need to be taken
into account when considering benchmarking problems for constrained
optimization. Current benchmark environments for testing Evolutionary
Algorithms are reviewed in the light of these principles. Along with this line,
the reader is provided with an overview of the available problem domains in the
field of constrained benchmarking. Hence, the review supports algorithms
developers with information about the merits and demerits of the available
frameworks.Comment: This manuscript is a preprint version of an article published in
Swarm and Evolutionary Computation, Elsevier, 2018. Number of pages: 4
OPTION: OPTImization Algorithm Benchmarking ONtology
Many optimization algorithm benchmarking platforms allow users to share their
experimental data to promote reproducible and reusable research. However,
different platforms use different data models and formats, which drastically
complicates the identification of relevant datasets, their interpretation, and
their interoperability. Therefore, a semantically rich, ontology-based,
machine-readable data model that can be used by different platforms is highly
desirable. In this paper, we report on the development of such an ontology,
which we call OPTION (OPTImization algorithm benchmarking ONtology). Our
ontology provides the vocabulary needed for semantic annotation of the core
entities involved in the benchmarking process, such as algorithms, problems,
and evaluation measures. It also provides means for automatic data integration,
improved interoperability, and powerful querying capabilities, thereby
increasing the value of the benchmarking data. We demonstrate the utility of
OPTION, by annotating and querying a corpus of benchmark performance data from
the BBOB collection of the COCO framework and from the Yet Another Black-Box
Optimization Benchmark (YABBOB) family of the Nevergrad environment. In
addition, we integrate features of the BBOB functional performance landscape
into the OPTION knowledge base using publicly available datasets with
exploratory landscape analysis. Finally, we integrate the OPTION knowledge base
into the IOHprofiler environment and provide users with the ability to perform
meta-analysis of performance data
Modular Differential Evolution
New contributions in the field of iterative optimisation heuristics are often
made in an iterative manner. Novel algorithmic ideas are not proposed in
isolation, but usually as an extension of a preexisting algorithm. Although
these contributions are often compared to the base algorithm, it is challenging
to make fair comparisons between larger sets of algorithm variants. This
happens because even small changes in the experimental setup, parameter
settings, or implementation details can cause results to become incomparable.
Modular algorithms offer a way to overcome these challenges. By implementing
the algorithmic modifications into a common framework, many algorithm variants
can be compared, while ensuring that implementation details match in all
versions.
In this work, we propose a version of a modular framework for the popular
Differential Evolution (DE) algorithm. We show that this modular approach not
only aids in comparison, but also allows for a much more detailed exploration
of the space of possible DE variants. This is illustrated by showing that
tuning the settings of modular DE vastly outperforms a set of commonly used DE
versions which have been recreated in our framework. We then investigate these
tuned algorithms in detail, highlighting the relation between modules and
performance on particular problems
- …