We present a budget-free experimental setup and procedure for benchmarking
numericaloptimization algorithms in a black-box scenario. This procedure can be
applied with the COCO benchmarking platform. We describe initialization of and
input to the algorithm and touch upon therelevance of termination and restarts.Comment: ArXiv e-prints, arXiv:1603.0877

Auger, Anne

Brockhoff, Dimo

Hansen, Nikolaus

Mersmann, Olaf

Tusar, Tea

English

arXiv

ArXiv e-prints, arXiv:1603.08776We present a budget-free experimental setup and procedure for benchmarking numericaloptimization algorithms in a black-box scenario. This procedure can be applied with the COCO benchmarking platform. We describe initialization of and input to the algorithm and touch upon therelevance of termination and restarts

Tušar, Tea

HAL Descartes

COCO: The Experimental Procedure

Hal-Diderot

INRIA a CCSD electronic archive server

HAL Id: hal-01294167https://hal.inria.fr/hal-01294167v2Preprint submitted on 19 May 2016HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.COCO: The Experimental ProcedureNikolaus Hansen, Tea Tušar, Olaf Mersmann, Anne Auger, Dimo BrockhoffTo cite this version:Nikolaus Hansen, Tea Tušar, Olaf Mersmann, Anne Auger, Dimo Brockhoff. COCO: The Experimen-tal Procedure. 2016. ￿hal-01294167v2￿COCO: The Experimental ProcedureNikolaus Hansen1,2, Tea Tušar3, Olaf Mersmann4, Anne Auger1,2, Dimo Brockhoff31Inria, research centre Saclay, France2Université Paris-Saclay, LRI, France3Inria, research centre Lille, France4TU Dortmund University, Chair of Computational Statistics, GermanyAbstractWe present a budget-free experimental setup and procedure for benchmarking numeri-cal optimization algorithms in a black-box scenario. This procedure can be applied with theCOCO benchmarking platform. We describe i itialization of an input to the a gorithm andtouch upon the relevance of termination and restarts.Contents1 Introduction 21.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Conducting the Experiment 22.1 Initialization and Input to the Algorithm . . . . . . . . . . . . . . . . . . . . . . . 32 Budget, Termi ation Criteria, and Restarts 43 Parameter Setting and Tuning of Algorithms 44 Time Complexity Experiment 511Based on [HAN2009] and [HAN2010], we describe a comparatively simple experimental setupfor black-box optimization benchmarking. We recommend to use this procedure within theplatform [HAN2016co].1Our central measure of performance, to which the experimental procedure is adapted, is thenumber of calls to the objective function to reach a certain solution quality (function value orf -value or indicator value), also denoted as runtime.1.1function We talk about an objective function f as a parametrized mapping Rn → Rm with scal-able input space, that is, n is not (yet) determined, and usually m ∈ {1, 2}. Functions areparametrized such that different instances of the “same” function are available, e.g. trans-lated or shifted versions.problem We talk about a problem, coco_problem_t , as a specific function instance on whichthe optimization algorithm is run. Specifically, a problem can be described as the triple(dimension, function, instance). A problem can be evaluated and returns anf -value or -vector. In the context of performance assessment, a target f - or indicator-valueis attached to each problem. That is, a target value is added to the above triple to define asingle problem in this case.runtime We define runtime, or run-length as the number of evaluations conducted ona given problem, also referred to as number of function evaluations. Our central performancemeasure is the runtime until a given target value is hit [HAN2016perf].suite A test- or benchmark-suite is a collection of problems, typically between twenty and a hun-dred, where the number of objectives m is fixed.2The optimization algorithm to be benchmarked is run on each problem of the given test suite once.On each problem, the very same algorithm with the same parameter setting, the same initialzationprocedure, the same budget, the same termination and/or restart criteria etc. is used. There is noprescribed minimal or maximal allowed budget, the benchmarking setup is budget-free. The longerthe experiment, the more data are available to assess the performance accurately. See also SectionBudget, Termination Criteria, and Restarts .1 The platform provides several (single and bi-objective) test suites with a collection of black-box optimiza-tion problems of different dimensions to be minimized. automatically collects the relevant data to display theperformance results after a post-processing is applied.22.1An algorithm can use the following input information from each problem. At any time:Input and output dimensions as a defining interface to the problem, specifically:• The search space (input) dimension via coco_problem_get_dimension ,• The number of objectives via coco_problem_get_number_of_objectives ,which is the “output” dimension of coco_evaluate_function . All functions ofa single benchmark suite have the same number of objectives, currently either one ortwo.• The number of constraints via coco_problem_get_number_of_constraints ,which is the “output” dimension of coco_evaluate_constraint . All problemsof a single benchmark suite have either no constraints, or one or more constraints.Search domain of interest defined from coco_problem_get_largest_values_of_interestand coco_problem_get_smallest_values_of_interest . The optimum (oreach extremal solution of the Pareto set) lies within the search domain of interest. If theoptimizer operates on a bounded domain only, the domain of interest can be interpreted aslower and upper bounds.Feasible (initial) solution provided by coco_problem_get_initial_solution .The initial state of the optimization algorithm and its parameters shall only be based on these inputvalues. The initial algorithm setting is considered as part of the algorithm and must thereforefollow the same procedure for all problems of the suite. The problem identifier or the positioningof the problem in the suite or any (other) known characteristics of the problem are not allowed asinput to the algorithm, see also Section Parameter Setting and Tuning of Algorithms.During an optimization run, the following (new) information is available to the algorithm:1. The result, i.e., the f -value(s) from evaluating the problem at a given search point viacoco_evaluate_function .2. The result from evaluating the constraints of the problem at a given search point viacoco_evaluate_constraint .3. The result of coco_problem_final_target_hit , which can be used to terminate arun conclusively without changing the performance assessment in any way. Currently, if thenumber of objectives m > 1, this function returns always zero.The number of evaluations of the problem and/or constraints are the search costs, also referred toas runtime, and used for the performance assessment of the algorithm.22 coco_problem_get_evaluations(const coco_problem_t * problem) is a convenience func-tion that returns the number of evaluations done on problem. Because this information is available to the optimizationalgorithm anyway, the convenience function might be used additionally.32.2Algorithms and/or setups with any budget of function evaluations are eligible, the benchmarkingsetup is budget-free. We consider termination criteria to be part of the benchmarked algorithm. Thechoice of termination is a relevant part of the algorithm. On the one hand, allowing a larger numberof function evaluations increases the chance to find solutions with better quality. On the other hand,a timely termination of stagnating runs can improve the performance, as these evaluations can beused more effectively.To exploit a large(r) number of function evaluations effectively, we encourage to use independentrestarts3, in particular for algorithms which terminate naturally within a comparatively small bud-get. Independent restarts are a natural way to approach difficult optimization problems and do notchange the central performance measure used in (hence it is budget-free), however, inde-pendent restarts improve the reliability, comparability4, precision, and “visibility” of the measuredresults.Moreover, any multistart procedure (which relies on an interim termination of the algorithm)is encouraged. Multistarts may not be independent as they can feature a parameter sweep (e.g.,increasing population size [HAR1999] ), can be based on the outcome of the previousstarts, and/or feature a systematic change of the initial conditions for the algorithm.After a multistart procedure has been established, a recommended procedure is to use a budgetproportional to the dimension, k × n, and run repeated experiments with increase k, e.g. like3, 10, 30, 100, 300, . . ., which is a good compromise between availability of the latest results andcomputational overhead.An algorithm can be conclusively terminated if coco_problem_final_target_hit returns1.5 This saves CPU cycles without affecting the performance assessment, because there is no targetleft to hit.3Any tuning of algorithm parameters to the test suite should be described and the approximateoverall number of tested parameter settings or algorithm variants and the approximate overallinvested budget should be given.The only recommended tuning procedure is the verification that termination conditions of thealgorithm are suited to the given testbed and, in case, tuning of termination parameters.6 Too early3 The platform provides example code implementing independent restarts.4 Algorithms are only comparable up to the smallest budget given to any of them.5 For the bbob-biobj suite this is however currently never the case.6 For example in the single objective case, care should be taken to apply termination conditions that allow to hit thefinal target on the most basic functions, like the sphere function f1, that is on the problems 0, 360, 720, 1080, 1440,and 1800 of the bbob suite.In our experience, numerical optimization software frequently terminates too early by default, while evolutionarycomputation software often terminates too late by default.4or too late termination can be identified and adjusted comparatively easy. This is also a usefulprerequisite for allowing restarts to become more effective.On all functions the very same parameter setting must be used (which might well depend on thedimensionality, see Section Initialization and Input to the Algorithm). That means, the a priori useof function-dependent parameter settings is prohibited (since 2012). The function ID or any func-tion characteristics (like separability, multi-modality, ...) cannot be considered as input parameterto the algorithm.On the other hand, benchmarking different parameter settings as “different algorithms” on theentire test suite is encouraged.4In order to get a rough measurement of the time complexity of the algorithm, the wall-clock orCPU time should be measured when running the algorithm on the benchmark suite. The chosensetup should reflect a “realistic average scenario”.7 The time divided by the number of functionevaluations shall be presented separately for each dimension. The chosen setup, coding language,compiler and computational architecture for conducting these experiments should be given.AcknowledgmentsThe authors would like to thank Raymond Ros, Steffen Finck, Marc Schoenauer, Petr Posik andDejan Tušar for their many invaluable contributions to this work.This work was support by the grant ANR-12-MONU-0009 (NumBBO) of the French NationalResearch Agency.References[AUG2005] A. Auger and N. Hansen. A restart CMA evolution strategy with increasing pop-ulation size. In Proceedings of the IEEE Congress on Evolutionary Computation(CEC 2005), pages 1769–1776. IEEE Press, 2005.[HAN2016perf] N. Hansen, A. Auger, D. Brockhoff, D. Tušar, T. Tušar.Assessment. ArXiv e-prints,, 2016.7 The example experiment code provides the timing output measured over all problems of a single dimension bydefault. It also can be used to make a record of the same timing experiment with “pure random search”, which canserve as additional base-line data. On the bbob test suite, also only the first instance of the Rosenbrock function f8had been used for this experiment previously, that is, the suite indices 105, 465, 825, 1185, 1545, 1905.5[HAN2009] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-Parameter Black-Box Optimiza-tion Benchmarking 2009: Experimental Setup, Inria Research Report RR-6828http://hal.inria.fr/inria-00362649/en, 2009.[HAN2010] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-Parameter Black-Box Optimiza-tion Benchmarking 2010: Experimental Setup, Inria Research Report RR-7215http://hal.inria.fr/inria-00362649/en, 2010.[HAN2016co] N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff.for Comparing Continuous Optimizers in a Black-Box Setting, ArXiv e-prints,arXiv:1603.08785, 2016.[HAR1999] G.R. Harik and F.G. Lobo. A parameter-less genetic algorithm. In Proceedingsof the Genetic and Evolutionary Computation Conference (GECCO), volume 1,pages 258-265. ACM, 1999.[HOO1998] H.H. Hoos and T. Stützle. Evaluating Las Vegas algorithms: pitfalls and reme-dies. In Proceedings of the Fourteenth Conference on Uncertainty in ArtificialIntelligence (UAI-98), pages 238-245, 1998.6

HAL-CentraleSupelec

https://hal.inria.fr/hal-01294167/document

COCO: The Experimental Procedure

Abstract

Similar works

Full text

Available Versions

HAL Descartes

Hal-Diderot

INRIA a CCSD electronic archive server

HAL-CentraleSupelec