Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges

Research on new optimization algorithms is often funded based on the motivation that such algorithms might improve the capabilities to deal with real-world and industrially relevant optimization challenges. Besides a huge variety of different evolutionary and metaheuristic optimization algorithms, also a large number of test problems and benchmark suites have been developed and used for comparative assessments of algorithms, in the context of global, continuous, and black-box optimization. For many of the commonly used synthetic benchmark problems or artificial fitness landscapes, there are however, no methods available, to relate the resulting algorithm performance assessments to technologically relevant real-world optimization problems, or vice versa. Also, from a theoretical perspective, many of the commonly used benchmark problems and approaches have little to no generalization value. Based on a mini-review of publications with critical comments, advice, and new approaches, this communication aims to give a constructive perspective on several open challenges and prospective research directions related to systematic and generalizable benchmarking for black-box optimization

Comparative run-time performance of evolutionary algorithms on multi-objective interpolated continuous optimisation problems.

We propose a new class of multi-objective benchmark problems on which we analyse the performance of four well established multi-objective evolutionary algorithms (MOEAs) – each implementing a different search paradigm – by comparing run-time convergence behaviour over a set of 1200 problem instances. The new benchmarks are created by fusing previously proposed single-objective interpolated continuous optimisation problems (ICOPs) via a common set of Pareto non-dominated seeds. They thus inherit the ICOP property of having tunable fitness landscape features. The benchmarks are of intrinsic interest as they derive from interpolation methods and so can approximate general problem instances. This property is revealed to be of particular importance as our extensive set of numerical experiments indicates that choices pertaining to (i) the weighting of the inverse distance interpolation function and (ii) the problem dimension can be used to construct problems that are challenging to all tested multi-objective search paradigms. This in turn means that the new multi-objective ICOPs problems (MO-ICOPs) can be used to construct well-balanced benchmark sets that discriminate well between the run-time convergence behaviour of different solvers

Metaheuristic optimization of power and energy systems: underlying principles and main issues of the 'rush to heuristics'

In the power and energy systems area, a progressive increase of literature contributions containing applications of metaheuristic algorithms is occurring. In many cases, these applications are merely aimed at proposing the testing of an existing metaheuristic algorithm on a specific problem, claiming that the proposed method is better than other methods based on weak comparisons. This 'rush to heuristics' does not happen in the evolutionary computation domain, where the rules for setting up rigorous comparisons are stricter, but are typical of the domains of application of the metaheuristics. This paper considers the applications to power and energy systems, and aims at providing a comprehensive view of the main issues concerning the use of metaheuristics for global optimization problems. A set of underlying principles that characterize the metaheuristic algorithms is presented. The customization of metaheuristic algorithms to fit the constraints of specific problems is discussed. Some weaknesses and pitfalls found in literature contributions are identified, and specific guidelines are provided on how to prepare sound contributions on the application of metaheuristic algorithms to specific problems

Metaheuristic Optimization of Power and Energy Systems: Underlying Principles and Main Issues of the `Rush to Heuristics'

In the power and energy systems area, a progressive increase of literature contributions that contain applications of metaheuristic algorithms is occurring. In many cases, these applications are merely aimed at proposing the testing of an existing metaheuristic algorithm on a specific problem, claiming that the proposed method is better than other methods that are based on weak comparisons. This ‘rush to heuristics’ does not happen in the evolutionary computation domain, where the rules for setting up rigorous comparisons are stricter but are typical of the domains of application of the metaheuristics. This paper considers the applications to power and energy systems and aims at providing a comprehensive view of the main issues that concern the use of metaheuristics for global optimization problems. A set of underlying principles that characterize the metaheuristic algorithms is presented. The customization of metaheuristic algorithms to fit the constraints of specific problems is discussed. Some weaknesses and pitfalls that are found in literature contributions are identified, and specific guidelines are provided regarding how to prepare sound contributions on the application of metaheuristic algorithms to specific problems

MONEDA: scalable multi-objective optimization with a neural network-based estimation of distribution algorithm

The Extension Of Estimation Of Distribution Algorithms (Edas) To The Multiobjective Domain Has Led To Multi-Objective Optimization Edas (Moedas). Most Moedas Have Limited Themselves To Porting Single-Objective Edas To The Multi-Objective Domain. Although Moedas Have Proved To Be A Valid Approach, The Last Point Is An Obstacle To The Achievement Of A Significant Improvement Regarding "Standard" Multi-Objective Optimization Evolutionary Algorithms. Adapting The Model-Building Algorithm Is One Way To Achieve A Substantial Advance. Most Model-Building Schemes Used So Far By Edas Employ Off-The-Shelf Machine Learning Methods. However, The Model-Building Problem Has Particular Requirements That Those Methods Do Not Meet And Even Evade. The Focus Of This Paper Is On The Model- Building Issue And How It Has Not Been Properly Understood And Addressed By Most Moedas. We Delve Down Into The Roots Of This Matter And Hypothesize About Its Causes. To Gain A Deeper Understanding Of The Subject We Propose A Novel Algorithm Intended To Overcome The Draw-Backs Of Current Moedas. This New Algorithm Is The Multi-Objective Neural Estimation Of Distribution Algorithm (Moneda). Moneda Uses A Modified Growing Neural Gas Network For Model-Building (Mb-Gng). Mb-Gng Is A Custom-Made Clustering Algorithm That Meets The Above Demands. Thanks To Its Custom-Made Model-Building Algorithm, The Preservation Of Elite Individuals And Its Individual Replacement Scheme, Moneda Is Capable Of Scalably Solving Continuous Multi-Objective Optimization Problems. It Performs Better Than Similar Algorithms In Terms Of A Set Of Quality Indicators And Computational Resource Requirements.This work has been funded in part by projects CNPq BJT 407851/2012-7, FAPERJ APQ1 211.451/2015, MINECO TEC2014-57022-C2-2-R and TEC2012-37832-C02-01

Optimisation and Bayesian optimality

This doctoral thesis will present the results of work into optimisation algorithms. We first give a detailed exploration of the problems involved in comparing optimisation algorithms. In particular we provide extensions and refinements to no free lunch results, exploring algorithms with arbitrary stopping conditions, optimisation under restricted metrics, parallel computing and free lunches, and head-to-head minimax behaviour. We also characterise no free lunch results in terms of order statistics. We then ask what really constitutes understanding of an optimisation algorithm. We argue that one central part of understanding an optimiser is knowing its Bayesian prior and cost function. We then pursue a general Bayesian framing of optimisation, and prove that this Bayesian perspective is applicable to all optimisers, and that even seemingly non-Bayesian optimisers can be understood in this way. Specifically we prove that arbitrary optimisation algorithms can be represented as a prior and a cost function. We examine the relationship between the Kolmogorov complexity of the optimiser and the Kolmogorov complexity of it’s corresponding prior. We also extended our results from deterministic optimisers to stochastic optimisers and forgetful optimisers, and we show that uniform randomly selecting a prior is not equivalent to uniform randomly selecting an optimisation behaviour. Lastly we consider what the best way to go about gaining a Bayesian understanding of real optimisation algorithms is. We use the developed Bayesian framework to explore the affects of some common approaches to constructing meta-heuristic optimisation algorithms, such as on-line parameter adaptation. We conclude by exploring an approach to uncovering the probabilistic beliefs of optimisers with a “shattering” method

Parallel Multi-Objective Evolutionary Algorithms: A Comprehensive Survey

Multi-Objective Evolutionary Algorithms (MOEAs) are powerful search techniques that have been extensively used to solve difficult problems in a wide variety of disciplines. However, they can be very demanding in terms of computational resources. Parallel implementations of MOEAs (pMOEAs) provide considerable gains regarding performance and scalability and, therefore, their relevance in tackling computationally expensive applications. This paper presents a survey of pMOEAs, describing a refined taxonomy, an up-to-date review of methods and the key contributions to the field. Furthermore, some of the open questions that require further research are also briefly discussed