Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges

Research on new optimization algorithms is often funded based on the motivation that such algorithms might improve the capabilities to deal with real-world and industrially relevant optimization challenges. Besides a huge variety of different evolutionary and metaheuristic optimization algorithms, also a large number of test problems and benchmark suites have been developed and used for comparative assessments of algorithms, in the context of global, continuous, and black-box optimization. For many of the commonly used synthetic benchmark problems or artificial fitness landscapes, there are however, no methods available, to relate the resulting algorithm performance assessments to technologically relevant real-world optimization problems, or vice versa. Also, from a theoretical perspective, many of the commonly used benchmark problems and approaches have little to no generalization value. Based on a mini-review of publications with critical comments, advice, and new approaches, this communication aims to give a constructive perspective on several open challenges and prospective research directions related to systematic and generalizable benchmarking for black-box optimization

Treasure hunt : a framework for cooperative, distributed parallel optimization

Orientador: Prof. Dr. Daniel WeingaertnerCoorientadora: Profa. Dra. Myriam Regattieri DelgadoTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa : Curitiba, 27/05/2019Inclui referências: p. 18-20Área de concentração: Ciência da ComputaçãoResumo: Este trabalho propõe um framework multinível chamado Treasure Hunt, que é capaz de distribuir algoritmos de busca independentes para um grande número de nós de processamento. Com o objetivo de obter uma convergência conjunta entre os nós, este framework propõe um mecanismo de direcionamento que controla suavemente a cooperação entre múltiplas instâncias independentes do Treasure Hunt. A topologia em árvore proposta pelo Treasure Hunt garante a rápida propagação da informação pelos nós, ao mesmo tempo em que provê simutaneamente explorações (pelos nós-pai) e intensificações (pelos nós-filho), em vários níveis de granularidade, independentemente do número de nós na árvore. O Treasure Hunt tem boa tolerância à falhas e está parcialmente preparado para uma total tolerância à falhas. Como parte dos métodos desenvolvidos durante este trabalho, um método automatizado de Particionamento Iterativo foi proposto para controlar o balanceamento entre explorações e intensificações ao longo da busca. Uma Modelagem de Estabilização de Convergência para operar em modo Online também foi proposto, com o objetivo de encontrar pontos de parada com bom custo/benefício para os algoritmos de otimização que executam dentro das instâncias do Treasure Hunt. Experimentos em benchmarks clássicos, aleatórios e de competição, de vários tamanhos e complexidades, usando os algoritmos de busca PSO, DE e CCPSO2, mostram que o Treasure Hunt melhora as características inerentes destes algoritmos de busca. O Treasure Hunt faz com que os algoritmos de baixa performance se tornem comparáveis aos de boa performance, e os algoritmos de boa performance possam estender seus limites até problemas maiores. Experimentos distribuindo instâncias do Treasure Hunt, em uma rede cooperativa de até 160 processos, demonstram a escalabilidade robusta do framework, apresentando melhoras nos resultados mesmo quando o tempo de processamento é fixado (wall-clock) para todas as instâncias distribuídas do Treasure Hunt. Resultados demonstram que o mecanismo de amostragem fornecido pelo Treasure Hunt, aliado à maior cooperação entre as múltiplas populações em evolução, reduzem a necessidade de grandes populações e de algoritmos de busca complexos. Isto é especialmente importante em problemas de mundo real que possuem funções de fitness muito custosas. Palavras-chave: Inteligência artificial. Métodos de otimização. Algoritmos distribuídos. Modelagem de convergência. Alta dimensionalidade.Abstract: This work proposes a multilevel framework called Treasure Hunt, which is capable of distributing independent search algorithms to a large number of processing nodes. Aiming to obtain joint convergences between working nodes, Treasure Hunt proposes a driving mechanism that smoothly controls the cooperation between the multiple independent Treasure Hunt instances. The tree topology proposed by Treasure Hunt ensures quick propagation of information, while providing simultaneous explorations (by parents) and exploitations (by children), on several levels of granularity, regardless the number of nodes in the tree. Treasure Hunt has good fault tolerance and is partially prepared to full fault tolerance. As part of the methods developed during this work, an automated Iterative Partitioning method is proposed to control the balance between exploration and exploitation as the search progress. A Convergence Stabilization Modeling to operate in Online mode is also proposed, aiming to find good cost/benefit stopping points for the optimization algorithms running within the Treasure Hunt instances. Experiments on classic, random and competition benchmarks of various sizes and complexities, using the search algorithms PSO, DE and CCPSO2, show that Treasure Hunt boosts the inherent characteristics of these search algorithms. Treasure Hunt makes algorithms with poor performances to become comparable to good ones, and algorithms with good performances to be capable of extending their limits to larger problems. Experiments distributing Treasure Hunt instances in a cooperative network up to 160 processes show the robust scaling of the framework, presenting improved results even when fixing a wall-clock time for the instances. Results show that the sampling mechanism provided by Treasure Hunt, allied to the increased cooperation between multiple evolving populations, reduce the need for large population sizes and complex search algorithms. This is specially important on real-world problems with time-consuming fitness functions. Keywords: Artificial intelligence. Optimization methods. Distributed algorithms. Convergence modeling. High dimensionality

Reproducibility in evolutionary computation

Experimental studies are prevalent in Evolutionary Computation (EC), and concerns about the reproducibility and replicability of such studies have increased in recent times, reflecting similar concerns in other scientific fields. In this article, we discuss, within the context of EC, the different types of reproducibility and suggest a classification that refines the badge system of the Association of Computing Machinery (ACM) adopted by ACM Transactions on Evolutionary Learning and Optimization (TELO). We identify cultural and technical obstacles to reproducibility in the EC field. Finally, we provide guidelines and suggest tools that may help to overcome some of these reproducibility obstacles

Beyond Particular Problem Instances: How to Create Meaningful and Generalizable Results

Computational intelligence methods have gained importance in several real-world domains such as process optimization, system identification, data mining, or statistical quality control. Tools are missing, which determine the applicability of computational intelligence methods in these application domains in an objective manner. Statistics provide methods for comparing algorithms on certain data sets. In the past, several test suites were presented and considered as state of the art. However, there are several drawbacks of these test suites, namely: (i) problem instances are somehow artificial and have no direct link to real-world settings; (ii) since there is a fixed number of test instances, algorithms can be fitted or tuned to this specific and very limited set of test functions; (iii) statistical tools for comparisons of several algorithms on several test problem instances are relatively complex and not easily to analyze. We propose a methodology to overcome these difficulties. It is based on standard ideas from statistics: analysis of variance and its extension to mixed models. This paper combines essential ideas from two approaches: problem generation and statistical analysis of computer experiments

Statistical Models for the Analysis of Optimization Algorithms with Benchmark Functions

Frequentist statistical methods, such as hypothesis testing, are standard practice in papers that provide benchmark comparisons. Unfortunately, these methods have often been misused, e.g., without testing for their statistical test assumptions or without controlling for family-wise errors in multiple group comparisons, among several other problems. Bayesian Data Analysis (BDA) addresses many of the previously mentioned shortcomings but its use is not widely spread in the analysis of empirical data in the evolutionary computing community. This paper provides three main contributions. First, we motivate the need for utilizing Bayesian data analysis and provide an overview of this topic. Second, we discuss the practical aspects of BDA to ensure that our models are valid and the results transparent. Finally, we provide five statistical models that can be used to answer multiple research questions. The online appendix provides a step-by-step guide on how to perform the analysis of the models discussed in this paper, including the code for the statistical models, the data transformations and the discussed tables and figures.Comment: In submissio

On Experimentation in Software-Intensive Systems

Context: Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain. Objective: This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies. Method: To achieve these objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods: case studies, literature reviews, simulations, and empirical evaluations. Results: This thesis provides six main results. First, it proposes an architecture framework for automated experimentation that can be used with different types of experimental designs in both embedded systems and web-facing systems. Second, it proposes a new experimentation process to capture the details of a trustworthy experimentation process that can be used as the basis for an automated experimentation process. Third, it identifies the restrictions and pitfalls of different multi-armed bandit algorithms for automating experiments in industry. This thesis also proposes a set of guidelines to help practitioners select a technique that minimizes the occurrence of these pitfalls. Fourth, it proposes statistical models to analyze optimization algorithms that can be used in automated experimentation. Fifth, it identifies the key challenges faced by embedded systems companies when adopting controlled experimentation, and we propose a set of strategies to address these challenges. Sixth, it identifies experimentation techniques and proposes a new continuous experimentation model for mission-critical and business-to-business. Conclusion: The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint

Perfiles cognitivos y técnicas para la generación de ideas: el caso de una empresa del sector gráfico de Medellín

Resumen: en los últimos años la economía global ha evolucionado, viviendo un proceso evolutivo que hoy por hoy se traduce en la búsqueda constante de la innovación como estrategia para lograr permanencia y rentabilidad. Reconociendo el origen de la innovación en la generación de ideas, es importante cuestionarse sobre ¿cómo se influencia la generación de las ideas, partiendo de las técnicas creativas y del ser humano, como ser cognitivo y generador de las mismas?, esto con el fin de lograr la mayor cantidad posible de ideas de calidad, novedosas y variadas. El objetivo de este trabajo es identificar cómo se generan las ideas y caracterizar según el estilo cognitivo y las técnicas creativas, la calidad de las ideas generadas por los colaboradores de una empresa del sector gráfico de Medellín; La estrategia de investigación consiste en el estudio de caso (Yin, 1992) desde un enfoque cualitativo y cuantitativo. El análisis cualitativo se basa en el método de la teoría fundamentada (de Carvalho Dantas, Leite, de Lima, and Stipp, 2009), en donde la recolección de datos, se realiza a partir de la observación participante del investigador y las entrevistas a los observadores de las pruebas, con el objetivo de lograr la triangulación de los hallazgos cualitativos; por su parte el análisis cuantitativo se basa en el diseño de experimentos factorial mixto (Montgomery, 2005), la recolección de los datos se realiza con el juego gerencial, en donde se aplican las técnicas creativas, el método para clasificar el estilo cognitivo y el método para calificar las ideas generadas; El nivel del estudio es individual, sin embargo la unidad de análisis son las ideas que cada uno de estos 250 colaboradores genera. En el caso de estudio se encontró con el análisis cuantitativo, una relación significativa entre la interacción del estilo cognitivo, la técnica creativa y la calidad de las ideas; mientras que con el análisis cualitativo se encontró la importancia de considerar no solo el estilo cognitivo como aspecto psicológico, sino también otros aspectos como la motivación y la personalidad. Adicionalmente es importante resaltar otros aprendizajes relacionados con: el análisis de las situaciones sociales desde enfoques diferentes (cuantitativo y cualitativo), partiendo del hecho de que con el análisis cualitativo no se pierde la información que debe omitirse con el análisis cuantitativo, para garantizar el ajuste estadístico y cumplimiento de los supuestos del diseño de experimentos seleccionado; la inclusión de variables específicas, como las variables latentes, para modelar efectos relacionados con el ser humano y su conducta; las pruebas piloto para las investigaciones cuya estrategia de investigación es el estudio de caso; la pertinencia de los juegos gerenciales para modelar ambientes organizacionales y la importancia de identificar según la etapa del proceso de innovación que se esté analizando la métrica de calidad que mejor se ajusta, entre otrosAbstract: in recent years the global economy has evolved, living an evolutionary process which today results in the constant search of innovation as a strategy for retention and profitability. Recognizing the source of innovation in the ideas generation, it is important to ask about how it can be influence, based on the creative techniques and human, such as cognitive and generator of them?, This in order to achieve the greatest possible number of ideas of quality, novelty and varied. The aim of this work is to identify how ideas are generated and characterized according to the cognitive profile and the creative techniques, the quality of the ideas generated by employees of a company of the graphics sector of Medellín; research strategy involves the case study (Yin, 1992) with a qualitative and quantitative approach. The qualitative analysis, is based on grounded theory method (de Carvalho Dantas Leite Lima, and Stipp , 2009), where data collection is done with the participant observation of the researcher, and interviews with observers, with the goal of triangulation of qualitative findings. The quantitative analysis is based on mixed factorial experiment (Montgomery, 2005), data collection is done with the game management, where are apply: the creative techniques, the method to classify the cognitive profile and the method to rate the ideas generated, the level of study is individual, however the unit of analysis is the ideas that each one of these 250 employees generated. In the case study, with the quantitative analysis found a significant relationship between the interactions of cognitive profile, creative technique and quality of ideas, with the qualitative analysis found the importance of considering not only the cognitive profile, but also other aspects such as motivation and personality. Additionally it is important to highlight other related learning like: the analysis of social situations from different approaches (quantitative and qualitative), based on the fact that with the qualitative analysis is not necessary to lost the information, that has to be omitted with quantitative analysis, to guarantee the statistical fit and compliance of the selected experimental design, the inclusion of specific variables as latent variables to model effects related to human beings and their conduct, the importance of pilot tests for research that use case study , the relevance of managerial games to simulate organizational environments and the importance of identifying according to the stage of the innovation process the quality metrics that best fits, etcMaestrí

Mixed Models for the Analysis of Optimization Algorithms

We review linear statistical models for the analysis of computational experiments on optimization algorithms. The models offer the mathematical framework to separate the effects of algorithmic components and instance features included in the analysis. We regard test instances as drawn from a population and we focus our interest not on those single instances but on the whole population. Hence, instances are treated as a random factor. Overall these experimental designs leads to mixed effects linear models. We present both the theory to justify these models and a computational example in which we analyze and comment several possible experimental designs. The example is a component-wise analysis of local search algorithms for the 2-edge-connectivity augmentation problem. We use standard statistical software to perform the analysis and report the R commands. Data sets and the analysis in SAS are available in an online compendium.