Evolutionary Reinforcement Learning: A Survey

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field

Lifetime learning in evolutionary robotics

Inspired by animals ability to learn and adapt to changes in their environment during life, hybrid evolutionary algorithms which include local optimization between generations have been developed and successfully applied in a number of research areas. Despite the possible benefits this kind of algorithm could have in the field of evolutionary robotics, very little research has been done on this topic. This thesis explores the effects of learning used in cooperation with a genetic algorithm to evolve control system parameters for a fixedmorphology robot, where learning corresponds to the application of a local search algorithm on individuals during evolution. Two types of lifetime learning were implemented and tested, i.e. Baldwinian and Lamarckian learning. On the direct results from evolution, Lamarckian learning showed promising results, with a significant increase in final fitness compared with the results from evolution without learning. Machine learning is sometimes used to reduce the reality gap between performance in simulation and the real world. Based on the possibility that individuals evolved with Baldwinian learning can develop a potential to learn, this thesis also examines if learning could be advantageous when such a method is used. On this topic, the results obtained in this thesis showed promise in some sample sets, but were inconclusive in others. In order to conclude in this matter, a larger quantity of samples would be necessary

A reduced-uncertainty hybrid evolutionary algorithm for solving dynamic shortest-path routing problem

The need for effective packet transmission to deliver advanced performance in wireless networks creates the need to find shortest network paths efficiently and quickly. This paper addresses a Reduced Uncertainty Based Hybrid Evolutionary Algorithm (RUBHEA) to solve Dynamic Shortest Path Routing Problem (DSPRP) effectively and rapidly. Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) are integrated as a hybrid algorithm to find the best solution within the search space of dynamically changing networks. Both GA and PSO share context of individuals to reduce uncertainty in RUBHEA. Various regions of search space are explored and learned by RUBHEA. By employing a modified priority encoding method, each individual in both GA and PSO are represented as a potential solution for DSPRP. A Complete statistical analysis has been performed to compare the performance of RUBHEA with various state-of-the-art algorithms. It shows that RUBHEA is considerably superior (reducing the failure rate by up to 50%) to similar approaches with increasing number of nodes encountered in the networks

Bio-inspired computation: where we stand and what's next

In recent years, the research community has witnessed an explosion of literature dealing with the adaptation of behavioral patterns and social phenomena observed in nature towards efficiently solving complex computational tasks. This trend has been especially dramatic in what relates to optimization problems, mainly due to the unprecedented complexity of problem instances, arising from a diverse spectrum of domains such as transportation, logistics, energy, climate, social networks, health and industry 4.0, among many others. Notwithstanding this upsurge of activity, research in this vibrant topic should be steered towards certain areas that, despite their eventual value and impact on the field of bio-inspired computation, still remain insufficiently explored to date. The main purpose of this paper is to outline the state of the art and to identify open challenges concerning the most relevant areas within bio-inspired optimization. An analysis and discussion are also carried out over the general trajectory followed in recent years by the community working in this field, thereby highlighting the need for reaching a consensus and joining forces towards achieving valuable insights into the understanding of this family of optimization techniques

Bio-inspired computation: where we stand and what's next

In recent years, the research community has witnessed an explosion of literature dealing with the adaptation of behavioral patterns and social phenomena observed in nature towards efficiently solving complex computational tasks. This trend has been especially dramatic in what relates to optimization problems, mainly due to the unprecedented complexity of problem instances, arising from a diverse spectrum of domains such as transportation, logistics, energy, climate, social networks, health and industry 4.0, among many others. Notwithstanding this upsurge of activity, research in this vibrant topic should be steered towards certain areas that, despite their eventual value and impact on the field of bio-inspired computation, still remain insufficiently explored to date. The main purpose of this paper is to outline the state of the art and to identify open challenges concerning the most relevant areas within bio-inspired optimization. An analysis and discussion are also carried out over the general trajectory followed in recent years by the community working in this field, thereby highlighting the need for reaching a consensus and joining forces towards achieving valuable insights into the understanding of this family of optimization techniques

Traveling Salesman Problem

The idea behind TSP was conceived by Austrian mathematician Karl Menger in mid 1930s who invited the research community to consider a problem from the everyday life from a mathematical point of view. A traveling salesman has to visit exactly once each one of a list of m cities and then return to the home city. He knows the cost of traveling from any city i to any other city j. Thus, which is the tour of least possible cost the salesman can take? In this book the problem of finding algorithmic technique leading to good/optimal solutions for TSP (or for some other strictly related problems) is considered. TSP is a very attractive problem for the research community because it arises as a natural subproblem in many applications concerning the every day life. Indeed, each application, in which an optimal ordering of a number of items has to be chosen in a way that the total cost of a solution is determined by adding up the costs arising from two successively items, can be modelled as a TSP instance. Thus, studying TSP can never be considered as an abstract research with no real importance

Artificial intelligence within the interplay between natural and artificial computation:Advances in data science, trends and applications

Artificial intelligence and all its supporting tools, e.g. machine and deep learning in computational intelligence-based systems, are rebuilding our society (economy, education, life-style, etc.) and promising a new era for the social welfare state. In this paper we summarize recent advances in data science and artificial intelligence within the interplay between natural and artificial computation. A review of recent works published in the latter field and the state the art are summarized in a comprehensive and self-contained way to provide a baseline framework for the international community in artificial intelligence. Moreover, this paper aims to provide a complete analysis and some relevant discussions of the current trends and insights within several theoretical and application fields covered in the essay, from theoretical models in artificial intelligence and machine learning to the most prospective applications in robotics, neuroscience, brain computer interfaces, medicine and society, in general.BMS - Pfizer(U01 AG024904). Spanish Ministry of Science, projects: TIN2017-85827-P, RTI2018-098913-B-I00, PSI2015-65848-R, PGC2018-098813-B-C31, PGC2018-098813-B-C32, RTI2018-101114-B-I, TIN2017-90135-R, RTI2018-098743-B-I00 and RTI2018-094645-B-I00; the FPU program (FPU15/06512, FPU17/04154) and Juan de la Cierva (FJCI-2017–33022). Autonomous Government of Andalusia (Spain) projects: UMA18-FEDERJA-084. Consellería de Cultura, Educación e Ordenación Universitaria of Galicia: ED431C2017/12, accreditation 2016–2019, ED431G/08, ED431C2018/29, Comunidad de Madrid, Y2018/EMT-5062 and grant ED431F2018/02. PPMI – a public – private partnership – is funded by The Michael J. Fox Foundation for Parkinson’s Research and funding partners, including Abbott, Biogen Idec, F. Hoffman-La Roche Ltd., GE Healthcare, Genentech and Pfizer Inc

Técnicas de optimización paralelas : esquema híbrido basado en hiperheurísticas y computación evolutiva

Optimisation is the process of selecting the best element fr om a set of available alternatives. Solutions are termed good or bad depending on its performance for a set of objectives. Several algorithms to deal with such kind of problems have been defined in the literature. Metaheuristics are one of the most prominent techniques. They are a class of modern heuristics whose main goal is to com bine heuristics in a problem independent way with the aim of improving their per formance. Meta- heuristics have reported high-quality solutions in severa l fields. One of the reasons of the good behaviour of metaheuristics is that they are defin ed in general terms. Therefore, metaheuristic algorithms can be adapted to fit th e needs of most real-life optimisation. However, such an adaptation is a hard task, and it requires a high computational and user effort. There are two main ways of reducing the effort associated to th e usage of meta- heuristics. First, the application of hyperheuristics and parameter setting strategies facilitates the process of tackling novel optimisation pro blems and instances. A hyperheuristic can be viewed as a heuristic that iterativel y chooses between a set of given low-level metaheuristics in order to solve an optim isation problem. By using hyperheuristics, metaheuristic practitioners do no t need to manually test a large number of metaheuristics and parameterisations for d iscovering the proper algorithms to use. Instead, they can define the set of configur ations which must be tested, and the model tries to automatically detect the be st-behaved ones, in order to grant more resources to them. Second, the usage of pa rallel environments might speedup the process of automatic testing, so high qual ity solutions might be achieved in less time. This research focuses on the design of novel hyperheuristic s and defines a set of models to allow their usage in parallel environments. Differ ent hyperheuristics for controlling mono-objective and multi-objective multi-po int optimisation strategies have been defined. Moreover, a set of novel multiobjectivisa tion techniques has been proposed. In addition, with the aim of facilitating the usage of multiobjectivi- sation, the performance of models that combine the usage of m ultiobjectivisation and hyperheuristics has been studied. The proper performance of the proposed techniques has been v alidated with a set of well-known benchmark optimisation problems. In addi tion, several practical and complex optimisation problems have been addressed. Som e of the analysed problems arise in the communication field. In addition, a pac king problem proposed in a competition has been faced up. The proposals for such pro blems have not been limited to use the problem-independent schemes. Inste ad, new metaheuristics, operators and local search strategies have been defined. Suc h schemes have been integrated with the designed parallel hyperheuristics wit h the aim of accelerating the achievement of high quality solutions, and with the aim of fa cilitating their usage. In several complex optimisation problems, the current best -known solutions have been found with the methods defined in this dissertation.Los problemas de optimización son aquellos en los que hay que elegir cuál es la solución más adecuada entre un conjunto de alternativas. Actualmente existe una gran cantidad de algoritmos que permiten abordar este tipo de problemas. Entre ellos, las metaheurísticas son una de las técnicas más usadas. El uso de metaheurísticas ha posibilitado la resolución de una gran cantidad de problemas en diferentes campos. Esto se debe a que las metaheurísticas son técnicas generales, con lo que disponen de una gran cantidad de elementos o parámetros que pueden ser adaptados a la hora de afrontar diferentes problemas de optimización. Sin embargo, la elección de dichos parámetros no es sencilla, por lo que generalmente se requiere un gran esfuerzo computacional, y un gran esfuerzo por parte del usuario de estas técnicas. Existen diversas técnicas que atenúan este inconveniente. Por un lado, existen varios mecanismos que permiten seleccionar los valores de dichos parámetros de forma automática. Las técnicas más simples utilizan valores fijos durante toda la ejecución, mientras que las técnicas más avanzadas, como las hiperheurísticas, adaptan los valores usados a las necesidades de cada fase de optimización. Además, estas técnicas permiten usar varias metaheurísticas de forma simultánea. Por otro lado, el uso de técnicas paralelas permite acelerar el proceso de testeo automático, reduciendo el tiempo necesario para obtener soluciones de alta calidad. El objetivo principal de esta tesis ha sido diseñar nuevas hiperheurísticas e integrarlas en el modelo paralelo basado en islas. Estas técnicas se han usado para controlar los parámetros de varias metaheurísticas evolutivas. Se han definido diversas hiperheurísticas que han permitido abordar tanto problemas mono-objetivo como problemas multi-objetivo. Además, se han definido un conjunto de multiobjetivizaciones, que a su vez se han beneficiado de las hiperheurísticas propuestas. Las técnicas diseñadas se han validado con algunos de los problemas de test más ampliamente utilizados. Además, se han abordado un conjunto de problemas de optimización prácticos. Concretamente, se han tratado tres problemas que surgen en el ámbito de las telecomunicaciones, y un problema de empaquetado. En dichos problemas, además de usar las hiperheurísticas y multiobjetivizaciones, se han definido nuevos algoritmos, operadores, y estrategias de búsqueda local. En varios de los problemas, el uso combinado de todas estas técnicas ha posibilitado obtener las mejores soluciones encontradas hasta el momento

Efficient evolutionary-based neural architecture search in few GPU hours for image classification and medical image segmentation

Orientador: Lucas Ferrari de OliveiraTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa : Curitiba, 20/09/2021Inclui referências: p. 132-139Área de concentração: Ciência da ComputaçãoResumo: O uso de aprendizagem profunda (AP) está crescendo rapidamente, já que o poder computacional atual fornece otimização e inferência rápidas. Além disso, vários métodos exclusivos de AP estão evoluindo, permitindo resultados superiores em visão computacional, reconhecimento de voz e análise de texto. Os métodos AP extraem característica automaticamente para melhor representação de um problema específico, removendo o árduo trabalho do desenvolvimento de descritores de características dos métodos convencionais. Mesmo que esse processo sejaautomatizado, a criação inteligente de redes neurais é necessária para o aprendizado adequado da representação, o que requer conhecimento em AP. O campo de busca de arquiteturas neurais (BAN) foca no desenvolvimento de abordagens inteligentes que projetam redes robustas automaticamente para reduzir o conhecimento exigido para o desenvolvimento de redes eficientes. BAN pode fornecer maneiras de descobrir diferentes representações de rede, melhorando o estado da arte em diferentes aplicações. Embora BAN seja relativamente nova, várias abordagens foram desenvolvidas para descobrir modelos robustos. Métodos eficientes baseados em evolução são amplamente populares em BAN, mas seu alto consumo de placa gráfica (de alguns dias a meses)desencoraja o uso prático. No presente trabalho, propomos duas abordagens BAN baseadas na evolução eficiente com baixo custo de processamento, exigindo apenas algumas horas de processamento na placa gráfica (menos de doze em uma RTX 2080Ti) para descobrir modelos competitivos. Nossas abordagens extraem conceitos da programação de expressão gênica para representar e gerar redes baseadas em células robustas combinadas com rápido treinamento de candidatos, compartilhamento de peso e combinações dinâmicas. Além disso, os métodos propostos são empregados em um espaço de busca mais amplo, com mais células representando uma rede única. Nossa hipótese central é que BAN baseado na evolução pode ser usado em uma busca com baixo custo (combinada com uma estratégia robusta e busca eficiente) em diversas tarefas de visão computacional sem perder competitividade. Nossos métodos são avaliados em diferentes problemas para validar nossa hipótese: classificação de imagens e segmentação semântica de imagens médicas. Para tanto, as bases de dados CIFAR são estudadas para atarefa de classificação e o desafio CHAOS para a tarefa de segmentação. As menores taxas de erro encontradas nas bases CIFAR-10 e CIFAR-100 foram 2,17% ± 0,10 e 15,47% ± 0,51,respectivamente. Quanto às tarefas do desafio CHAOS, os valores de Dice ficaram entre 90% e96%. Os resultados obtidos com nossas propostas em ambas as tarefas mostraram a descoberta de redes robustas para ambas as tarefas com baixo custo na fase de busca, sendo competitivas em relação ao estado da arte em ambos os desafios.Abstract: Deep learning (DL) usage is growing fast since current computational power provides fast optimization and inference. Furthermore, several unique DL methods are evolving, enabling superior computer vision, speech recognition, and text analysis results. DL methods automatically extract features to represent a specific problem better, removing the hardworking of feature engineering from conventional methods. Even if this process is automated, intelligent network design is necessary for proper representation learning, which requires expertise in DL. The neural architecture search (NAS) field focuses on developing intelligent approaches that automatically design robust networks to reduce the expertise required for developing efficient networks. NAS may provide ways to discover different network representations, improving the state-of-the-art indifferent applications. Although NAS is relatively new, several approaches were developed for discovering robust models. Efficient evolutionary-based methods are widely popular in NAS, buttheir high GPU consumption (from a few days to months) discourages practical use. In the presentwork, we propose two efficient evolutionary-based NAS approaches with low-GPU cost, requiring only a few GPU hours (less than twelve in an RTX 2080Ti) to discover competitive models. Our approaches extract concepts from gene expression programming to represent and generate robust cell-based networks combined with fast candidate training, weight sharing, and dynamic combinations. Furthermore, the proposed methods are employed in a broader search space, withmore cells representing a unique network. Our central hypothesis is that evolutionary-based NAScan be used in a low-cost GPU search (combined with a robust strategy and efficient search) indiverse computer vision tasks without losing competitiveness. Our methods are evaluated indifferent problems to validate our hypothesis: image classification and medical image semantic segmentation. For this purpose, the CIFAR datasets are studied for the classification task andthe CHAOS challenge for the segmentation task. The lowest error rates found in CIFAR-10 andCIFAR-100 datasets were 2.17% ± 0.10 and 15.47% ± 0.51, respectively. As for the CHAOS challenge tasks, the dice scores were between 90% and 96%. The obtained results from our proposal in both tasks shown the discovery of robust networks for both tasks with little GPU costin the search phase, being competitive to state-of-the-art approaches in both challenges