Comparing and Combining Lexicase Selection and Novelty Search

Lexicase selection and novelty search, two parent selection methods used in evolutionary computation, emphasize exploring widely in the search space more than traditional methods such as tournament selection. However, lexicase selection is not explicitly driven to select for novelty in the population, and novelty search suffers from lack of direction toward a goal, especially in unconstrained, highly-dimensional spaces. We combine the strengths of lexicase selection and novelty search by creating a novelty score for each test case, and adding those novelty scores to the normal error values used in lexicase selection. We use this new novelty-lexicase selection to solve automatic program synthesis problems, and find it significantly outperforms both novelty search and lexicase selection. Additionally, we find that novelty search has very little success in the problem domain of program synthesis. We explore the effects of each of these methods on population diversity and long-term problem solving performance, and give evidence to support the hypothesis that novelty-lexicase selection resists converging to local optima better than lexicase selection

Discovering Representations for Black-box Optimization

The encoding of solutions in black-box optimization is a delicate, handcrafted balance between expressiveness and domain knowledge -- between exploring a wide variety of solutions, and ensuring that those solutions are useful. Our main insight is that this process can be automated by generating a dataset of high-performing solutions with a quality diversity algorithm (here, MAP-Elites), then learning a representation with a generative model (here, a Variational Autoencoder) from that dataset. Our second insight is that this representation can be used to scale quality diversity optimization to higher dimensions -- but only if we carefully mix solutions generated with the learned representation and those generated with traditional variation operators. We demonstrate these capabilities by learning an low-dimensional encoding for the inverse kinematics of a thousand joint planar arm. The results show that learned representations make it possible to solve high-dimensional problems with orders of magnitude fewer evaluations than the standard MAP-Elites, and that, once solved, the produced encoding can be used for rapid optimization of novel, but similar, tasks. The presented techniques not only scale up quality diversity algorithms to high dimensions, but show that black-box optimization encodings can be automatically learned, rather than hand designed.Comment: Presented at GECCO 2020 -- v2 (Previous title 'Automating Representation Discovery with MAP-Elites'

Distributed MAP-Elites and its Application in Evolutionary Design

Quality-Diversity search is the process of finding diverse solutions within the search space which do not sacrifice performance. MAP-Elites is a quality-diversity algorithm which measures n phenotypes/behaviours of a solution and places it into an $n$-dimensional hypercube based off its phenotype values. This thesis proposes an approach to addressing MAP-Elites' problem of exponential growth of hypercubes. The exponential growth of evaluation and computational time as the phenotypes/behaviours grow is potentially worse for optimization performance. The exponential growth in individuals results in the user being given too many candidate solutions at the end of processing. Therefore, MAP-Elites highlights diversity, but with the exponential growth, the said diversity is arguably impractical. This research proposes an enhancement to MAP-Elites with Distributed island-model evolution. This will introduce a linear growth in population as well as a reasonable number of candidate solutions to consider. Each island consists of a two dimensional MAP which allows for a realistic analysis and visualization of these individuals. Since the system increases on a linear scale, and MAP-Elites on an exponential scale, high-dimensional problems will show an even greater decrease in total candidate solution counts, which aids in the realistic analysis of a run. This system will then be tested on procedural texture generation with multiple computer vision fitness functions. This Distributed MAP-Elites algorithm was tested against vanilla GP, island-model evolution, and traditional MAP-Elites on multiple fitness functions and target images. The proposed algorithm was found, at the very minimum, to be competitive in fitness to the other algorithms and in some cases outperformed them. On top of this performance, when visually observing the best solutions, the algorithm was found to have been able to produce visually interesting textures

Illuminating meaningful diversity in complex feature spaces through adaptive grid-based genetic algorithms

In many fields there exist problems for which multiple solutions of suitably high performance may be found across distinct regions of the search space. Optimisation of the search towards including these distinct solutions is important not only to understanding these spaces but also to avoiding local optima. This is the goal of a type of genetic algorithms called illumination algorithms. In Chapter 2, we demonstrate the use of an illumination algorithm in the exploration of networks sharing only a given set of structural features (valid networks). This method produces a population of valid networks that are more diverse than those produced using state of the art methods, however, it was found to be too inefficient to be usable in real-world problems. Additionally, setting an appropriate resolution of the search requires some amount of prior knowledge of the space of solutions. Addressing this problem is the focus of Chapter 3, in which we develop three extensions to the method: a) an exact method of mutation whereby only valid networks are explored, b) an adaptive mechanism for setting the resolution of the search, c) a principle for tuning mutations parameters to the search’ s resolution. We show that with these additions our method is able to increase the diversity of solutions found in significantly fewer iterations. Finally, in Chapter 4 we expand our method for use in more general problem spaces. We benchmark it against the state of the art. In all tested landscapes, we show that our method is able to identify more meaningful niches in the spaces in the same number of iterations. We conclude by highlighting the limits of our framework and discuss further directions

Evolutionary Reinforcement Learning: A Survey

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field

Generating Levels That Teach Mechanics

The automatic generation of game tutorials is a challenging AI problem. While it is possible to generate annotations and instructions that explain to the player how the game is played, this paper focuses on generating a gameplay experience that introduces the player to a game mechanic. It evolves small levels for the Mario AI Framework that can only be beaten by an agent that knows how to perform specific actions in the game. It uses variations of a perfect A* agent that are limited in various ways, such as not being able to jump high or see enemies, to test how failing to do certain actions can stop the player from beating the level.Comment: 8 pages, 7 figures, PCG Workshop at FDG 2018, 9th International Workshop on Procedural Content Generation (PCG2018