Forcing neurocontrollers to exploit sensory symmetry through hard-wired modularity in the game of Cellz

Several attempts have been made in the past to construct encoding schemes that allow modularity to emerge in evolving systems, but success is limited. We believe that in order to create successful and scalable encodings for emerging modularity, we first need to explore the benefits of different types of modularity by hard-wiring these into evolvable systems. In this paper we explore different ways of exploiting sensory symmetry inherent in the agent in the simple game Cellz by evolving symmetrically identical modules. It is concluded that significant increases in both speed of evolution and final fitness can be achieved relative to monolithic controllers. Furthermore, we show that a simple function approximation task that exhibits sensory symmetry can be used as a quick approximate measure of the utility of an encoding scheme for the more complex game-playing task

Evolutionary Reinforcement Learning: A Survey

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field

Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients

While neuroevolution (evolving neural networks) has a successful track record across a variety of domains from reinforcement learning to artificial life, it is rarely applied to large, deep neural networks. A central reason is that while random mutation generally works in low dimensions, a random perturbation of thousands or millions of weights is likely to break existing functionality, providing no learning signal even if some individual weight changes were beneficial. This paper proposes a solution by introducing a family of safe mutation (SM) operators that aim within the mutation operator itself to find a degree of change that does not alter network behavior too much, but still facilitates exploration. Importantly, these SM operators do not require any additional interactions with the environment. The most effective SM variant capitalizes on the intriguing opportunity to scale the degree of mutation of each individual weight according to the sensitivity of the network's outputs to that weight, which requires computing the gradient of outputs with respect to the weights (instead of the gradient of error, as in conventional deep learning). This safe mutation through gradients (SM-G) operator dramatically increases the ability of a simple genetic algorithm-based neuroevolution method to find solutions in high-dimensional domains that require deep and/or recurrent neural networks (which tend to be particularly brittle to mutation), including domains that require processing raw pixels. By improving our ability to evolve deep neural networks, this new safer approach to mutation expands the scope of domains amenable to neuroevolution

Meta-parametric design: Developing a computational approach for early stage collaborative practice

Computational design is the study of how programmable computers can be integrated into the process of design. It is not simply the use of pre-compiled computer aided design software that aims to replicate the drawing board, but rather the development of computer algorithms as an integral part of the design process. Programmable machines have begun to challenge traditional modes of thinking in architecture and engineering, placing further emphasis on process ahead of the final result. Just as Darwin and Wallace had to think beyond form and inquire into the development of biological organisms to understand evolution, so computational methods enable us to rethink how we approach the design process itself. The subject is broad and multidisciplinary, with influences from design, computer science, mathematics, biology and engineering. This thesis begins similarly wide in its scope, addressing both the technological aspects of computational design and its application on several case study projects in professional practice. By learning through participant observation in combination with secondary research, it is found that design teams can be most effective at the early stage of projects by engaging with the additional complexity this entails. At this concept stage, computational tools such as parametric models are found to have insufficient flexibility for wide design exploration. In response, an approach called Meta-Parametric Design is proposed, inspired by developments in genetic programming (GP). By moving to a higher level of abstraction as computational designers, a Meta-Parametric approach is able to adapt to changing constraints and requirements whilst maintaining an explicit record of process for collaborative working

Embryomorphic Engineering: Emergent innovation through evolutionary development

Embryomorphic Engineering, a particular instance of Morpho-genetic Engineering, takes its inspiration directly from biological development to create new hardware, software or network architectures by decentralized self-assembly of elementary agents. At its core, it combines three key principles of multicellular embryogenesis: chemical gradient di usion (providing positional information to the agents), gene regulatory networks (triggering their diferentiation into types, thus patterning), and cell division (creating structural constraints, thus reshaping). This chapter illustrates the potential of Embryomorphic Engineering in di erent spaces: 2D/3D physical swarms, which can nd applications in collective robotics, synthetic biology or nan- otechnology; and nD graph topologies, which can nd applications in dis- tributed software and peer-to-peer techno-social networks. In all cases, the speci c genotype shared by all the agents makes the phenotype's complex architecture and function modular, programmable and reproducible

Training Neural Networks Through the Integration of Evolution and Gradient Descent

Neural networks have achieved widespread adoption due to both their applicability to a wide range of problems and their success relative to other machine learning algorithms. The training of neural networks is achieved through any of several paradigms, most prominently gradient-based approaches (including deep learning), but also through up-and-coming approaches like neuroevolution. However, while both of these neural network training paradigms have seen major improvements over the past decade, little work has been invested in developing algorithms that incorporate the advances from both deep learning and neuroevolution. This dissertation introduces two new algorithms that are steps towards the integration of gradient descent and neuroevolution for training neural networks. The first is (1) the Limited Evaluation Evolutionary Algorithm (LEEA), which implements a novel form of evolution where individuals are partially evaluated, allowing rapid learning and enabling the evolutionary algorithm to behave more like gradient descent. This conception provides a critical stepping stone to future algorithms that more tightly couple evolutionary and gradient descent components. The second major algorithm (2) is Divergent Discriminative Feature Accumulation (DDFA), which combines a neuroevolution phase, where features are collected in an unsupervised manner, with a gradient descent phase for fine tuning of the neural network weights. The neuroevolution phase of DDFA utilizes an indirect encoding and novelty search, which are sophisticated neuroevolution components rarely incorporated into gradient descent-based systems. Further contributions of this work that build on DDFA include (3) an empirical analysis to identify an effective distance function for novelty search in high dimensions and (4) the extension of DDFA for the purpose of discovering convolutional features. The results of these DDFA experiments together show that DDFA discovers features that are effective as a starting point for gradient descent, with significant improvement over gradient descent alone. Additionally, the method of collecting features in an unsupervised manner allows DDFA to be applied to domains with abundant unlabeled data and relatively sparse labeled data. This ability is highlighted in the STL-10 domain, where DDFA is shown to make effective use of unlabeled data

Neuroevolution for solving multiobjective knapsack problems

The multiobjective knapsack problem (MOKP) is an important combinatorial problem that arises in various applications, including resource allocation, computer science and finance. When tackling this problem by evolutionary multiobjective optimization algorithms (EMOAs), it has been demonstrated that traditional recombination operators acting on binary solution representations are susceptible to a loss of diversity and poor scalability. To address those issues, we propose to use artificial neural networks for generating solutions by performing a binary classification of items using the information about their profits and weights. As gradient-based learning cannot be used when target values are unknown, neuroevolution is adapted to adjust the neural network parameters. The main contribution of this study resides in developing a solution encoding and genotype-phenotype mapping for EMOAs to solve MOKPs. The proposal is implemented within a state-of-the-art EMOA and benchmarked against traditional variation operators based on binary crossovers. The obtained experimental results indicate a superior performance of the proposed approach. Furthermore, it is advantageous in terms of scalability and can be readily incorporated into different EMOAs.Portuguese “Fundação para a Ciência e Tecnologia” under grant PEst-C/CTM/LA0025/2013 (Projecto Estratégico - LA 25 - 2013-2014 - Strategic Project - LA 25 - 2013-2014

Neuroevolution of Self-Interpretable Agents

Inattentional blindness is the psychological phenomenon that causes one to miss things in plain sight. It is a consequence of the selective attention in perception that lets us remain focused on important parts of our world without distraction from irrelevant details. Motivated by selective attention, we study the properties of artificial agents that perceive the world through the lens of a self-attention bottleneck. By constraining access to only a small fraction of the visual input, we show that their policies are directly interpretable in pixel space. We find neuroevolution ideal for training self-attention architectures for vision-based reinforcement learning (RL) tasks, allowing us to incorporate modules that can include discrete, non-differentiable operations which are useful for our agent. We argue that self-attention has similar properties as indirect encoding, in the sense that large implicit weight matrices are generated from a small number of key-query parameters, thus enabling our agent to solve challenging vision based tasks with at least 1000x fewer parameters than existing methods. Since our agent attends to only task critical visual hints, they are able to generalize to environments where task irrelevant elements are modified while conventional methods fail. Videos of our results and source code available at https://attentionagent.github.io/Comment: To appear at the Genetic and Evolutionary Computation Conference (GECCO 2020) as a full pape

Evolutionary Design Model for the design of complex engineered systems : Masdar City as a case study

Thesis (S.M.)--Massachusetts Institute of Technology, Computation for Design and Optimization Program, 2009."September 2009." Cataloged from PDF version of thesis.Includes bibliographical references (p. 150-157).This thesis develops a framework for constructing an Evolutionary Design Model (EDM) that would enhance the design of complex systems through an efficient process. The framework proposed is generic and suggests a group of systematic methodologies that eventually lead to a fully realized and integrated design model. Within this model, complexities of the design are handled and the uncertainties of the design evolution are managed. Using the framework, vast design spaces can be searched while solutions are intelligently modified, their performance evaluated, and their results aggregated into a compatible set for design decisions. The EDM is composed of several design states as well as design evolving processes. A design state describes a design at a particular point in time and maps the system's object to the system's requirements and identifies its relation to the context in which the system will operate. A design evolving process involves many sub-processes which include formulation, decomposition, modeling, and integration. These sub-processes are not always carried out in a sequential manner, but rather a continuous move back and forth to previous and subsequent stages is expected. The resulting design model is described as an evolutionary model that moves a system's design from simple abstract states to more complex and detailed states throughout its evolution.(cont.) The framework utilizes system modeling methodologies that include both logical and mathematical modeling methods. The type of model used within the EDM's evolving processes is highly dependent on and driven by design needs of each process. As the design progresses a shift from logical models to mathematical models occurs within the EDM. Finally, a partial EDM is implemented within the context of a computational design system for Masdar city to demonstrate the application of the proposed framework.By Anas Alfaris.S.M