Evolving Reinforcement Learning-Like Abilities for Robots

In [8] Yamauchi and Beer explored the abilities of continuous time recurrent neural networks (CTRNNs) to display reinforcement-learning like abilities. The investigated tasks were generation and learning of short bit sequences. This "learning'' came about without modifications of synaptic strengths, but simply from internal dynamics of the evolved networks. In this paper this approach will be extended to two embodied agent tasks, where simulated robots have acquire and retain "knowledge'' while moving around different mazes. The evolved controllers are analyzed and the results are discussed

Levels of Dynamics and Adaptive Behavior in Evolutionary Neural Controllers

Two classes of dynamical recurrent neural networks, Continuous Time Recurrent Neural Networks (CTRNNs) (Yamauchi and Beer, 1994) and Plastic Neural Networks (PNNs) (Floreano and Urzelai, 2000) are compared on two behavioral tasks aimed at exploring their capabilities to display reinforcement-learning like behaviors and adaptation to unpredictable environmental changes. The networks report similar performances on both tasks, but PNNs display significantly better performance when sensory-motor re-adaptation is required after the evolutionary process. These results are discussed in the context of behavioral, biological, and computational definitions of learning

Levels of Dynamics and Adaptive Behavior in Evolutionary Neural Controllers

Two classes of dynamical recurrent neural networks, Continuous Time Recurrent Neural Networks (CTRNNs) (Yamauchi and Beer, 1994) and Plastic Neural Networks (PNNs) (Floreano and Urzelai, 2000) are compared on two behavioral tasks aimed at exploring their capabilities to display reinforcement-learning like behaviors and adaptation to unpredictable environmental changes. The networks report similar performances on both tasks, but PNNs display significantly better performance when sensory-motor re-adaptation is required after the evolutionary process. These results are discussed in the context of behavioral, biological, and computational definitions of learning

Evolutionary Bits'n'Spikes

We describe a model and implementation of evolutionary spiking neurons for embedded microcontrollers with few bytes of memory and very low power consumption. The approach is tested with an autonomous microrobot of less than 1 in^3 that evolves the ability to move in a small maze without human intervention and external computers. Considering the very large diffusion, small size, and low cost of embedded microcontrollers, the approach described here could find its way in several intelligent devices with sensors and/or actuators, as well as in smart credit cards

Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems

Rapid online adaptation to changing tasks is an important problem in machine learning and, recently, a focus of meta-reinforcement learning. However, reinforcement learning (RL) algorithms struggle in POMDP environments because the state of the system, essential in a RL framework, is not always visible. Additionally, hand-designed meta-RL architectures may not include suitable computational structures for specific learning problems. The evolution of online learning mechanisms, on the contrary, has the ability to incorporate learning strategies into an agent that can (i) evolve memory when required and (ii) optimize adaptation speed to specific online learning problems. In this paper, we exploit the highly adaptive nature of neuromodulated neural networks to evolve a controller that uses the latent space of an autoencoder in a POMDP. The analysis of the evolved networks reveals the ability of the proposed algorithm to acquire inborn knowledge in a variety of aspects such as the detection of cues that reveal implicit rewards, and the ability to evolve location neurons that help with navigation. The integration of inborn knowledge and online plasticity enabled fast adaptation and better performance in comparison to some non-evolutionary meta-reinforcement learning algorithms. The algorithm proved also to succeed in the 3D gaming environment Malmo Minecraft.Comment: 9 pages. Accepted as a full paper in the Genetic and Evolutionary Computation Conference (GECCO 2020

Evolving unipolar memristor spiking neural networks

© 2015 Taylor & Francis. Neuromorphic computing – brain-like computing in hardware – typically requires myriad complimentary metal oxide semiconductor spiking neurons interconnected by a dense mesh of nanoscale plastic synapses. Memristors are frequently cited as strong synapse candidates due to their statefulness and potential for low-power implementations. To date, plentiful research has focused on the bipolar memristor synapse, which is capable of incremental weight alterations and can provide adaptive self-organisation under a Hebbian learning scheme. In this paper, we consider the unipolar memristor synapse – a device capable of non-Hebbian switching between only two states (conductive and resistive) through application of a suitable input voltage – and discuss its suitability for neuromorphic systems. A self-adaptive evolutionary process is used to autonomously find highly fit network configurations. Experimentation on two robotics tasks shows that unipolar memristor networks evolve task-solving controllers faster than both bipolar memristor networks and networks containing constant non-plastic connections whilst performing at least comparably

The Effect of Proprioceptive Feedback on the Distribution of Sensory Information in a Model of an Undulatory Organism

In an animal, a crucial factor concerning the arrival of information at the sensors and subsequent transmission to the effectors, is how it is distributed. At the same time, higher animals also employ proprioceptive feedback so that their respective neural circuits have information regarding the state of the animal body. In order to disseminate what this practically means for the distribution of sensory information, we have modeled a segmented swimming organism (animat) coevolving its nervous system and body plan morphology. In a simulated aquatic environment, we find that animats artificially endowed with proprioceptive feedback are able to evolve completely decoupled central pattern generators (CPGs) meaning that they emerge without any connections made to neural circuits in adjacent body segments. Without such feedback however, we also find that the distribution of sensory information from the head of the animat becomes far more important, with adjacent CPG circuits becoming interconnected. Crucially, this demonstrates that where proprioceptive mechanisms are lacking, more effective delivery of sensory input is essential

Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks

Biological plastic neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifetime learning. The interplay of these elements leads to the emergence of adaptive behavior and intelligence. Inspired by such intricate natural phenomena, Evolved Plastic Artificial Neural Networks (EPANNs) use simulated evolution in-silico to breed plastic neural networks with a large variety of dynamics, architectures, and plasticity rules: these artificial systems are composed of inputs, outputs, and plastic components that change in response to experiences in an environment. These systems may autonomously discover novel adaptive algorithms, and lead to hypotheses on the emergence of biological adaptation. EPANNs have seen considerable progress over the last two decades. Current scientific and technological advances in artificial neural networks are now setting the conditions for radically new approaches and results. In particular, the limitations of hand-designed networks could be overcome by more flexible and innovative solutions. This paper brings together a variety of inspiring ideas that define the field of EPANNs. The main methods and results are reviewed. Finally, new opportunities and developments are presented

Indirectly Encoding Neural Plasticity as a Pattern of Local Rules

Biological brains can adapt and learn from past experience. In neuroevolution, i.e. evolving artificial neural networks (ANNs), one way that agents controlled by ANNs can evolve the ability to adapt is by encoding local learning rules. However, a significant problem with most such approaches is that local learning rules for every connection in the network must be discovered separately. This paper aims to show that learning rules can be effectively indirectly encoded by extending the Hypercube-based NeuroEvolution of Augmenting Topologies (HyperNEAT) method. Adaptive HyperNEAT is introduced to allow not only patterns of weights across the connectivity of an ANN to be generated by a function of its geometry, but also patterns of arbitrary learning rules. Several such adaptive models with different levels of generality are explored and compared. The long-term promise of the new approach is to evolve large-scale adaptive ANNs, which is a major goal for neuroevolution. © 2010 Springer-Verlag