876 research outputs found
Self-adaptive exploration in evolutionary search
We address a primary question of computational as well as biological research
on evolution: How can an exploration strategy adapt in such a way as to exploit
the information gained about the problem at hand? We first introduce an
integrated formalism of evolutionary search which provides a unified view on
different specific approaches. On this basis we discuss the implications of
indirect modeling (via a ``genotype-phenotype mapping'') on the exploration
strategy. Notions such as modularity, pleiotropy and functional phenotypic
complex are discussed as implications. Then, rigorously reflecting the notion
of self-adaptability, we introduce a new definition that captures
self-adaptability of exploration: different genotypes that map to the same
phenotype may represent (also topologically) different exploration strategies;
self-adaptability requires a variation of exploration strategies along such a
``neutral space''. By this definition, the concept of neutrality becomes a
central concern of this paper. Finally, we present examples of these concepts:
For a specific grammar-type encoding, we observe a large variability of
exploration strategies for a fixed phenotype, and a self-adaptive drift towards
short representations with highly structured exploration strategy that matches
the ``problem's structure''.Comment: 24 pages, 5 figure
How the Morphology Encoding Influences the Learning Ability in Body-Brain Co-Optimization
Embedding the learning of controllers within the evolution of morphologies has emerged as an effective strategy for the co-optimization of agents' bodies and brains. Intuitively, that is how nature shaped animal life on Earth. Still, the design of such co-optimization is a complex endeavor; one issue is the choice of the genetic encoding for the morphology. Such choice can be crucial for the effectiveness of learning, i.e., how fast and to what degree agents adapt, through learning, during their life. Here we evolve the morphologies of voxel-based soft agents with two different encodings, direct and indirect while learning the controllers with reinforcement learning. We experiment with three tasks, ranging from cave crawling to beam toppling, and study how the encoding influences the learning outcome. Our results show that the direct encoding corresponds to increased ability to learn, mostly in terms of learning speed. The same is not always true for the indirect one. We link these results to different shades of the Baldwin effect, consisting of morphologies being selected for increasing an agent’s ability to learn during its lifetime
Towards the Evolution of Multi-Layered Neural Networks: A Dynamic Structured Grammatical Evolution Approach
Current grammar-based NeuroEvolution approaches have several shortcomings. On
the one hand, they do not allow the generation of Artificial Neural Networks
(ANNs composed of more than one hidden-layer. On the other, there is no way to
evolve networks with more than one output neuron. To properly evolve ANNs with
more than one hidden-layer and multiple output nodes there is the need to know
the number of neurons available in previous layers. In this paper we introduce
Dynamic Structured Grammatical Evolution (DSGE): a new genotypic representation
that overcomes the aforementioned limitations. By enabling the creation of
dynamic rules that specify the connection possibilities of each neuron, the
methodology enables the evolution of multi-layered ANNs with more than one
output neuron. Results in different classification problems show that DSGE
evolves effective single and multi-layered ANNs, with a varying number of
output neurons
Evolutionary Reinforcement Learning: A Survey
Reinforcement learning (RL) is a machine learning approach that trains agents
to maximize cumulative rewards through interactions with environments. The
integration of RL with deep learning has recently resulted in impressive
achievements in a wide range of challenging tasks, including board games,
arcade games, and robot control. Despite these successes, there remain several
crucial challenges, including brittle convergence properties caused by
sensitive hyperparameters, difficulties in temporal credit assignment with long
time horizons and sparse rewards, a lack of diverse exploration, especially in
continuous search space scenarios, difficulties in credit assignment in
multi-agent reinforcement learning, and conflicting objectives for rewards.
Evolutionary computation (EC), which maintains a population of learning agents,
has demonstrated promising performance in addressing these limitations. This
article presents a comprehensive survey of state-of-the-art methods for
integrating EC into RL, referred to as evolutionary reinforcement learning
(EvoRL). We categorize EvoRL methods according to key research fields in RL,
including hyperparameter optimization, policy search, exploration, reward
shaping, meta-RL, and multi-objective RL. We then discuss future research
directions in terms of efficient methods, benchmarks, and scalable platforms.
This survey serves as a resource for researchers and practitioners interested
in the field of EvoRL, highlighting the important challenges and opportunities
for future research. With the help of this survey, researchers and
practitioners can develop more efficient methods and tailored benchmarks for
EvoRL, further advancing this promising cross-disciplinary research field
Learning From Geometry In Learning For Tactical And Strategic Decision Domains
Artificial neural networks (ANNs) are an abstraction of the low-level architecture of biological brains that are often applied in general problem solving and function approximation. Neuroevolution (NE), i.e. the evolution of ANNs, has proven effective at solving problems in a variety of domains. Information from the domain is input to the ANN, which outputs its desired actions. This dissertation presents a new NE algorithm called Hypercube-based NeuroEvolution of Augmenting Topologies (HyperNEAT), based on a novel indirect encoding of ANNs. The key insight in HyperNEAT is to make the algorithm aware of the geometry in which the ANNs are embedded and thereby exploit such domain geometry to evolve ANNs more effectively. The dissertation focuses on applying HyperNEAT to tactical and strategic decision domains. These domains involve simultaneously considering short-term tactics while also balancing long-term strategies. Board games such as checkers and Go are canonical examples of such domains; however, they also include real-time strategy games and military scenarios. The dissertation details three proposed extensions to HyperNEAT designed to work in tactical and strategic decision domains. The first is an action selector ANN architecture that allows the ANN to indicate its judgements on every possible action all at once. The second technique is called substrate extrapolation. It allows learning basic concepts at a low resolution, and then increasing the resolution to learn more advanced concepts. The iii final extension is geometric game-tree pruning, whereby HyperNEAT can endow the ANN the ability to focus on specific areas of a domain (such as a checkers board) that deserve more inspection. The culminating contribution is to demonstrate the ability of HyperNEAT with these extensions to play Go, a most challenging game for artificial intelligence, by combining HyperNEAT with UC
Generating walking behaviours in legged robots
Many legged robots have boon built with a variety of different abilities, from running
to liopping to climbing stairs. Despite this however, there has been no consistency of
approach to the problem of getting them to walk. Approaches have included breaking
down the walking step into discrete parts and then controlling them separately, using
springs and linkages to achieve a passive walking cycle, and even working out the
necessary movements in simulation and then imposing them on the real robot. All of
these have limitations, although most were successful at the task for which they were
designed. However, all of them fall into one of two categories: either they alter the
dynamics of the robots physically so that the robot, whilst very good at walking, is
not as general purpose as it once was (as with the passive robots), or they control the
physical mechanism of the robot directly to achieve their goals, and this is a difficult
task.In this thesis a design methodology is described for building controllers for 3D dynam¬
ically stable walking, inspired by the best walkers and runners around — ourselves —
so the controllers produced are based 011 the vertebrate Central Nervous System. This
means that there is a low-level controller which adapts itself to the robot so that, when
switched on, it can be considered to simulate the springs and linkages of the passive
robots to produce a walking robot, and this now active mechanism is then controlled
by a relatively simple higher level controller. This is the best of both worlds — we
have a robot which is inherently capable of walking, and thus is easy to control like
the passive walkers, but also retains the general purpose abilities which makes it so
potentially useful.This design methodology uses an evolutionary algorithm to generate low-level control¬
lers for a selection of simulated legged robots. The thesis also looks in detail at previous
walking robots and their controllers and shows that some approaches, including staged
evolution and hand-coding designs, may be unnecessary, and indeed inappropriate, at
least for a general purpose controller. The specific algorithm used is evolutionary, using
a simple genetic algorithm to allow adaptation to different robot configurations, and
the controllers evolved are continuous time neural networks. These are chosen because
of their ability to entrain to the movement of the robot, allowing the whole robot and
network to be considered as a single dynamical system, which can then be controlled
by a higher level system.An extensive program of experiments investigates the types of neural models and net¬
work structures which are best suited to this task, and it is shown that stateless and
simple dynamic neural models are significantly outperformed as controllers by more
complex, biologically plausible ones but that other ideas taken from biological systems,
including network connectivities, are not generally as useful and reasons for this are
examined.The thesis then shows that this system, although only developed 011 a single robot,
is capable of automatically generating controllers for a wide selection of different test
designs. Finally it shows that high level controllers, at least to control steering and
speed, can be easily built 011 top of this now active walking mechanism
Neuroevolution in Games: State of the Art and Open Challenges
This paper surveys research on applying neuroevolution (NE) to games. In
neuroevolution, artificial neural networks are trained through evolutionary
algorithms, taking inspiration from the way biological brains evolved. We
analyse the application of NE in games along five different axes, which are the
role NE is chosen to play in a game, the different types of neural networks
used, the way these networks are evolved, how the fitness is determined and
what type of input the network receives. The article also highlights important
open research challenges in the field.Comment: - Added more references - Corrected typos - Added an overview table
(Table 1
Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks
Biological plastic neural networks are systems of extraordinary computational
capabilities shaped by evolution, development, and lifetime learning. The
interplay of these elements leads to the emergence of adaptive behavior and
intelligence. Inspired by such intricate natural phenomena, Evolved Plastic
Artificial Neural Networks (EPANNs) use simulated evolution in-silico to breed
plastic neural networks with a large variety of dynamics, architectures, and
plasticity rules: these artificial systems are composed of inputs, outputs, and
plastic components that change in response to experiences in an environment.
These systems may autonomously discover novel adaptive algorithms, and lead to
hypotheses on the emergence of biological adaptation. EPANNs have seen
considerable progress over the last two decades. Current scientific and
technological advances in artificial neural networks are now setting the
conditions for radically new approaches and results. In particular, the
limitations of hand-designed networks could be overcome by more flexible and
innovative solutions. This paper brings together a variety of inspiring ideas
that define the field of EPANNs. The main methods and results are reviewed.
Finally, new opportunities and developments are presented
Search-based procedural content generation
Recently, a small number of papers have appeared in which the authors implement stochastic search algorithms, such as evolutionary computation, to generate game content, such as levels, rules and weapons. We propose a taxonomy of such approaches, centring on what sort of content is generated, how the content is represented, and how the quality of the content is evaluated. The relation between search-based and other types of procedural content generation is described, as are some of the main research challenges in this new field. The paper ends with some successful examples of this approach.peer-reviewe
- …