40 research outputs found
Discovering Evolutionary Stepping Stones through Behavior Domination
Behavior domination is proposed as a tool for understanding and harnessing
the power of evolutionary systems to discover and exploit useful stepping
stones. Novelty search has shown promise in overcoming deception by collecting
diverse stepping stones, and several algorithms have been proposed that combine
novelty with a more traditional fitness measure to refocus search and help
novelty search scale to more complex domains. However, combinations of novelty
and fitness do not necessarily preserve the stepping stone discovery that
novelty search affords. In several existing methods, competition between
solutions can lead to an unintended loss of diversity. Behavior domination
defines a class of algorithms that avoid this problem, while inheriting
theoretical guarantees from multiobjective optimization. Several existing
algorithms are shown to be in this class, and a new algorithm is introduced
based on fast non-dominated sorting. Experimental results show that this
algorithm outperforms existing approaches in domains that contain useful
stepping stones, and its advantage is sustained with scale. The conclusion is
that behavior domination can help illuminate the complex dynamics of
behavior-driven search, and can thus lead to the design of more scalable and
robust algorithms.Comment: To Appear in Proceedings of the Genetic and Evolutionary Computation
Conference (GECCO 2017
Multi-criteria Evolution of Neural Network Topologies: Balancing Experience and Performance in Autonomous Systems
Majority of Artificial Neural Network (ANN) implementations in autonomous
systems use a fixed/user-prescribed network topology, leading to sub-optimal
performance and low portability. The existing neuro-evolution of augmenting
topology or NEAT paradigm offers a powerful alternative by allowing the network
topology and the connection weights to be simultaneously optimized through an
evolutionary process. However, most NEAT implementations allow the
consideration of only a single objective. There also persists the question of
how to tractably introduce topological diversification that mitigates
overfitting to training scenarios. To address these gaps, this paper develops a
multi-objective neuro-evolution algorithm. While adopting the basic elements of
NEAT, important modifications are made to the selection, speciation, and
mutation processes. With the backdrop of small-robot path-planning
applications, an experience-gain criterion is derived to encapsulate the amount
of diverse local environment encountered by the system. This criterion
facilitates the evolution of genes that support exploration, thereby seeking to
generalize from a smaller set of mission scenarios than possible with
performance maximization alone. The effectiveness of the single-objective
(optimizing performance) and the multi-objective (optimizing performance and
experience-gain) neuro-evolution approaches are evaluated on two different
small-robot cases, with ANNs obtained by the multi-objective optimization
observed to provide superior performance in unseen scenarios
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
Evolution strategies (ES) are a family of black-box optimization algorithms
able to train deep neural networks roughly as well as Q-learning and policy
gradient methods on challenging deep reinforcement learning (RL) problems, but
are much faster (e.g. hours vs. days) because they parallelize better. However,
many RL problems require directed exploration because they have reward
functions that are sparse or deceptive (i.e. contain local optima), and it is
unknown how to encourage such exploration with ES. Here we show that algorithms
that have been invented to promote directed exploration in small-scale evolved
neural networks via populations of exploring agents, specifically novelty
search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to
improve its performance on sparse or deceptive deep RL tasks, while retaining
scalability. Our experiments confirm that the resultant new algorithms, NS-ES
and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES
to achieve higher performance on Atari and simulated robots learning to walk
around a deceptive trap. This paper thus introduces a family of fast, scalable
algorithms for reinforcement learning that are capable of directed exploration.
It also adds this new family of exploration algorithms to the RL toolbox and
raises the interesting possibility that analogous algorithms with multiple
simultaneous paths of exploration might also combine well with existing RL
algorithms outside ES
Evolution of Swarm Robotics Systems with Novelty Search
Novelty search is a recent artificial evolution technique that challenges
traditional evolutionary approaches. In novelty search, solutions are rewarded
based on their novelty, rather than their quality with respect to a predefined
objective. The lack of a predefined objective precludes premature convergence
caused by a deceptive fitness function. In this paper, we apply novelty search
combined with NEAT to the evolution of neural controllers for homogeneous
swarms of robots. Our empirical study is conducted in simulation, and we use a
common swarm robotics task - aggregation, and a more challenging task - sharing
of an energy recharging station. Our results show that novelty search is
unaffected by deception, is notably effective in bootstrapping the evolution,
can find solutions with lower complexity than fitness-based evolution, and can
find a broad diversity of solutions for the same task. Even in non-deceptive
setups, novelty search achieves solution qualities similar to those obtained in
traditional fitness-based evolution. Our study also encompasses variants of
novelty search that work in concert with fitness-based evolution to combine the
exploratory character of novelty search with the exploitatory character of
objective-based evolution. We show that these variants can further improve the
performance of novelty search. Overall, our study shows that novelty search is
a promising alternative for the evolution of controllers for robotic swarms.Comment: To appear in Swarm Intelligence (2013), ANTS Special Issue. The final
publication will be available at link.springer.co
Enhanced Optimization with Composite Objectives and Novelty Selection
An important benefit of multi-objective search is that it maintains a diverse
population of candidates, which helps in deceptive problems in particular. Not
all diversity is useful, however: candidates that optimize only one objective
while ignoring others are rarely helpful. This paper proposes a solution: The
original objectives are replaced by their linear combinations, thus focusing
the search on the most useful tradeoffs between objectives. To compensate for
the loss of diversity, this transformation is accompanied by a selection
mechanism that favors novelty. In the highly deceptive problem of discovering
minimal sorting networks, this approach finds better solutions, and finds them
faster and more consistently than standard methods. It is therefore a promising
approach to solving deceptive problems through multi-objective optimization.Comment: 7 page
Fitness and novelty in evolutionary art
In this paper the effects of introducing novelty search in evolutionary art are explored. Our algorithm combines fitness and novelty metrics to frame image evolution as a multi-objective optimisation problem, promoting the creation of images that are both suitable and diverse. The method is illustrated by using two evolutionary art engines for the evolution of figurative objects and context free design grammars. The results demonstrate the ability of the algorithm to obtain a larger set of fit images compared to traditional fitness-based evolution, regardless of the engine used