Evolutionary Algorithms for Reinforcement Learning

There are two distinct approaches to solving reinforcement learning problems, namely, searching in value function space and searching in policy space. Temporal difference methods and evolutionary algorithms are well-known examples of these approaches. Kaelbling, Littman and Moore recently provided an informative survey of temporal difference methods. This article focuses on the application of evolutionary algorithms to the reinforcement learning problem, emphasizing alternative policy representations, credit assignment methods, and problem-specific genetic operators. Strengths and weaknesses of the evolutionary approach to reinforcement learning are presented, along with a survey of representative applications

Towards adaptive multi-robot systems: self-organization and self-adaptation

Dieser Beitrag ist mit Zustimmung des Rechteinhabers aufgrund einer (DFG geförderten) Allianz- bzw. Nationallizenz frei zugänglich.This publication is with permission of the rights owner freely accessible due to an Alliance licence and a national licence (funded by the DFG, German Research Foundation) respectively.The development of complex systems ensembles that operate in uncertain environments is a major challenge. The reason for this is that system designers are not able to fully specify the system during specification and development and before it is being deployed. Natural swarm systems enjoy similar characteristics, yet, being self-adaptive and being able to self-organize, these systems show beneficial emergent behaviour. Similar concepts can be extremely helpful for artificial systems, especially when it comes to multi-robot scenarios, which require such solution in order to be applicable to highly uncertain real world application. In this article, we present a comprehensive overview over state-of-the-art solutions in emergent systems, self-organization, self-adaptation, and robotics. We discuss these approaches in the light of a framework for multi-robot systems and identify similarities, differences missing links and open gaps that have to be addressed in order to make this framework possible

Degeneracy: a link between evolvability, robustness and complexity in biological systems

A full accounting of biological robustness remains elusive; both in terms of the mechanisms by which robustness is achieved and the forces that have caused robustness to grow over evolutionary time. Although its importance to topics such as ecosystem services and resilience is well recognized, the broader relationship between robustness and evolution is only starting to be fully appreciated. A renewed interest in this relationship has been prompted by evidence that mutational robustness can play a positive role in the discovery of adaptive innovations (evolvability) and evidence of an intimate relationship between robustness and complexity in biology. This paper offers a new perspective on the mechanics of evolution and the origins of complexity, robustness, and evolvability. Here we explore the hypothesis that degeneracy, a partial overlap in the functioning of multi-functional components, plays a central role in the evolution and robustness of complex forms. In support of this hypothesis, we present evidence that degeneracy is a fundamental source of robustness, it is intimately tied to multi-scaled complexity, and it establishes conditions that are necessary for system evolvability

Dynamic Models of Appraisal Networks Explaining Collective Learning

This paper proposes models of learning process in teams of individuals who collectively execute a sequence of tasks and whose actions are determined by individual skill levels and networks of interpersonal appraisals and influence. The closely-related proposed models have increasing complexity, starting with a centralized manager-based assignment and learning model, and finishing with a social model of interpersonal appraisal, assignments, learning, and influences. We show how rational optimal behavior arises along the task sequence for each model, and discuss conditions of suboptimality. Our models are grounded in replicator dynamics from evolutionary games, influence networks from mathematical sociology, and transactive memory systems from organization science.Comment: A preliminary version has been accepted by the 53rd IEEE Conference on Decision and Control. The journal version has been submitted to IEEE Transactions on Automatic Contro

Embodied Evolution in Collective Robotics: A Review

This paper provides an overview of evolutionary robotics techniques applied to on-line distributed evolution for robot collectives -- namely, embodied evolution. It provides a definition of embodied evolution as well as a thorough description of the underlying concepts and mechanisms. The paper also presents a comprehensive summary of research published in the field since its inception (1999-2017), providing various perspectives to identify the major trends. In particular, we identify a shift from considering embodied evolution as a parallel search method within small robot collectives (fewer than 10 robots) to embodied evolution as an on-line distributed learning method for designing collective behaviours in swarm-like collectives. The paper concludes with a discussion of applications and open questions, providing a milestone for past and an inspiration for future research.Comment: 23 pages, 1 figure, 1 tabl

Applications of Biological Cell Models in Robotics

In this paper I present some of the most representative biological models applied to robotics. In particular, this work represents a survey of some models inspired, or making use of concepts, by gene regulatory networks (GRNs): these networks describe the complex interactions that affect gene expression and, consequently, cell behaviour

MULTI AGENT-BASED ENVIRONMENTAL LANDSCAPE (MABEL) - AN ARTIFICIAL INTELLIGENCE SIMULATION MODEL: SOME EARLY ASSESSMENTS

The Multi Agent-Based Environmental Landscape model (MABEL) introduces a Distributed Artificial Intelligence (DAI) systemic methodology, to simulate land use and transformation changes over time and space. Computational agents represent abstract relations among geographic, environmental, human and socio-economic variables, with respect to land transformation pattern changes. A multi-agent environment is developed providing task-nonspecific problem-solving abilities, flexibility on achieving goals and representing existing relations observed in real-world scenarios, and goal-based efficiency. Intelligent MABEL agents acquire spatial expressions and perform specific tasks demonstrating autonomy, environmental interactions, communication and cooperation, reactivity and proactivity, reasoning and learning capabilities. Their decisions maximize both task-specific marginal utility for their actions and joint, weighted marginal utility for their time-stepping. Agent behavior is achieved by personalizing a dynamic utility-based knowledge base through sequential GIS filtering, probability-distributed weighting, joint probability Bayesian correlational weighting, and goal-based distributional properties, applied to socio-economic and behavioral criteria. First-order logics, heuristics and appropriation of time-step sequences employed, provide a simulation-able environment, capable of re-generating space-time evolution of the agents.Environmental Economics and Policy,

Adaptive Network Dynamics and Evolution of Leadership in Collective Migration

The evolution of leadership in migratory populations depends not only on costs and benefits of leadership investments but also on the opportunities for individuals to rely on cues from others through social interactions. We derive an analytically tractable adaptive dynamic network model of collective migration with fast timescale migration dynamics and slow timescale adaptive dynamics of individual leadership investment and social interaction. For large populations, our analysis of bifurcations with respect to investment cost explains the observed hysteretic effect associated with recovery of migration in fragmented environments. Further, we show a minimum connectivity threshold above which there is evolutionary branching into leader and follower populations. For small populations, we show how the topology of the underlying social interaction network influences the emergence and location of leaders in the adaptive system. Our model and analysis can describe other adaptive network dynamics involving collective tracking or collective learning of a noisy, unknown signal, and likewise can inform the design of robotic networks where agents use decentralized strategies that balance direct environmental measurements with agent interactions.Comment: Submitted to Physica D: Nonlinear Phenomen