Echo state model of non-Markovian reinforcement learning, An

Department Head: Dale H. Grit.2008 Spring.Includes bibliographical references (pages 137-142).There exists a growing need for intelligent, autonomous control strategies that operate in real-world domains. Theoretically the state-action space must exhibit the Markov property in order for reinforcement learning to be applicable. Empirical evidence, however, suggests that reinforcement learning also applies to domains where the state-action space is approximately Markovian, a requirement for the overwhelming majority of real-world domains. These domains, termed non-Markovian reinforcement learning domains, raise a unique set of practical challenges. The reconstruction dimension required to approximate a Markovian state-space is unknown a priori and can potentially be large. Further, spatial complexity of local function approximation of the reinforcement learning domain grows exponentially with the reconstruction dimension. Parameterized dynamic systems alleviate both embedding length and state-space dimensionality concerns by reconstructing an approximate Markovian state-space via a compact, recurrent representation. Yet this representation extracts a cost; modeling reinforcement learning domains via adaptive, parameterized dynamic systems is characterized by instability, slow-convergence, and high computational or spatial training complexity. The objectives of this research are to demonstrate a stable, convergent, accurate, and scalable model of non-Markovian reinforcement learning domains. These objectives are fulfilled via fixed point analysis of the dynamics underlying the reinforcement learning domain and the Echo State Network, a class of parameterized dynamic system. Understanding models of non-Markovian reinforcement learning domains requires understanding the interactions between learning domains and their models. Fixed point analysis of the Mountain Car Problem reinforcement learning domain, for both local and nonlocal function approximations, suggests a close relationship between the locality of the approximation and the number and severity of bifurcations of the fixed point structure. This research suggests the likely cause of this relationship: reinforcement learning domains exist within a dynamic feature space in which trajectories are analogous to states. The fixed point structure maps dynamic space onto state-space. This explanation suggests two testable hypotheses. Reinforcement learning is sensitive to state-space locality because states cluster as trajectories in time rather than space. Second, models using trajectory-based features should exhibit good modeling performance and few changes in fixed point structure. Analysis of performance of lookup table, feedforward neural network, and Echo State Network (ESN) on the Mountain Car Problem reinforcement learning domain confirm these hypotheses. The ESN is a large, sparse, randomly-generated, unadapted recurrent neural network, which adapts a linear projection of the target domain onto the hidden layer. ESN modeling results on reinforcement learning domains show it achieves performance comparable to lookup table and neural network architectures on the Mountain Car Problem with minimal changes to fixed point structure. Also, the ESN achieves lookup table caliber performance when modeling Acrobot, a four-dimensional control problem, but is less successful modeling the lower dimensional Modified Mountain Car Problem. These performance discrepancies are attributed to the ESN’s excellent ability to represent complex short term dynamics, and its inability to consolidate long temporal dependencies into a static memory. Without memory consolidation, reinforcement learning domains exhibiting attractors with multiple dynamic scales are unlikely to be well-modeled via ESN. To mediate this problem, a simple ESN memory consolidation method is presented and tested for stationary dynamic systems. These results indicate the potential to improve modeling performance in reinforcement learning domains via memory consolidation

Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Adaptive and learning-based formation control of swarm robots

Autonomous aerial and wheeled mobile robots play a major role in tasks such as search and rescue, transportation, monitoring, and inspection. However, these operations are faced with a few open challenges including robust autonomy, and adaptive coordination based on the environment and operating conditions, particularly in swarm robots with limited communication and perception capabilities. Furthermore, the computational complexity increases exponentially with the number of robots in the swarm. This thesis examines two different aspects of the formation control problem. On the one hand, we investigate how formation could be performed by swarm robots with limited communication and perception (e.g., Crazyflie nano quadrotor). On the other hand, we explore human-swarm interaction (HSI) and different shared-control mechanisms between human and swarm robots (e.g., BristleBot) for artistic creation. In particular, we combine bio-inspired (i.e., flocking, foraging) techniques with learning-based control strategies (using artificial neural networks) for adaptive control of multi- robots. We first review how learning-based control and networked dynamical systems can be used to assign distributed and decentralized policies to individual robots such that the desired formation emerges from their collective behavior. We proceed by presenting a novel flocking control for UAV swarm using deep reinforcement learning. We formulate the flocking formation problem as a partially observable Markov decision process (POMDP), and consider a leader-follower configuration, where consensus among all UAVs is used to train a shared control policy, and each UAV performs actions based on the local information it collects. In addition, to avoid collision among UAVs and guarantee flocking and navigation, a reward function is added with the global flocking maintenance, mutual reward, and a collision penalty. We adapt deep deterministic policy gradient (DDPG) with centralized training and decentralized execution to obtain the flocking control policy using actor-critic networks and a global state space matrix. In the context of swarm robotics in arts, we investigate how the formation paradigm can serve as an interaction modality for artists to aesthetically utilize swarms. In particular, we explore particle swarm optimization (PSO) and random walk to control the communication between a team of robots with swarming behavior for musical creation

OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond

An introduction to the OpenCog Hyperon framework for Artificiai General Intelligence is presented. Hyperon is a new, mostly from-the-ground-up rewrite/redesign of the OpenCog AGI framework, based on similar conceptual and cognitive principles to the previous OpenCog version, but incorporating a variety of new ideas at the mathematical, software architecture and AI-algorithm level. This review lightly summarizes: 1) some of the history behind OpenCog and Hyperon, 2) the core structures and processes underlying Hyperon as a software system, 3) the integration of this software system with the SingularityNET ecosystem's decentralized infrastructure, 4) the cognitive model(s) being experimentally pursued within Hyperon on the hopeful path to advanced AGI, 5) the prospects seen for advanced aspects like reflective self-modification and self-improvement of the codebase, 6) the tentative development roadmap and various challenges expected to be faced, 7) the thinking of the Hyperon team regarding how to guide this sort of work in a beneficial direction ... and gives links and references for readers who wish to delve further into any of these aspects

Education Reform at the "Edge of Chaos": Constructing ETCH (An Education Theory Complexity Hybrid) for an Optimal Learning Education Environment

EDUCATION REFORM AT THE "EDGE OF CHAOS":CONSTRUCTING ETCH (AN EDUCATION THEORY COMPLEXITY HYBRID) FOR AN OPTIMAL LEARNING EDUCATION ENVIRONMENT AbstractCurrently, the theoretical foundation that inspires educational theory, which in turn shapes the systemic structure of institutions of learning, is based on three key interconnected, interacting underpinnings -mechanism, reductionism, and linearity. My dissertation explores this current theoretical underpinning including its fallacies and inconsistencies, and then frames an alternative educational theoretical base - a hybrid complex adaptive systems theory model for education - that more effectively meets the demands to prepare students for the 21st century. My Education Theory Complexity Hybrid (ETCH) differs by focusing on the systemic, autopoietic nature of schools, the open, fluid processes of school systems as a dissipative structure, and nonlinearity or impossibility of completely predicting the results of any specific intervention within a school system.. In addition, I show how ETCH principles, when applied by educational system leaders, permit them to facilitate an optimal learning environment for a student-centered complex adaptive system.ETCH is derived from Complexity Theory and is a coherent, valid, and verifiable systems' framework that accurately aligns the education system with its goal as a student-centered complex adaptive system. In contrast to most dissertations in the School Leadership Program, which are empirical studies, mine explores this new theoretical orientation and illustrates the power of that orientation through a series of examples taken from my experiences in founding and operating the Lancaster Institute for Learning, a private state-licensed alternative high school in eastern Pennsylvania

Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field

Machines That Learn: Aesthetics of Adaptive Behaviors in Agent-based Art

Since the post-war era, artists have been exploring the use of embodied, artificial agents. This artistic activity runs parallel to research in computer science, in domains such as Cybernetics, Artificial Intelligence and Artificial Life. This thesis offers an account of a particular facet of this broader work — namely, a study of the artistic practice of agent-based, adaptive computational artistic installations that make use of Machine Learning methods. Machine Learning is a sub-field of the computer science area of Artificial Intelligence that employs mathematical models to classify and make predictions based on data or experience rather than on logical rules. These artworks that integrate Machine Learning into their structures raise a number of important questions: (1) What new forms of aesthetic experience do Machine Learning methods enable or make possible when utilized outside of their intended context, and are instead carried over into artistic works? (2) What characterizes the practice of using adaptive computational methods in agent-based artworks? And finally, (3) what kind of worldview are these works fostering? To address these questions, I examine the history of Machine Learning in both art and science, illustrating how artists and engineers alike have made use of these methods historically. I also analyze the defining scientific characteristics of Machine Learning through a practitioner’s lens, concretely articulating how properties of Machine Learning interplay in media artworks that behave and evolve in real time. I later develop a framework for understanding machine behaviors based on the morphological aspects of the temporal unfolding of agent behaviors as a tool for comprehending both adaptive and non-adaptive behaviors in works of art. Finally, I expose how adaptive technologies suggest a new worldview for art that accounts for the performative engagement of agents adapting to one another, which implies a certain way of losing control in the face of the indeterminacy and the unintelligibility of alien agencies and their behaviors