Learning Representations in Model-Free Hierarchical Reinforcement Learning

Common approaches to Reinforcement Learning (RL) are seriously challenged by large-scale applications involving huge state spaces and sparse delayed reward feedback. Hierarchical Reinforcement Learning (HRL) methods attempt to address this scalability issue by learning action selection policies at multiple levels of temporal abstraction. Abstraction can be had by identifying a relatively small set of states that are likely to be useful as subgoals, in concert with the learning of corresponding skill policies to achieve those subgoals. Many approaches to subgoal discovery in HRL depend on the analysis of a model of the environment, but the need to learn such a model introduces its own problems of scale. Once subgoals are identified, skills may be learned through intrinsic motivation, introducing an internal reward signal marking subgoal attainment. In this paper, we present a novel model-free method for subgoal discovery using incremental unsupervised learning over a small memory of the most recent experiences (trajectories) of the agent. When combined with an intrinsic motivation learning mechanism, this method learns both subgoals and skills, based on experiences in the environment. Thus, we offer an original approach to HRL that does not require the acquisition of a model of the environment, suitable for large-scale applications. We demonstrate the efficiency of our method on two RL problems with sparse delayed feedback: a variant of the rooms environment and the first screen of the ATARI 2600 Montezuma's Revenge game

Balancing Selection Pressures, Multiple Objectives, and Neural Modularity to Coevolve Cooperative Agent Behavior

Previous research using evolutionary computation in Multi-Agent Systems indicates that assigning fitness based on team vs.\ individual behavior has a strong impact on the ability of evolved teams of artificial agents to exhibit teamwork in challenging tasks. However, such research only made use of single-objective evolution. In contrast, when a multiobjective evolutionary algorithm is used, populations can be subject to individual-level objectives, team-level objectives, or combinations of the two. This paper explores the performance of cooperatively coevolved teams of agents controlled by artificial neural networks subject to these types of objectives. Specifically, predator agents are evolved to capture scripted prey agents in a torus-shaped grid world. Because of the tension between individual and team behaviors, multiple modes of behavior can be useful, and thus the effect of modular neural networks is also explored. Results demonstrate that fitness rewarding individual behavior is superior to fitness rewarding team behavior, despite being applied to a cooperative task. However, the use of networks with multiple modules allows predators to discover intelligent behavior, regardless of which type of objectives are used

Computing data for Levin-Wen with defects

We demonstrate how to do many computations for non-chiral topological phases with defects. These defects may be 1-dimensional domain walls or 0-dimensional point defects. Using $\operatorname{Vec}(S_3)$ as a guiding example, we demonstrate how domain wall fusion and associators can be computed using generalized tube algebra techniques. These domain walls can be both between distinct or identical phases. Additionally, we show how to compute all possible point defects, and the fusion and associator data of these. Worked examples, tabulated data and Mathematica code are provided.Comment: 17+25 pages, many tables and attached cod

Anomalies and entanglement renormalization

We study 't Hooft anomalies of discrete groups in the framework of (1+1)-dimensional multiscale entanglement renormalization ansatz states on the lattice. Using matrix product operators, general topological restrictions on conformal data are derived. An ansatz class allowing for optimization of MERA with an anomalous symmetry is introduced. We utilize this class to numerically study a family of Hamiltonians with a symmetric critical line. Conformal data is obtained for all irreducible projective representations of each anomalous symmetry twist, corresponding to definite topological sectors. It is numerically demonstrated that this line is a protected gapless phase. Finally, we implement a duality transformation between a pair of critical lines using our subclass of MERA.Comment: 12+18 pages, 6+5 figures, 0+2 tables, v2 published versio

Fusing Binary Interface Defects in Topological Phases: The Vec⁡(Z/pZ)\operatorname{Vec}(\mathbb{Z}/p\mathbb{Z}) case

A binary interface defect is any interface between two (not necessarily invertible) domain walls. We compute all possible binary interface defects in Kitaev's $\mathbb{Z}/p\mathbb{Z}$ model and all possible fusions between them. Our methods can be applied to any Levin-Wen model. We also give physical interpretations for each of the defects in the $\mathbb{Z}/p\mathbb{Z}$ model. These physical interpretations provide a new graphical calculus which can be used to compute defect fusion.Comment: 27+10 pages, 2+5 tables, comments welcom

Tensor Networks with a Twist: Anyon-permuting domain walls and defects in PEPS

We study the realization of anyon-permuting symmetries of topological phases on the lattice using tensor networks. Working on the virtual level of a projected entangled pair state, we find matrix product operators (MPOs) that realize all unitary topological symmetries for the toric and color codes. These operators act as domain walls that enact the symmetry transformation on anyons as they cross. By considering open boundary conditions for these domain wall MPOs, we show how to introduce symmetry twists and defect lines into the state.Comment: 11 pages, 6 figures, 2 appendices, v2 published versio