Creating Multi-Level Skill Hierarchies in Reinforcement Learning

What is a useful skill hierarchy for an autonomous agent? We propose an answer based on a graphical representation of how the interaction between an agent and its environment may unfold. Our approach uses modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The result is a collection of skills that operate at varying time scales, organised into a hierarchy, where skills that operate over longer time scales are composed of skills that operate over shorter time scales. The entire skill hierarchy is generated automatically, with no human intervention, including the skills themselves (their behaviour, when they can be called, and when they terminate) as well as the hierarchical dependency structure between them. In a wide range of environments, this approach generates skill hierarchies that are intuitively appealing and that considerably improve the learning performance of the agent

Creating Multi-Level Skill Hierarchies in Reinforcement Learning

What is a useful skill hierarchy for an autonomous agent? We propose an answer based on a graphical representation of how the interaction between an agent and its environment may unfold. Our approach uses modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The result is a collection of skills that operate at varying time scales, organised into a hierarchy, where skills that operate over longer time scales are composed of skills that operate over shorter time scales. The entire skill hierarchy is generated automatically, with no human intervention, including the skills themselves (their behaviour, when they can be called, and when they terminate) as well as the hierarchical dependency structure between them. In a wide range of environments, this approach generates skill hierarchies that are intuitively appealing and that considerably improve the learning performance of the agent

Creating Multi-Level Skill Hierarchies in Reinforcement Learning

What is a useful skill hierarchy for an autonomous agent? We propose an answer based on the graphical structure of an agent's interaction with its environment. Our approach uses hierarchical graph partitioning to expose the structure of the graph at varying timescales, producing a skill hierarchy with multiple levels of abstraction. At each level of the hierarchy, skills move the agent between regions of the state space that are well connected within themselves but weakly connected to each other. We illustrate the utility of the proposed skill hierarchy in a wide variety of domains in the context of reinforcement learning

Statistical Measures for Usage‐Based Linguistics

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/111781/1/lang12119.pd

Learning the Structure of Continuous Markov Decision Processes

There is growing interest in artificial, intelligent agents which can operate autonomously for an extended period of time in complex environments and fulfill a variety of different tasks. Such agents will face different problems during their lifetime which may not be foreseeable at the time of their deployment. Thus, the capacity for lifelong learning of new behaviors is an essential prerequisite for this kind of agents as it enables them to deal with unforeseen situations. However, learning every complex behavior anew from scratch would be cumbersome for the agent. It is more plausible to consider behavior to be modular and let the agent acquire a set of reusable building blocks for behavior, the so-called skills. These skills might, once acquired, facilitate fast learning and adaptation of behavior to new situations. This work focuses on computational approaches for skill acquisition, namely which kind of skills shall be acquired and how to acquire them. The former is commonly denoted as skill discovery and the latter as skill learning . The main contribution of this thesis is a novel incremental skill acquisition approach which is suited for lifelong learning. In this approach, the agent learns incrementally a graph-based representation of a domain and exploits certain properties of this graph such as its bottlenecks for skill discovery. This thesis proposes a novel approach for learning a graph-based representation of continuous domains based on formalizing the problem as a probabilistic generative model. Furthermore, a new incremental agglomerative clustering approach for identifying bottlenecks of such graphs is presented. Thereupon, the thesis proposes a novel intrinsic motivation system which enables an agent to intelligently allocate time between skill discovery and skill learning in developmental settings, where the agent is not constrained by external tasks. The results of this thesis show that the resulting skill acquisition approach is suited for continuous domains and can deal with domain stochasticity and different explorative behavior of the agent. The acquired skills are reusable and versatile and can be used in multi-task and lifelong learning settings in high-dimensional problems

Graph Analysis of EEG Functional Connectivity Networks During a Letter-Speech Sound Binding Task in Adult Dyslexics

We performed an EEG graph analysis on data from 31 typical readers (22.27 ± 2.53 y/o) and 24 dyslexics (22.99 ± 2.29 y/o), recorded while they were engaged in an audiovisual task and during resting-state. The task simulates reading acquisition as participants learned new letter-sound mappings via feedback. EEG data was filtered for the delta (0.5–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), and beta (13–30 Hz) bands. We computed the Phase Lag Index (PLI) to provide an estimate of the functional connectivity between all pairs of electrodes per band. Then, networks were constructed using a Minimum Spanning Tree (MST), a unique sub-graph connecting all nodes (electrodes) without loops, aimed at minimizing bias in between groups and conditions comparisons. Both groups showed a comparable accuracy increase during task blocks, indicating that they correctly learned the new associations. The EEG results revealed lower task-specific theta connectivity, and lower theta degree correlation over both rest and task recordings, indicating less network integration in dyslexics compared to typical readers. This pattern suggests a role of theta oscillations in dyslexia and may reflect differences in task engagement between the groups, although robust correlations between MST metrics and performance indices were lacking