1,430 research outputs found
Recommended from our members
Curriculum learning in reinforcement learning
In recent years, reinforcement learning (RL) has been increasingly successful at solving complex tasks. Despite these successes, one of the fundamental challenges is that many RL methods require large amounts of experience, and thus can be slow to train in practice. Transfer learning is a recent area of research that has been shown to speed up learning on a complex task by transferring knowledge from one or more easier source tasks. Most existing transfer learning methods treat this transfer of knowledge as a one-step process, where knowledge from all the sources are directly transferred to the target. However, for complex tasks, it may be more beneficial (and even necessary) to gradually acquire skills over multiple tasks in sequence, where each subsequent task requires and builds upon knowledge gained in a previous task. This idea is pervasive throughout human learning, where people learn complex skills gradually by training via a curriculum.
The goal of this thesis is to explore whether autonomous reinforcement learning agents can also benefit by training via a curriculum, and whether such curricula can be designed fully autonomously. In order to answer these questions, this thesis first formalizes the concept of a curriculum, and the methodology of curriculum learning in reinforcement learning. Curriculum learning consists of 3 main elements: 1) task generation, which creates a suitable set of source tasks; 2) sequencing, which focuses on how to order these tasks into a curriculum; and 3) transfer learning, which considers how to transfer knowledge between tasks in the curriculum. This thesis introduces several methods to both create suitable source tasks and automatically sequence them into a curriculum. We show that these methods produce curricula that are tailored to the individual sensing and action capabilities of different agents, and show how the curricula learned can be adapted for new, but related target tasks. Together, these methods form the components of an autonomous curriculum design agent, that can suggest a training curriculum customized to both the unique abilities of each agent and the task in question. We expect this research on the curriculum learning approach will increase the applicability and scalability of RL methods by providing a faster way of training reinforcement learning agents, compared to learning tabula rasa.Computer Science
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning
A lot of efforts have been devoted to investigating how agents can learn
effectively and achieve coordination in multiagent systems. However, it is
still challenging in large-scale multiagent settings due to the complex
dynamics between the environment and agents and the explosion of state-action
space. In this paper, we design a novel Dynamic Multiagent Curriculum Learning
(DyMA-CL) to solve large-scale problems by starting from learning on a
multiagent scenario with a small size and progressively increasing the number
of agents. We propose three transfer mechanisms across curricula to accelerate
the learning process. Moreover, due to the fact that the state dimension varies
across curricula,, and existing network structures cannot be applied in such a
transfer setting since their network input sizes are fixed. Therefore, we
design a novel network structure called Dynamic Agent-number Network (DyAN) to
handle the dynamic size of the network input. Experimental results show that
DyMA-CL using DyAN greatly improves the performance of large-scale multiagent
learning compared with state-of-the-art deep reinforcement learning approaches.
We also investigate the influence of three transfer mechanisms across curricula
through extensive simulations.Comment: Accepted by AAAI202
Automaton-Guided Curriculum Generation for Reinforcement Learning Agents
Despite advances in Reinforcement Learning, many sequential decision making
tasks remain prohibitively expensive and impractical to learn. Recently,
approaches that automatically generate reward functions from logical task
specifications have been proposed to mitigate this issue; however, they scale
poorly on long-horizon tasks (i.e., tasks where the agent needs to perform a
series of correct actions to reach the goal state, considering future
transitions while choosing an action). Employing a curriculum (a sequence of
increasingly complex tasks) further improves the learning speed of the agent by
sequencing intermediate tasks suited to the learning capacity of the agent.
However, generating curricula from the logical specification still remains an
unsolved problem. To this end, we propose AGCL, Automaton-guided Curriculum
Learning, a novel method for automatically generating curricula for the target
task in the form of Directed Acyclic Graphs (DAGs). AGCL encodes the
specification in the form of a deterministic finite automaton (DFA), and then
uses the DFA along with the Object-Oriented MDP (OOMDP) representation to
generate a curriculum as a DAG, where the vertices correspond to tasks, and
edges correspond to the direction of knowledge transfer. Experiments in
gridworld and physics-based simulated robotics domains show that the curricula
produced by AGCL achieve improved time-to-threshold performance on a complex
sequential decision-making problem relative to state-of-the-art curriculum
learning (e.g, teacher-student, self-play) and automaton-guided reinforcement
learning baselines (e.g, Q-Learning for Reward Machines). Further, we
demonstrate that AGCL performs well even in the presence of noise in the task's
OOMDP description, and also when distractor objects are present that are not
modeled in the logical specification of the tasks' objectives.Comment: To be presented at The International Conference on Automated Planning
and Scheduling (ICAPS) 202
- …