15,581 research outputs found
Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning
Deep Reinforcement Learning (DRL) has been applied successfully to many
robotic applications. However, the large number of trials needed for training
is a key issue. Most of existing techniques developed to improve training
efficiency (e.g. imitation) target on general tasks rather than being tailored
for robot applications, which have their specific context to benefit from. We
propose a novel framework, Assisted Reinforcement Learning, where a classical
controller (e.g. a PID controller) is used as an alternative, switchable policy
to speed up training of DRL for local planning and navigation problems. The
core idea is that the simple control law allows the robot to rapidly learn
sensible primitives, like driving in a straight line, instead of random
exploration. As the actor network becomes more advanced, it can then take over
to perform more complex actions, like obstacle avoidance. Eventually, the
simple controller can be discarded entirely. We show that not only does this
technique train faster, it also is less sensitive to the structure of the DRL
network and consistently outperforms a standard Deep Deterministic Policy
Gradient network. We demonstrate the results in both simulation and real-world
experiments.Comment: Published in ICRA2018. The code is now available at
https://github.com/xie9187/AsDDP
Temporal-Difference Learning to Assist Human Decision Making during the Control of an Artificial Limb
In this work we explore the use of reinforcement learning (RL) to help with
human decision making, combining state-of-the-art RL algorithms with an
application to prosthetics. Managing human-machine interaction is a problem of
considerable scope, and the simplification of human-robot interfaces is
especially important in the domains of biomedical technology and rehabilitation
medicine. For example, amputees who control artificial limbs are often required
to quickly switch between a number of control actions or modes of operation in
order to operate their devices. We suggest that by learning to anticipate
(predict) a user's behaviour, artificial limbs could take on an active role in
a human's control decisions so as to reduce the burden on their users.
Recently, we showed that RL in the form of general value functions (GVFs) could
be used to accurately detect a user's control intent prior to their explicit
control choices. In the present work, we explore the use of temporal-difference
learning and GVFs to predict when users will switch their control influence
between the different motor functions of a robot arm. Experiments were
performed using a multi-function robot arm that was controlled by muscle
signals from a user's body (similar to conventional artificial limb control).
Our approach was able to acquire and maintain forecasts about a user's
switching decisions in real time. It also provides an intuitive and reward-free
way for users to correct or reinforce the decisions made by the machine
learning system. We expect that when a system is certain enough about its
predictions, it can begin to take over switching decisions from the user to
streamline control and potentially decrease the time and effort needed to
complete tasks. This preliminary study therefore suggests a way to naturally
integrate human- and machine-based decision making systems.Comment: 5 pages, 4 figures, This version to appear at The 1st
Multidisciplinary Conference on Reinforcement Learning and Decision Making,
Princeton, NJ, USA, Oct. 25-27, 201
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
Intrinsically motivated spontaneous exploration is a key enabler of
autonomous lifelong learning in human children. It enables the discovery and
acquisition of large repertoires of skills through self-generation,
self-selection, self-ordering and self-experimentation of learning goals. We
present an algorithmic approach called Intrinsically Motivated Goal Exploration
Processes (IMGEP) to enable similar properties of autonomous or self-supervised
learning in machines. The IMGEP algorithmic architecture relies on several
principles: 1) self-generation of goals, generalized as fitness functions; 2)
selection of goals based on intrinsic rewards; 3) exploration with incremental
goal-parameterized policy search and exploitation of the gathered data with a
batch learning algorithm; 4) systematic reuse of information acquired when
targeting a goal for improving towards other goals. We present a particularly
efficient form of IMGEP, called Modular Population-Based IMGEP, that uses a
population-based policy and an object-centered modularity in goals and
mutations. We provide several implementations of this architecture and
demonstrate their ability to automatically generate a learning curriculum
within several experimental setups including a real humanoid robot that can
explore multiple spaces of goals with several hundred continuous dimensions.
While no particular target goal is provided to the system, this curriculum
allows the discovery of skills that act as stepping stone for learning more
complex skills, e.g. nested tool use. We show that learning diverse spaces of
goals with intrinsic motivations is more efficient for learning complex skills
than only trying to directly learn these complex skills
Reinforcement Learning for Racecar Control
This thesis investigates the use of reinforcement learning to learn to drive a racecar in the simulated environment of the Robot Automobile Racing Simulator. Real-life race driving is known to be difficult for humans, and expert human drivers use complex sequences of actions. There are a large number of variables, some of which change stochastically and all of which may affect the outcome. This makes driving a promising domain for testing and developing Machine Learning techniques that have the potential to be robust enough to work in the real world. Therefore the principles of the algorithms from this work may be applicable to a range of problems.
The investigation starts by finding a suitable data structure to represent the information learnt. This is tested using supervised learning. Reinforcement learning is added and roughly tuned, and the supervised learning is then removed. A simple tabular representation is found satisfactory, and this avoids difficulties with more complex methods and allows the investigation to concentrate on the essentials of learning. Various reward sources are tested and a combination of three are found to produce the best performance. Exploration of the problem space is investigated. Results show exploration is essential but controlling how much is done is also important. It turns out the learning episodes need to be very long and because of this the task needs to be treated as continuous by using discounting to limit the size of the variables stored. Eligibility traces are used with success to make the learning more efficient. The tabular representation is made more compact by hashing and more accurate by using smaller buckets. This slows the learning but produces better driving. The improvement given by a rough form of generalisation indicates the replacement of the tabular method by a function approximator is warranted. These results show reinforcement learning can work within the Robot Automobile Racing Simulator, and lay the foundations for building a more efficient and competitive agent
- …