3,146 research outputs found
Towards safe reinforcement-learning in industrial grid-warehousing
publishedVersionHybri
Advancements in Safe Deep Reinforcement Learning for Real-Time Strategy Games and Industry Applications
publishedVersio
Augmented Modular Reinforcement Learning based on Heterogeneous Knowledge
In order to mitigate some of the inefficiencies of Reinforcement Learning
(RL), modular approaches composing different decision-making policies to derive
agents capable of performing a variety of tasks have been proposed. The modules
at the basis of these architectures are generally reusable, also allowing for
"plug-and-play" integration. However, such solutions still lack the ability to
process and integrate multiple types of information (knowledge), such as rules,
sub-goals, and skills. We propose Augmented Modular Reinforcement Learning
(AMRL) to address these limitations. This new framework uses an arbitrator to
select heterogeneous modules and seamlessly incorporate different types of
knowledge. Additionally, we introduce a variation of the selection mechanism,
namely the Memory-Augmented Arbitrator, which adds the capability of exploiting
temporal information. We evaluate the proposed mechanisms on established as
well as new environments and benchmark them against prominent deep RL
algorithms. Our results demonstrate the performance improvements that can be
achieved by augmenting traditional modular RL with other forms of heterogeneous
knowledge.Comment: 17 pages, 15 figure
Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
Language is often considered a key aspect of human thinking, providing us
with exceptional abilities to generalize, explore, plan, replan, and adapt to
new situations. However, Reinforcement Learning (RL) agents are far from
human-level performance in any of these abilities. We hypothesize one reason
for such cognitive deficiencies is that they lack the benefits of thinking in
language and that we can improve AI agents by training them to think like
humans do. We introduce a novel Imitation Learning framework, Thought Cloning,
where the idea is to not just clone the behaviors of human demonstrators, but
also the thoughts humans have as they perform these behaviors. While we expect
Thought Cloning to truly shine at scale on internet-sized datasets of humans
thinking out loud while acting (e.g. online videos with transcripts), here we
conduct experiments in a domain where the thinking and action data are
synthetically generated. Results reveal that Thought Cloning learns much faster
than Behavioral Cloning and its performance advantage grows the further out of
distribution test tasks are, highlighting its ability to better handle novel
situations. Thought Cloning also provides important benefits for AI Safety and
Interpretability, and makes it easier to debug and improve AI. Because we can
observe the agent's thoughts, we can (1) more easily diagnose why things are
going wrong, making it easier to fix the problem, (2) steer the agent by
correcting its thinking, or (3) prevent it from doing unsafe things it plans to
do. Overall, by training agents how to think as well as behave, Thought Cloning
creates safer, more powerful agents
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
AI systems are increasingly applied to complex tasks that involve interaction
with humans. During training, such systems are potentially dangerous, as they
haven't yet learned to avoid actions that could cause serious harm. How can an
AI system explore and learn without making a single mistake that harms humans
or otherwise causes serious damage? For model-free reinforcement learning,
having a human "in the loop" and ready to intervene is currently the only way
to prevent all catastrophes. We formalize human intervention for RL and show
how to reduce the human labor required by training a supervised learner to
imitate the human's intervention decisions. We evaluate this scheme on Atari
games, with a Deep RL agent being overseen by a human for four hours. When the
class of catastrophes is simple, we are able to prevent all catastrophes
without affecting the agent's learning (whereas an RL baseline fails due to
catastrophic forgetting). However, this scheme is less successful when
catastrophes are more complex: it reduces but does not eliminate catastrophes
and the supervised learner fails on adversarial examples found by the agent.
Extrapolating to more challenging environments, we show that our implementation
would not scale (due to the infeasible amount of human labor required). We
outline extensions of the scheme that are necessary if we are to train
model-free agents without a single catastrophe
- …