59,412 research outputs found
Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback
Exploration and reward specification are fundamental and intertwined
challenges for reinforcement learning. Solving sequential decision-making tasks
requiring expansive exploration requires either careful design of reward
functions or the use of novelty-seeking exploration bonuses. Human supervisors
can provide effective guidance in the loop to direct the exploration process,
but prior methods to leverage this guidance require constant synchronous
high-quality human feedback, which is expensive and impractical to obtain. In
this work, we present a technique called Human Guided Exploration (HuGE), which
uses low-quality feedback from non-expert users that may be sporadic,
asynchronous, and noisy. HuGE guides exploration for reinforcement learning not
only in simulation but also in the real world, all without meticulous reward
specification. The key concept involves bifurcating human feedback and policy
learning: human feedback steers exploration, while self-supervised learning
from the exploration data yields unbiased policies. This procedure can leverage
noisy, asynchronous human feedback to learn policies with no hand-crafted
reward design or exploration bonuses. HuGE is able to learn a variety of
challenging multi-stage robotic navigation and manipulation tasks in simulation
using crowdsourced feedback from non-expert users. Moreover, this paradigm can
be scaled to learning directly on real-world robots, using occasional,
asynchronous feedback from human supervisors
DOP: Deep Optimistic Planning with Approximate Value Function Evaluation
Research on reinforcement learning has demonstrated promising results in manifold applications and domains. Still, efficiently learning effective robot behaviors is very difficult, due to unstructured scenarios, high uncertainties, and large state dimensionality (e.g. multi-agent systems or hyper-redundant robots). To alleviate this problem, we present DOP, a deep model-based reinforcement learning algorithm, which exploits action values to both (1) guide the exploration of the state space and (2) plan effective policies. Specifically, we exploit deep neural networks to learn Q-functions that are used to attack the curse of dimensionality during a Monte-Carlo tree search. Our algorithm, in fact, constructs upper confidence bounds on the learned value function to select actions optimistically. We implement and evaluate DOP on different scenarios: (1) a cooperative navigation problem, (2) a fetching task for a 7-DOF KUKA robot, and (3) a human-robot handover with a humanoid robot (both in simulation and real). The obtained results show the effectiveness of DOP in the chosen applications, where action values drive the exploration and reduce the computational demand of the planning process while achieving good performance
- …