3 research outputs found
Hierarchical Reinforcement Learning with Abductive Planning
One of the key challenges in applying reinforcement learning to real-life
problems is that the amount of train-and-error required to learn a good policy
increases drastically as the task becomes complex. One potential solution to
this problem is to combine reinforcement learning with automated symbol
planning and utilize prior knowledge on the domain. However, existing methods
have limitations in their applicability and expressiveness. In this paper we
propose a hierarchical reinforcement learning method based on abductive
symbolic planning. The planner can deal with user-defined evaluation functions
and is not based on the Herbrand theorem. Therefore it can utilize prior
knowledge of the rewards and can work in a domain where the state space is
unknown. We demonstrate empirically that our architecture significantly
improves learning efficiency with respect to the amount of training examples on
the evaluation domain, in which the state space is unknown and there exist
multiple goals.Comment: 7 pages, 6 figures, ICML/IJCAI/AAMAS 2018 Workshop on Planning and
Learning (PAL-18
From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
Reinforcement learning is an appropriate and successful method to robustly
perform low-level robot control under noisy conditions. Symbolic action
planning is useful to resolve causal dependencies and to break a causally
complex problem down into a sequence of simpler high-level actions. A problem
with the integration of both approaches is that action planning is based on
discrete high-level action- and state spaces, whereas reinforcement learning is
usually driven by a continuous reward function. However, recent advances in
reinforcement learning, specifically, universal value function approximators
and hindsight experience replay, have focused on goal-independent methods based
on sparse rewards. In this article, we build on these novel methods to
facilitate the integration of action planning with reinforcement learning by
exploiting the reward-sparsity as a bridge between the high-level and low-level
state- and control spaces. As a result, we demonstrate that the integrated
neuro-symbolic method is able to solve object manipulation problems that
involve tool use and non-trivial causal dependencies under noisy conditions,
exploiting both data and knowledge
Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming
Recently deep reinforcement learning has achieved tremendous success in wide
ranges of applications. However, it notoriously lacks data-efficiency and
interpretability. Data-efficiency is important as interacting with the
environment is expensive. Further, interpretability can increase the
transparency of the black-box-style deep RL models and hence gain trust from
the users. In this work, we propose a new hierarchical framework via symbolic
RL, leveraging a symbolic transition model to improve the data-efficiency and
introduce the interpretability for learned policy. This framework consists of a
high-level agent, a subtask solver and a symbolic transition model. Without
assuming any prior knowledge on the state transition, we adopt inductive logic
programming (ILP) to learn the rules of symbolic state transitions, introducing
interpretability and making the learned behavior understandable to users. In
empirical experiments, we confirmed that the proposed framework offers
approximately between 30\% to 40\% more data efficiency over previous methods