1,558 research outputs found
Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation
Robotic systems are ever more capable of automation and fulfilment of complex
tasks, particularly with reliance on recent advances in intelligent systems,
deep learning and artificial intelligence. However, as robots and humans come
closer in their interactions, the matter of interpretability, or explainability
of robot decision-making processes for the human grows in importance. A
successful interaction and collaboration will only take place through mutual
understanding of underlying representations of the environment and the task at
hand. This is currently a challenge in deep learning systems. We present a
hierarchical deep reinforcement learning system, consisting of a low-level
agent handling the large actions/states space of a robotic system efficiently,
by following the directives of a high-level agent which is learning the
high-level dynamics of the environment and task. This high-level agent forms a
representation of the world and task at hand that is interpretable for a human
operator. The method, which we call Dot-to-Dot, is tested on a MuJoCo-based
model of the Fetch Robotics Manipulator, as well as a Shadow Hand, to test its
performance. Results show efficient learning of complex actions/states spaces
by the low-level agent, and an interpretable representation of the task and
decision-making process learned by the high-level agent
Explainable agents adapt to human behaviour
When integrating artificial agents into physical or digital environments that are shared with humans, agents are often equipped with opaque Machine Learning methods to enable adapting their behaviour to dynamic human needs and environment. This brings about agents that are also opaque and therefore hard to explain. In previous work, we show that we can reduce an opaque agent into an explainable Policy Graph (PG) which works accurately in multi-agent environments. Policy Graphs are based on a discretisation of the world into propositional logic to identify states, and the choice of which discretiser to apply is key to the performance of the reduced agent. In this work, we explore this further by 1) reducing a single agent into an explainable PG, and 2) enforcing collaboration between this agent and an agent trained from human behaviour. The human agent is computed by using GAIL from a series of human-played episodes, and kept unchanged. We show that an opaque agent created and trained to collaborate with the human agent can be reduced to an explainable, non-opaque PG, so long as predicates regarding collaboration are included in the state representation, by showing the difference in reward between the agent and its PG. Code is available at https://github.com/HPAI-BSC/explainable-agents-with-humansThis work has been partially supported by EU Horizon 2020 Project StairwAI (grant agreement No. 101017142).Peer ReviewedPostprint (published version
Explainability in Deep Reinforcement Learning
A large set of the explainable Artificial Intelligence (XAI) literature is
emerging on feature relevance techniques to explain a deep neural network (DNN)
output or explaining models that ingest image source data. However, assessing
how XAI techniques can help understand models beyond classification tasks, e.g.
for reinforcement learning (RL), has not been extensively studied. We review
recent works in the direction to attain Explainable Reinforcement Learning
(XRL), a relatively new subfield of Explainable Artificial Intelligence,
intended to be used in general public applications, with diverse audiences,
requiring ethical, responsible and trustable algorithms. In critical situations
where it is essential to justify and explain the agent's behaviour, better
explainability and interpretability of RL models could help gain scientific
insight on the inner workings of what is still considered a black box. We
evaluate mainly studies directly linking explainability to RL, and split these
into two categories according to the way the explanations are generated:
transparent algorithms and post-hoc explainaility. We also review the most
prominent XAI works from the lenses of how they could potentially enlighten the
further deployment of the latest advances in RL, in the demanding present and
future of everyday problems.Comment: Article accepted at Knowledge-Based System
Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
With advances in reinforcement learning (RL), agents are now being developed
in high-stakes application domains such as healthcare and transportation.
Explaining the behavior of these agents is challenging, as the environments in
which they act have large state spaces, and their decision-making can be
affected by delayed rewards, making it difficult to analyze their behavior. To
address this problem, several approaches have been developed. Some approaches
attempt to convey the behavior of the agent, describing the
actions it takes in different states. Other approaches devised
explanations which provide information regarding the agent's decision-making in
a particular state. In this paper, we combine global and local explanation
methods, and evaluate their joint and separate contributions, providing (to the
best of our knowledge) the first user study of combined local and global
explanations for RL agents. Specifically, we augment strategy summaries that
extract important trajectories of states from simulations of the agent with
saliency maps which show what information the agent attends to. Our results
show that the choice of what states to include in the summary (global
information) strongly affects people's understanding of agents: participants
shown summaries that included important states significantly outperformed
participants who were presented with agent behavior in a randomly set of chosen
world-states. We find mixed results with respect to augmenting demonstrations
with saliency maps (local information), as the addition of saliency maps did
not significantly improve performance in most cases. However, we do find some
evidence that saliency maps can help users better understand what information
the agent relies on in its decision making, suggesting avenues for future work
that can further improve explanations of RL agents
- …