Recent years saw a plethora of work on explaining complex intelligent agents.
One example is the development of several algorithms that generate saliency
maps which show how much each pixel attributed to the agents' decision.
However, most evaluations of such saliency maps focus on image classification
tasks. As far as we know, there is no work which thoroughly compares different
saliency maps for Deep Reinforcement Learning agents. This paper compares four
perturbation-based approaches to create saliency maps for Deep Reinforcement
Learning agents trained on four different Atari 2600 games. All four approaches
work by perturbing parts of the input and measuring how much this affects the
agent's output. The approaches are compared using three computational metrics:
dependence on the learned parameters of the agent (sanity checks), faithfulness
to the agent's reasoning (input degradation), and run-time.Comment: Presented on the Explainable Agency in Artificial Intelligence
  Workshop during the 35th AAAI Conference on Artificial Intelligenc

André, Elisabeth

Huber, Tobias

Limmer, Benedikt

English

arXiv

One of the most prominent methods for explaining the behavior of Deep Reinforcement Learning (DRL) agents is the generation of saliency maps that show how much each pixel attributed to the agents' decision. However, there is no work that computationally evaluates and compares the fidelity of different perturbation-based saliency map approaches specifically for DRL agents. It is particularly challenging to computationally evaluate saliency maps for DRL agents since their decisions are part of an overarching policy, which includes long-term decision making. For instance, the output neurons of value-based DRL algorithms encode both the value of the current state as well as the expected future reward after doing each action in this state. This ambiguity should be considered when evaluating saliency maps for such agents. In this paper, we compare five popular perturbation-based approaches to create saliency maps for DRL agents trained on four different Atari 2,600 games. The approaches are compared using two computational metrics: dependence on the learned parameters of the underlying deep Q-network of the agents (sanity checks) and fidelity to the agents' reasoning (input degradation). During the sanity checks, we found that a popular noise-based saliency map approach for DRL agents shows little dependence on the parameters of the output layer. We demonstrate that this can be fixed by tweaking the algorithm such that it focuses on specific actions instead of the general entropy within the output values. For fidelity, we identify two main factors that influence which saliency map approach should be chosen in which situation. Particular to value-based DRL agents, we show that analyzing the agents' choice of action requires different saliency map approaches than analyzing the agents' state value estimation

OPUS Augsburg

Benchmarking perturbation-based saliency maps for explaining Atari agents

PubMed Central

Benchmarking Perturbation-Based Saliency Maps for Explaining Atari Agents

https://opus.bibliothek.uni-augsburg.de/opus4/files/92481/arxiv 2101.07312.pdf

Benchmarking Perturbation-based Saliency Maps for Explaining Deep Reinforcement Learning Agents

Abstract

Similar works

Full text

Available Versions

OPUS Augsburg

PubMed Central