12 research outputs found
Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors
Convolutional neural network (CNN) models for computer vision are powerful
but lack explainability in their most basic form. This deficiency remains a key
challenge when applying CNNs in important domains. Recent work for explanations
through feature importance of approximate linear models has moved from
input-level features (pixels or segments) to features from mid-layer feature
maps in the form of concept activation vectors (CAVs). CAVs contain
concept-level information and could be learnt via clustering. In this work, we
rethink the ACE algorithm of Ghorbani et al., proposing an alternative
inevitable concept-based explanation (ICE) framework to overcome its
shortcomings. Based on the requirements of fidelity (approximate models to
target models) and interpretability (being meaningful to people), we design
measurements and evaluate a range of matrix factorization methods with our
framework. We find that \emph{non-negative concept activation vectors} (NCAVs)
from non-negative matrix factorization provide superior performance in
interpretability and fidelity based on computational and human subject
experiments. Our framework provides both local and global concept-level
explanations for pre-trained CNN models
Explainable Agency in Reinforcement Learning Agents
This thesis explores how reinforcement learning (RL) agents can provide explanations for their actions and behaviours. As humans, we build causal models to encode cause-effect relations of events and use these to explain why events happen. Taking inspiration from cognitive psychology and social science literature, I build causal explanation models and explanation dialogue models for RL agents. By mimicking human-like explanation models, these agents can provide explanations that are natural and intuitive to humans
Explainable Reinforcement Learning through a Causal Lens
Prominent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen by referring to counterfactuals — things that did not happen. In this paper, we use causal models to derive causal explanations of the behaviour of model-free reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We computationally evaluate the model in 6 domains and measure performance and task prediction accuracy. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigate: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results show that causal model explanations perform better on these measures compared to two other baseline explanation models