7 research outputs found
A study on the Interpretability of Neural Retrieval Models using DeepSHAP
A recent trend in IR has been the usage of neural networks to learn retrieval
models for text based adhoc search. While various approaches and architectures
have yielded significantly better performance than traditional retrieval models
such as BM25, it is still difficult to understand exactly why a document is
relevant to a query. In the ML community several approaches for explaining
decisions made by deep neural networks have been proposed -- including DeepSHAP
which modifies the DeepLift algorithm to estimate the relative importance
(shapley values) of input features for a given decision by comparing the
activations in the network for a given image against the activations caused by
a reference input. In image classification, the reference input tends to be a
plain black image. While DeepSHAP has been well studied for image
classification tasks, it remains to be seen how we can adapt it to explain the
output of Neural Retrieval Models (NRMs). In particular, what is a good "black"
image in the context of IR? In this paper we explored various reference input
document construction techniques. Additionally, we compared the explanations
generated by DeepSHAP to LIME (a model agnostic approach) and found that the
explanations differ considerably. Our study raises concerns regarding the
robustness and accuracy of explanations produced for NRMs. With this paper we
aim to shed light on interesting problems surrounding interpretability in NRMs
and highlight areas of future work.Comment: 4 pages; SIGIR 2019 Short Pape
From Anecdotal Evidence to Quantitative Evaluation Methods:A Systematic Review on Evaluating Explainable AI
The rising popularity of explainable artificial intelligence (XAI) to
understand high-performing black boxes, also raised the question of how to
evaluate explanations of machine learning (ML) models. While interpretability
and explainability are often presented as a subjectively validated binary
property, we consider it a multi-faceted concept. We identify 12 conceptual
properties, such as Compactness and Correctness, that should be evaluated for
comprehensively assessing the quality of an explanation. Our so-called Co-12
properties serve as categorization scheme for systematically reviewing the
evaluation practice of more than 300 papers published in the last 7 years at
major AI and ML conferences that introduce an XAI method. We find that 1 in 3
papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate
with users. We also contribute to the call for objective, quantifiable
evaluation methods by presenting an extensive overview of quantitative XAI
evaluation methods. This systematic collection of evaluation methods provides
researchers and practitioners with concrete tools to thoroughly validate,
benchmark and compare new and existing XAI methods. This also opens up
opportunities to include quantitative metrics as optimization criteria during
model training in order to optimize for accuracy and interpretability
simultaneously.Comment: Link to website added: https://utwente-dmb.github.io/xai-papers