2 research outputs found
Iterative and Adaptive Sampling with Spatial Attention for Black-Box Model Explanations
Deep neural networks have achieved great success in many real-world
applications, yet it remains unclear and difficult to explain their
decision-making process to an end-user. In this paper, we address the
explainable AI problem for deep neural networks with our proposed framework,
named IASSA, which generates an importance map indicating how salient each
pixel is for the model's prediction with an iterative and adaptive sampling
module. We employ an affinity matrix calculated on multi-level deep learning
features to explore long-range pixel-to-pixel correlation, which can shift the
saliency values guided by our long-range and parameter-free spatial attention.
Extensive experiments on the MS-COCO dataset show that our proposed approach
matches or exceeds the performance of state-of-the-art black-box explanation
methods.Comment: The paper was accepted to the IEEE Winter Conference on Applications
of Computer Vision (WACV'2020
Visualizing Color-wise Saliency of Black-Box Image Classification Models
Image classification based on machine learning is being commonly used.
However, a classification result given by an advanced method, including deep
learning, is often hard to interpret. This problem of interpretability is one
of the major obstacles in deploying a trained model in safety-critical systems.
Several techniques have been proposed to address this problem; one of which is
RISE, which explains a classification result by a heatmap, called a saliency
map, which explains the significance of each pixel. We propose MC-RISE
(Multi-Color RISE), which is an enhancement of RISE to take color information
into account in an explanation. Our method not only shows the saliency of each
pixel in a given image as the original RISE does, but the significance of color
components of each pixel; a saliency map with color information is useful
especially in the domain where the color information matters (e.g.,
traffic-sign recognition). We implemented MC-RISE and evaluate them using two
datasets (GTSRB and ImageNet) to demonstrate the effectiveness of our methods
in comparison with existing techniques for interpreting image classification
results.Comment: To appear in ACCV 202