232 research outputs found
Interactive Visual Feature Search
Many visualization techniques have been created to help explain the behavior
of convolutional neural networks (CNNs), but they largely consist of static
diagrams that convey limited information. Interactive visualizations can
provide more rich insights and allow users to more easily explore a model's
behavior; however, they are typically not easily reusable and are specific to a
particular model.
We introduce Visual Feature Search, a novel interactive visualization that is
generalizable to any CNN and can easily be incorporated into a researcher's
workflow. Our tool allows a user to highlight an image region and search for
images from a given dataset with the most similar CNN features. It supports
searching through large image datasets with an efficient cache-based search
implementation. We demonstrate how our tool elucidates different aspects of
model behavior by performing experiments on supervised, self-supervised, and
human-edited CNNs. We also release a portable Python library and several
IPython notebooks to enable researchers to easily use our tool in their own
experiments. Our code can be found at
https://github.com/lookingglasslab/VisualFeatureSearch
Occlusions for Effective Data Augmentation in Image Classification
Deep networks for visual recognition are known to leverage "easy to
recognise" portions of objects such as faces and distinctive texture patterns.
The lack of a holistic understanding of objects may increase fragility and
overfitting. In recent years, several papers have proposed to address this
issue by means of occlusions as a form of data augmentation. However, successes
have been limited to tasks such as weak localization and model interpretation,
but no benefit was demonstrated on image classification on large-scale
datasets. In this paper, we show that, by using a simple technique based on
batch augmentation, occlusions as data augmentation can result in better
performance on ImageNet for high-capacity models (e.g., ResNet50). We also show
that varying amounts of occlusions used during training can be used to study
the robustness of different neural network architectures.Comment: Accepted to 2019 ICCV Workshop on Interpreting and Explaining Visual
Artificial Intelligence Models (v2: corrected references
Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks
In an effort to understand the meaning of the intermediate representations
captured by deep networks, recent papers have tried to associate specific
semantic concepts to individual neural network filter responses, where
interesting correlations are often found, largely by focusing on extremal
filter responses. In this paper, we show that this approach can favor
easy-to-interpret cases that are not necessarily representative of the average
behavior of a representation.
A more realistic but harder-to-study hypothesis is that semantic
representations are distributed, and thus filters must be studied in
conjunction. In order to investigate this idea while enabling systematic
visualization and quantification of multiple filter responses, we introduce the
Net2Vec framework, in which semantic concepts are mapped to vectorial
embeddings based on corresponding filter responses. By studying such
embeddings, we are able to show that 1., in most cases, multiple filters are
required to code for a concept, that 2., often filters are not concept specific
and help encode multiple concepts, and that 3., compared to single filter
activations, filter embeddings are able to better characterize the meaning of a
representation and its relationship to other concepts.Comment: Camera-Ready for CVPR18; supplementary materials:
http://ruthcfong.github.io/files/net2vec_supps.pd
Understanding Deep Networks via Extremal Perturbations and Smooth Masks
The problem of attribution is concerned with identifying the parts of an
input that are responsible for a model's output. An important family of
attribution methods is based on measuring the effect of perturbations applied
to the input. In this paper, we discuss some of the shortcomings of existing
approaches to perturbation analysis and address them by introducing the concept
of extremal perturbations, which are theoretically grounded and interpretable.
We also introduce a number of technical innovations to compute extremal
perturbations, including a new area constraint and a parametric family of
smooth perturbations, which allow us to remove all tunable hyper-parameters
from the optimization problem. We analyze the effect of perturbations as a
function of their area, demonstrating excellent sensitivity to the spatial
properties of the deep neural network under stimulation. We also extend
perturbation analysis to the intermediate layers of a network. This application
allows us to identify the salient channels necessary for classification, which,
when visualized using feature inversion, can be used to elucidate model
behavior. Lastly, we introduce TorchRay, an interpretability library built on
PyTorch.Comment: Accepted at ICCV 2019 as oral; supp mat at
http://ruthcfong.github.io/files/fong19_extremal_supps.pd
Improving Fine-Grain Segmentation via Interpretable Modifications: A Case Study in Fossil Segmentation
Most interpretability research focuses on datasets containing thousands of
images of commonplace objects. However, many high-impact datasets, such as
those in medicine and the geosciences, contain fine-grain objects that require
domain-expert knowledge to recognize and are time-consuming to collect and
annotate. As a result, these datasets contain few annotated images, and current
machine vision models cannot train intensively on them. Thus, adapting
interpretability techniques to maximize the amount of information that models
can learn from small, fine-grain datasets is an important endeavor.
Using a Mask R-CNN to segment ancient reef fossils in rock sample images, we
present a general paradigm for identifying and mitigating model weaknesses.
Specifically, we apply image perturbations to expose the Mask R-CNN's inability
to distinguish between different classes of fossils and its inconsistency in
segmenting fossils with different textures. To address these shortcomings, we
extend an existing model-editing method for correcting systematic mistakes in
image classification to image segmentation and introduce a novel application of
the technique: encouraging a greater separation between positive and negative
pixels for a given class. Through extensive experiments, we find that editing
the model by perturbing all pixels for a given class in one image is most
effective (compared to using multiple images and/or fewer pixels). Our paradigm
may also generalize to other segmentation models trained on small, fine-grain
datasets
There and Back Again: Revisiting Backpropagation Saliency Methods
Saliency methods seek to explain the predictions of a model by producing an
importance map across each input sample. A popular class of such methods is
based on backpropagating a signal and analyzing the resulting gradient. Despite
much research on such methods, relatively little work has been done to clarify
the differences between such methods as well as the desiderata of these
techniques. Thus, there is a need for rigorously understanding the
relationships between different methods as well as their failure modes. In this
work, we conduct a thorough analysis of backpropagation-based saliency methods
and propose a single framework under which several such methods can be unified.
As a result of our study, we make three additional contributions. First, we use
our framework to propose NormGrad, a novel saliency method based on the spatial
contribution of gradients of convolutional weights. Second, we combine saliency
maps at different layers to test the ability of saliency methods to extract
complementary information at different network levels (e.g.~trading off spatial
resolution and distinctiveness) and we explain why some methods fail at
specific layers (e.g., Grad-CAM anywhere besides the last convolutional layer).
Third, we introduce a class-sensitivity metric and a meta-learning inspired
paradigm applicable to any saliency method for improving sensitivity to the
output class being explained.Comment: CVPR 202
UFO: A unified method for controlling Understandability and Faithfulness Objectives in concept-based explanations for CNNs
Concept-based explanations for convolutional neural networks (CNNs) aim to
explain model behavior and outputs using a pre-defined set of semantic concepts
(e.g., the model recognizes scene class ``bedroom'' based on the presence of
concepts ``bed'' and ``pillow''). However, they often do not faithfully (i.e.,
accurately) characterize the model's behavior and can be too complex for people
to understand. Further, little is known about how faithful and understandable
different explanation methods are, and how to control these two properties. In
this work, we propose UFO, a unified method for controlling Understandability
and Faithfulness Objectives in concept-based explanations. UFO formalizes
understandability and faithfulness as mathematical objectives and unifies most
existing concept-based explanations methods for CNNs. Using UFO, we
systematically investigate how explanations change as we turn the knobs of
faithfulness and understandability. Our experiments demonstrate a
faithfulness-vs-understandability tradeoff: increasing understandability
reduces faithfulness. We also provide insights into the ``disagreement
problem'' in explainable machine learning, by analyzing when and how
concept-based explanations disagree with each other
- …