67 research outputs found
Knowledge Transfer with Jacobian Matching
Classical distillation methods transfer representations from a "teacher"
neural network to a "student" network by matching their output activations.
Recent methods also match the Jacobians, or the gradient of output activations
with the input. However, this involves making some ad hoc decisions, in
particular, the choice of the loss function.
In this paper, we first establish an equivalence between Jacobian matching
and distillation with input noise, from which we derive appropriate loss
functions for Jacobian matching. We then rely on this analysis to apply
Jacobian matching to transfer learning by establishing equivalence of a recent
transfer learning procedure to distillation.
We then show experimentally on standard image datasets that Jacobian-based
penalties improve distillation, robustness to noisy inputs, and transfer
learning
Data-free parameter pruning for Deep Neural Networks
Deep Neural nets (NNs) with millions of parameters are at the heart of many
state-of-the-art computer vision systems today. However, recent works have
shown that much smaller models can achieve similar levels of performance. In
this work, we address the problem of pruning parameters in a trained NN model.
Instead of removing individual weights one at a time as done in previous works,
we remove one neuron at a time. We show how similar neurons are redundant, and
propose a systematic way to remove them. Our experiments in pruning the densely
connected layers show that we can remove upto 85\% of the total parameters in
an MNIST-trained network, and about 35\% for AlexNet without significantly
affecting performance. Our method can be applied on top of most networks with a
fully connected layer to give a smaller network.Comment: BMVC 201
Compensating for Large In-Plane Rotations in Natural Images
Rotation invariance has been studied in the computer vision community
primarily in the context of small in-plane rotations. This is usually achieved
by building invariant image features. However, the problem of achieving
invariance for large rotation angles remains largely unexplored. In this work,
we tackle this problem by directly compensating for large rotations, as opposed
to building invariant features. This is inspired by the neuro-scientific
concept of mental rotation, which humans use to compare pairs of rotated
objects. Our contributions here are three-fold. First, we train a Convolutional
Neural Network (CNN) to detect image rotations. We find that generic CNN
architectures are not suitable for this purpose. To this end, we introduce a
convolutional template layer, which learns representations for canonical
'unrotated' images. Second, we use Bayesian Optimization to quickly sift
through a large number of candidate images to find the canonical 'unrotated'
image. Third, we use this method to achieve robustness to large angles in an
image retrieval scenario. Our method is task-agnostic, and can be used as a
pre-processing step in any computer vision system.Comment: Accepted at Indian Conference on Computer Vision, Graphics and Image
Processing (ICVGIP) 201
Efficient Estimation of the Local Robustness of Machine Learning Models
Machine learning models often need to be robust to noisy input data.
Real-world noise (such as measurement noise) is often random and the effect of
such noise on model predictions is captured by a model's local robustness,
i.e., the consistency of model predictions in a local region around an input.
Local robustness is therefore an important characterization of real-world model
behavior and can be useful for debugging models and establishing user trust.
However, the na\"ive approach to computing local robustness based on
Monte-Carlo sampling is statistically inefficient, especially for
high-dimensional data, leading to prohibitive computational costs for
large-scale applications. In this work, we develop the first analytical
estimators to efficiently compute local robustness of multi-class
discriminative models. These estimators linearize models in the local region
around an input and compute the model's local robustness using the multivariate
Normal cumulative distribution function. Through the derivation of these
estimators, we show how local robustness is connected to such concepts as
randomized smoothing and softmax probability. In addition, we show empirically
that these estimators efficiently compute the local robustness of standard deep
learning models and demonstrate these estimators' usefulness for various tasks
involving local robustness, such as measuring robustness bias and identifying
examples that are vulnerable to noise perturbation in a dataset. To our
knowledge, this work is the first to investigate local robustness in a
multi-class setting and develop efficient analytical estimators for local
robustness. In doing so, this work not only advances the conceptual
understanding of local robustness, but also makes its computation practical,
enabling the use of local robustness in critical downstream applications
A Taxonomy of Deep Convolutional Neural Nets for Computer Vision
Traditional architectures for solving computer vision problems and the degree
of success they enjoyed have been heavily reliant on hand-crafted features.
However, of late, deep learning techniques have offered a compelling
alternative -- that of automatically learning problem-specific features. With
this new paradigm, every problem in computer vision is now being re-examined
from a deep learning perspective. Therefore, it has become important to
understand what kind of deep networks are suitable for a given problem.
Although general surveys of this fast-moving paradigm (i.e. deep-networks)
exist, a survey specific to computer vision is missing. We specifically
consider one form of deep networks widely used in computer vision -
convolutional neural networks (CNNs). We start with "AlexNet" as our base CNN
and then examine the broad variations proposed over time to suit different
applications. We hope that our recipe-style survey will serve as a guide,
particularly for novice practitioners intending to use deep-learning techniques
for computer vision.Comment: Published in Frontiers in Robotics and AI (http://goo.gl/6691Bm
Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability
With the increased deployment of machine learning models in various
real-world applications, researchers and practitioners alike have emphasized
the need for explanations of model behaviour. To this end, two broad strategies
have been outlined in prior literature to explain models. Post hoc explanation
methods explain the behaviour of complex black-box models by identifying
features critical to model predictions; however, prior work has shown that
these explanations may not be faithful, in that they incorrectly attribute high
importance to features that are unimportant or non-discriminative for the
underlying task. Inherently interpretable models, on the other hand, circumvent
these issues by explicitly encoding explanations into model architecture,
meaning their explanations are naturally faithful, but they often exhibit poor
predictive performance due to their limited expressive power. In this work, we
identify a key reason for the lack of faithfulness of feature attributions: the
lack of robustness of the underlying black-box models, especially to the
erasure of unimportant distractor features in the input. To address this issue,
we propose Distractor Erasure Tuning (DiET), a method that adapts black-box
models to be robust to distractor erasure, thus providing discriminative and
faithful attributions. This strategy naturally combines the ease of use of post
hoc explanations with the faithfulness of inherently interpretable models. We
perform extensive experiments on semi-synthetic and real-world datasets and
show that DiET produces models that (1) closely approximate the original
black-box models they are intended to explain, and (2) yield explanations that
match approximate ground truths available by construction. Our code is made
public at https://github.com/AI4LIFE-GROUP/DiET
- …