21 research outputs found
Scattering Vision Transformer: Spectral Mixing Matters
Vision transformers have gained significant attention and achieved
state-of-the-art performance in various computer vision tasks, including image
classification, instance segmentation, and object detection. However,
challenges remain in addressing attention complexity and effectively capturing
fine-grained information within images. Existing solutions often resort to
down-sampling operations, such as pooling, to reduce computational cost.
Unfortunately, such operations are non-invertible and can result in information
loss. In this paper, we present a novel approach called Scattering Vision
Transformer (SVT) to tackle these challenges. SVT incorporates a spectrally
scattering network that enables the capture of intricate image details. SVT
overcomes the invertibility issue associated with down-sampling operations by
separating low-frequency and high-frequency components. Furthermore, SVT
introduces a unique spectral gating network utilizing Einstein multiplication
for token and channel mixing, effectively reducing complexity. We show that SVT
achieves state-of-the-art performance on the ImageNet dataset with a
significant reduction in a number of parameters and FLOPS. SVT shows 2\%
improvement over LiTv2 and iFormer. SVT-H-S reaches 84.2\% top-1 accuracy,
while SVT-H-B reaches 85.2\% (state-of-art for base versions) and SVT-H-L
reaches 85.7\% (again state-of-art for large versions). SVT also shows
comparable results in other vision tasks such as instance segmentation. SVT
also outperforms other transformers in transfer learning on standard datasets
such as CIFAR10, CIFAR100, Oxford Flower, and Stanford Car datasets. The
project page is available on this
webpage.\url{https://badripatro.github.io/svt/}.Comment: Accepted @NeurIPS 202
Robust explanations for visual question answering
In this paper, we propose a method to obtain robust explanations for visual
question answering(VQA) that correlate well with the answers. Our model
explains the answers obtained through a VQA model by providing visual and
textual explanations. The main challenges that we address are i) Answers and
textual explanations obtained by current methods are not well correlated and
ii) Current methods for visual explanation do not focus on the right location
for explaining the answer. We address both these challenges by using a
collaborative correlated module which ensures that even if we do not train for
noise based attacks, the enhanced correlation ensures that the right
explanation and answer can be generated. We further show that this also aids in
improving the generated visual and textual explanations. The use of the
correlated module can be thought of as a robust method to verify if the answer
and explanations are coherent. We evaluate this model using VQA-X dataset. We
observe that the proposed method yields better textual and visual justification
that supports the decision. We showcase the robustness of the model against a
noise-based perturbation attack using corresponding visual and textual
explanations. A detailed empirical analysis is shown. Here we provide source
code link for our model \url{https://github.com/DelTA-Lab-IITK/CCM-WACV}.Comment: WACV-2020 (Accepted
SpectFormer: Frequency and Attention is what you need in a Vision Transformer
Vision transformers have been applied successfully for image recognition
tasks. There have been either multi-headed self-attention based (ViT
\cite{dosovitskiy2020image}, DeIT, \cite{touvron2021training}) similar to the
original work in textual models or more recently based on spectral layers
(Fnet\cite{lee2021fnet}, GFNet\cite{rao2021global},
AFNO\cite{guibas2021efficient}). We hypothesize that both spectral and
multi-headed attention plays a major role. We investigate this hypothesis
through this work and observe that indeed combining spectral and multi-headed
attention layers provides a better transformer architecture. We thus propose
the novel Spectformer architecture for transformers that combines spectral and
multi-headed attention layers. We believe that the resulting representation
allows the transformer to capture the feature representation appropriately and
it yields improved performance over other transformer representations. For
instance, it improves the top-1 accuracy by 2\% on ImageNet compared to both
GFNet-H and LiT. SpectFormer-S reaches 84.25\% top-1 accuracy on ImageNet-1K
(state of the art for small version). Further, Spectformer-L achieves 85.7\%
that is the state of the art for the comparable base version of the
transformers. We further ensure that we obtain reasonable results in other
scenarios such as transfer learning on standard datasets such as CIFAR-10,
CIFAR-100, Oxford-IIIT-flower, and Standford Car datasets. We then investigate
its use in downstream tasks such of object detection and instance segmentation
on the MS-COCO dataset and observe that Spectformer shows consistent
performance that is comparable to the best backbones and can be further
optimized and improved. Hence, we believe that combined spectral and attention
layers are what are needed for vision transformers.Comment: The project page is available at this webpage
\url{https://badripatro.github.io/SpectFormers/}. arXiv admin note: text
overlap with arXiv:2207.04978 by other author
U-CAM: Visual Explanation using Uncertainty based Class Activation Maps
Understanding and explaining deep learning models is an imperative task.
Towards this, we propose a method that obtains gradient-based certainty
estimates that also provide visual attention maps. Particularly, we solve for
visual question answering task. We incorporate modern probabilistic deep
learning methods that we further improve by using the gradients for these
estimates. These have two-fold benefits: a) improvement in obtaining the
certainty estimates that correlate better with misclassified samples and b)
improved attention maps that provide state-of-the-art results in terms of
correlation with human attention regions. The improved attention maps result in
consistent improvement for various methods for visual question answering.
Therefore, the proposed technique can be thought of as a recipe for obtaining
improved certainty estimates and explanation for deep learning models. We
provide detailed empirical analysis for the visual question answering task on
all standard benchmarks and comparison with state of the art methods.Comment: ICCV 2019 (accepted