10 research outputs found
Augmenting CLIP with Improved Visio-Linguistic Reasoning
Image-text contrastive models such as CLIP are useful for a variety of
downstream applications including zero-shot classification, image-text
retrieval and transfer learning. However, these contrastively trained
vision-language models often fail on compositional visio-linguistic tasks such
as Winoground with performance equivalent to random chance. In our paper, we
address this issue and propose a sample-efficient light-weight method called
SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
of CLIP. The core idea of our method is to use differentiable image
parameterizations to fine-tune CLIP with a distillation objective from large
text-to-image generative models such as Stable-Diffusion which are relatively
good at visio-linguistic reasoning tasks. On the challenging Winoground
compositional reasoning benchmark, our method improves the absolute
visio-linguistic performance of different CLIP models by up to 7%, while on the
ARO dataset, our method improves the visio-linguistic performance by upto 3%.
As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
that the zero-shot performance improves marginally on a variety of downstream
datasets. Our method reinforces that carefully designed distillation objectives
from generative models can be leveraged to extend existing contrastive
image-text models with improved visio-linguistic reasoning capabilities
Topic Segmentation in the Wild: Towards Segmentation of Semi-structured & Unstructured Chats
Breaking down a document or a conversation into multiple contiguous segments
based on its semantic structure is an important and challenging problem in NLP,
which can assist many downstream tasks. However, current works on topic
segmentation often focus on segmentation of structured texts. In this paper, we
comprehensively analyze the generalization capabilities of state-of-the-art
topic segmentation models on unstructured texts. We find that: (a) Current
strategies of pre-training on a large corpus of structured text such as
Wiki-727K do not help in transferability to unstructured texts. (b) Training
from scratch with only a relatively small-sized dataset of the target
unstructured domain improves the segmentation results by a significant margin.Comment: NeurIPS 2022 : ENLS
Topic Segmentation of Semi-Structured and Unstructured Conversational Datasets using Language Models
Breaking down a document or a conversation into multiple contiguous segments
based on its semantic structure is an important and challenging problem in NLP,
which can assist many downstream tasks. However, current works on topic
segmentation often focus on segmentation of structured texts. In this paper, we
comprehensively analyze the generalization capabilities of state-of-the-art
topic segmentation models on unstructured texts. We find that: (a) Current
strategies of pre-training on a large corpus of structured text such as
Wiki-727K do not help in transferability to unstructured conversational data.
(b) Training from scratch with only a relatively small-sized dataset of the
target unstructured domain improves the segmentation results by a significant
margin. We stress-test our proposed Topic Segmentation approach by
experimenting with multiple loss functions, in order to mitigate effects of
imbalance in unstructured conversational datasets. Our empirical evaluation
indicates that Focal Loss function is a robust alternative to Cross-Entropy and
re-weighted Cross-Entropy loss function when segmenting unstructured and
semi-structured chats.Comment: Accepted to IntelliSys 2023. arXiv admin note: substantial text
overlap with arXiv:2211.1495
On Surgical Fine-tuning for Language Encoders
Fine-tuning all the layers of a pre-trained neural language encoder (either
using all the parameters or using parameter-efficient methods) is often the
de-facto way of adapting it to a new task. We show evidence that for different
downstream language tasks, fine-tuning only a subset of layers is sufficient to
obtain performance that is close to and often better than fine-tuning all the
layers in the language encoder. We propose an efficient metric based on the
diagonal of the Fisher information matrix (FIM score), to select the candidate
layers for selective fine-tuning. We show, empirically on GLUE and SuperGLUE
tasks and across distinct language encoders, that this metric can effectively
select layers leading to a strong downstream performance. Our work highlights
that task-specific information corresponding to a given downstream task is
often localized within a few layers, and tuning only those is sufficient for
strong performance. Additionally, we demonstrate the robustness of the FIM
score to rank layers in a manner that remains constant during the optimization
process.Comment: Accepted to EMNLP 202
Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning
Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase on a set of base classes. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters. While these methods have shown promise, inconsistencies in experimental conditions make it difficult to disentangle their advantage from other experimental factors including the feature extractor architecture, pre-trained initialization and fine-tuning algorithm, amongst others. In our paper, we conduct a large-scale, experimentally consistent, empirical analysis to study PEFTs for few-shot image classification. Through a battery of over 1.8k controlled experiments on large-scale few-shot benchmarks including Meta-Dataset and ORBIT, we uncover novel insights on PEFTs that cast light on their efficacy in fine-tuning ViTs for few-shot classification. Through our controlled empirical study, we have two main findings: (i) Fine-tuning just the LayerNorm parameters (which we call LN-Tune) during few-shot adaptation is an extremely strong baseline across ViTs pre-trained with both self-supervised and supervised objectives, (ii) For self-supervised ViTs, we find that simply learning a set of scaling parameters for each attention matrix (which we call Attn-Scale) along with a domain-residual adapter (DRA) module leads to state-of-the-art performance (while being ~9x more parameter-efficient) on Meta-Dataset. Our empirical findings set strong baselines and call for rethinking the current design of PEFT methods for FSC
Localizing and Editing Knowledge in Text-to-Image Generative Models
Text-to-Image Diffusion Models such as Stable-Diffusion and Imagen have
achieved unprecedented quality of photorealism with state-of-the-art FID scores
on MS-COCO and other generation benchmarks. Given a caption, image generation
requires fine-grained knowledge about attributes such as object structure,
style, and viewpoint amongst others. Where does this information reside in
text-to-image generative models? In our paper, we tackle this question and
understand how knowledge corresponding to distinct visual attributes is stored
in large-scale text-to-image diffusion models. We adapt Causal Mediation
Analysis for text-to-image models and trace knowledge about distinct visual
attributes to various (causal) components in the (i) UNet and (ii) text-encoder
of the diffusion model. In particular, we show that unlike generative
large-language models, knowledge about different attributes is not localized in
isolated components, but is instead distributed amongst a set of components in
the conditional UNet. These sets of components are often distinct for different
visual attributes. Remarkably, we find that the CLIP text-encoder in public
text-to-image models such as Stable-Diffusion contains only one causal state
across different visual attributes, and this is the first self-attention layer
corresponding to the last subject token of the attribute in the caption. This
is in stark contrast to the causal states in other language models which are
often the mid-MLP layers. Based on this observation of only one causal state in
the text-encoder, we introduce a fast, data-free model editing method
Diff-QuickFix which can effectively edit concepts in text-to-image models.
DiffQuickFix can edit (ablate) concepts in under a second with a closed-form
update, providing a significant 1000x speedup and comparable editing
performance to existing fine-tuning based editing methods.Comment: 61 page