13,286 research outputs found
Text-to-Image Diffusion Models are Zero-Shot Classifiers
The excellent generative capabilities of text-to-image diffusion models
suggest they learn informative representations of image-text data. However,
what knowledge their representations capture is not fully understood, and they
have not been thoroughly explored on downstream tasks. We investigate diffusion
models by proposing a method for evaluating them as zero-shot classifiers. The
key idea is using a diffusion model's ability to denoise a noised image given a
text description of a label as a proxy for that label's likelihood. We apply
our method to Imagen, using it to probe fine-grained aspects of Imagen's
knowledge and comparing it with CLIP's zero-shot abilities. Imagen performs
competitively with CLIP on a wide range of zero-shot image classification
datasets. Additionally, it achieves state-of-the-art results on shape/texture
bias tests and can successfully perform attribute binding while CLIP cannot.
Although generative pre-training is prevalent in NLP, visual foundation models
often use other methods such as contrastive learning. Based on our findings, we
argue that generative pre-training should be explored as a compelling
alternative for vision and vision-language problems
Augmenting CLIP with Improved Visio-Linguistic Reasoning
Image-text contrastive models such as CLIP are useful for a variety of
downstream applications including zero-shot classification, image-text
retrieval and transfer learning. However, these contrastively trained
vision-language models often fail on compositional visio-linguistic tasks such
as Winoground with performance equivalent to random chance. In our paper, we
address this issue and propose a sample-efficient light-weight method called
SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
of CLIP. The core idea of our method is to use differentiable image
parameterizations to fine-tune CLIP with a distillation objective from large
text-to-image generative models such as Stable-Diffusion which are relatively
good at visio-linguistic reasoning tasks. On the challenging Winoground
compositional reasoning benchmark, our method improves the absolute
visio-linguistic performance of different CLIP models by up to 7%, while on the
ARO dataset, our method improves the visio-linguistic performance by upto 3%.
As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
that the zero-shot performance improves marginally on a variety of downstream
datasets. Our method reinforces that carefully designed distillation objectives
from generative models can be leveraged to extend existing contrastive
image-text models with improved visio-linguistic reasoning capabilities
DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability
Recently, large-scale diffusion models, e.g., Stable diffusion and DallE2,
have shown remarkable results on image synthesis. On the other hand,
large-scale cross-modal pre-trained models (e.g., CLIP, ALIGN, and FILIP) are
competent for various downstream tasks by learning to align vision and language
embeddings. In this paper, we explore the possibility of jointly modeling
generation and discrimination. Specifically, we propose DiffDis to unify the
cross-modal generative and discriminative pretraining into one single framework
under the diffusion process. DiffDis first formulates the image-text
discriminative problem as a generative diffusion process of the text embedding
from the text encoder conditioned on the image. Then, we propose a novel
dual-stream network architecture, which fuses the noisy text embedding with the
knowledge of latent images from different scales for image-text discriminative
learning. Moreover, the generative and discriminative tasks can efficiently
share the image-branch network structure in the multi-modality model.
Benefiting from diffusion-based unified training, DiffDis achieves both better
generation ability and cross-modal semantic alignment in one architecture.
Experimental results show that DiffDis outperforms single-task models on both
the image generation and the image-text discriminative tasks, e.g., 1.65%
improvement on average accuracy of zero-shot classification over 12 datasets
and 2.42 improvement on FID of zero-shot image synthesis.Comment: ICCV202
Generative and Discriminative Text Classification with Recurrent Neural Networks
We empirically characterize the performance of discriminative and generative
LSTM models for text classification. We find that although RNN-based generative
models are more powerful than their bag-of-words ancestors (e.g., they account
for conditional dependencies across words in a document), they have higher
asymptotic error rates than discriminatively trained RNN models. However we
also find that generative models approach their asymptotic error rate more
rapidly than their discriminative counterparts---the same pattern that Ng &
Jordan (2001) proved holds for linear classification models that make more
naive conditional independence assumptions. Building on this finding, we
hypothesize that RNN-based generative classification models will be more robust
to shifts in the data distribution. This hypothesis is confirmed in a series of
experiments in zero-shot and continual learning settings that show that
generative models substantially outperform discriminative models
Generating Visual Representations for Zero-Shot Classification
This paper addresses the task of learning an image clas-sifier when some
categories are defined by semantic descriptions only (e.g. visual attributes)
while the others are defined by exemplar images as well. This task is often
referred to as the Zero-Shot classification task (ZSC). Most of the previous
methods rely on learning a common embedding space allowing to compare visual
features of unknown categories with semantic descriptions. This paper argues
that these approaches are limited as i) efficient discrimi-native classifiers
can't be used ii) classification tasks with seen and unseen categories
(Generalized Zero-Shot Classification or GZSC) can't be addressed efficiently.
In contrast , this paper suggests to address ZSC and GZSC by i) learning a
conditional generator using seen classes ii) generate artificial training
examples for the categories without exemplars. ZSC is then turned into a
standard supervised learning problem. Experiments with 4 generative models and
5 datasets experimentally validate the approach, giving state-of-the-art
results on both ZSC and GZSC
- …