63,004 research outputs found
Context-Aware Zero-Shot Recognition
We present a novel problem setting in zero-shot learning, zero-shot object
recognition and detection in the context. Contrary to the traditional zero-shot
learning methods, which simply infers unseen categories by transferring
knowledge from the objects belonging to semantically similar seen categories,
we aim to understand the identity of the novel objects in an image surrounded
by the known objects using the inter-object relation prior. Specifically, we
leverage the visual context and the geometric relationships between all pairs
of objects in a single image, and capture the information useful to infer
unseen categories. We integrate our context-aware zero-shot learning framework
into the traditional zero-shot learning techniques seamlessly using a
Conditional Random Field (CRF). The proposed algorithm is evaluated on both
zero-shot region classification and zero-shot detection tasks. The results on
Visual Genome (VG) dataset show that our model significantly boosts performance
with the additional visual context compared to traditional methods
Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
Large-scale vision-language models (VLMs), e.g., CLIP, learn broad visual
concepts from tedious training data, showing superb generalization ability.
Amount of prompt learning methods have been proposed to efficiently adapt the
VLMs to downstream tasks with only a few training samples. We introduce a novel
method to improve the prompt learning of vision-language models by
incorporating pre-trained large language models (LLMs), called Dual-Aligned
Prompt Tuning (DuAl-PT). Learnable prompts, like CoOp, implicitly model the
context through end-to-end training, which are difficult to control and
interpret. While explicit context descriptions generated by LLMs, like GPT-3,
can be directly used for zero-shot classification, such prompts are overly
relying on LLMs and still underexplored in few-shot domains. With DuAl-PT, we
propose to learn more context-aware prompts, benefiting from both explicit and
implicit context modeling. To achieve this, we introduce a pre-trained LLM to
generate context descriptions, and we encourage the prompts to learn from the
LLM's knowledge by alignment, as well as the alignment between prompts and
local image features. Empirically, DuAl-PT achieves superior performance on 11
downstream datasets on few-shot recognition and base-to-new generalization.
Hopefully, DuAl-PT can serve as a strong baseline. Code will be available
Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions
We aim for zero-shot localization and classification of human actions in
video. Where traditional approaches rely on global attribute or object
classification scores for their zero-shot knowledge transfer, our main
contribution is a spatial-aware object embedding. To arrive at spatial
awareness, we build our embedding on top of freely available actor and object
detectors. Relevance of objects is determined in a word embedding space and
further enforced with estimated spatial preferences. Besides local object
awareness, we also embed global object awareness into our embedding to maximize
actor and object interaction. Finally, we exploit the object positions and
sizes in the spatial-aware embedding to demonstrate a new spatio-temporal
action retrieval scenario with composite queries. Action localization and
classification experiments on four contemporary action video datasets support
our proposal. Apart from state-of-the-art results in the zero-shot localization
and classification settings, our spatial-aware embedding is even competitive
with recent supervised action localization alternatives.Comment: ICC
Adaptive End-to-End Metric Learning for Zero-Shot Cross-Domain Slot Filling
Recently slot filling has witnessed great development thanks to deep learning
and the availability of large-scale annotated data. However, it poses a
critical challenge to handle a novel domain whose samples are never seen during
training. The recognition performance might be greatly degraded due to severe
domain shifts. Most prior works deal with this problem in a two-pass pipeline
manner based on metric learning. In practice, these dominant pipeline models
may be limited in computational efficiency and generalization capacity because
of non-parallel inference and context-free discrete label embeddings. To this
end, we re-examine the typical metric-based methods, and propose a new adaptive
end-to-end metric learning scheme for the challenging zero-shot slot filling.
Considering simplicity, efficiency and generalizability, we present a
cascade-style joint learning framework coupled with context-aware soft label
representations and slot-level contrastive representation learning to mitigate
the data and label shift problems effectively. Extensive experiments on public
benchmarks demonstrate the superiority of the proposed approach over a series
of competitive baselines.Comment: Accepted to EMNLP 2023 (Main, Long Paper
- …