19 research outputs found
Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification
© 2020, Springer Nature Switzerland AG. Zero-shot learning strives to classify unseen categories for which no data is available during training. In the generalized variant, the test samples can further belong to seen or unseen categories. The state-of-the-art relies on Generative Adversarial Networks that synthesize unseen class features by leveraging class-specific semantic embeddings. During training, they generate semantically consistent features, but discard this constraint during feature synthesis and classification. We propose to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification. We first introduce a feedback loop, from a semantic embedding decoder, that iteratively refines the generated features during both the training and feature synthesis stages. The synthesized features together with their corresponding latent embeddings from the decoder are then transformed into discriminative features and utilized during classification to reduce ambiguities among categories. Experiments on (generalized) zero-shot object and action classification reveal the benefit of semantic consistency and iterative feedback, outperforming existing methods on six zero-shot learning benchmarks. Source code at https://github.com/akshitac8/tfvaegan
PS-ARM: An End-to-End Attention-aware Relation Mixer Network for Person Search
Person search is a challenging problem with various real-world applications,
that aims at joint person detection and re-identification of a query person
from uncropped gallery images. Although, the previous study focuses on rich
feature information learning, it is still hard to retrieve the query person due
to the occurrence of appearance deformations and background distractors. In
this paper, we propose a novel attention-aware relation mixer (ARM) module for
person search, which exploits the global relation between different local
regions within RoI of a person and make it robust against various appearance
deformations and occlusion. The proposed ARM is composed of a relation mixer
block and a spatio-channel attention layer. The relation mixer block introduces
a spatially attended spatial mixing and a channel-wise attended channel mixing
for effectively capturing discriminative relation features within an RoI. These
discriminative relation features are further enriched by introducing a
spatio-channel attention where the foreground and background discriminability
is empowered in a joint spatio-channel space. Our ARM module is generic and it
does not rely on fine-grained supervision or topological assumptions, hence
being easily integrated into any Faster R-CNN based person search methods.
Comprehensive experiments are performed on two challenging benchmark datasets:
CUHKSYSU and PRW. Our PS-ARM achieves state-of-the-art performance on both
datasets. On the challenging PRW dataset, our PS-ARM achieves an absolute gain
of 5 in the mAP score over SeqNet, while operating at a comparable speed.Comment: Paper accepted in ACCV 202
ECCV (22) - Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification
Zero-shot learning strives to classify unseen categories for which no data is
available during training. In the generalized variant, the test samples can
further belong to seen or unseen categories. The state-of-the-art relies on
Generative Adversarial Networks that synthesize unseen class features by
leveraging class-specific semantic embeddings. During training, they generate
semantically consistent features, but discard this constraint during feature
synthesis and classification. We propose to enforce semantic consistency at all
stages of (generalized) zero-shot learning: training, feature synthesis and
classification. We first introduce a feedback loop, from a semantic embedding
decoder, that iteratively refines the generated features during both the
training and feature synthesis stages. The synthesized features together with
their corresponding latent embeddings from the decoder are then transformed
into discriminative features and utilized during classification to reduce
ambiguities among categories. Experiments on (generalized) zero-shot object and
action classification reveal the benefit of semantic consistency and iterative
feedback, outperforming existing methods on six zero-shot learning benchmarks.
Source code at https://github.com/akshitac8/tfvaegan.Comment: Accepted for publication at ECCV 202
Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification
Zero-shot learning strives to classify unseen categories for which no data is
available during training. In the generalized variant, the test samples can
further belong to seen or unseen categories. The state-of-the-art relies on
Generative Adversarial Networks that synthesize unseen class features by
leveraging class-specific semantic embeddings. During training, they generate
semantically consistent features, but discard this constraint during feature
synthesis and classification. We propose to enforce semantic consistency at all
stages of (generalized) zero-shot learning: training, feature synthesis and
classification. We first introduce a feedback loop, from a semantic embedding
decoder, that iteratively refines the generated features during both the
training and feature synthesis stages. The synthesized features together with
their corresponding latent embeddings from the decoder are then transformed
into discriminative features and utilized during classification to reduce
ambiguities among categories. Experiments on (generalized) zero-shot object and
action classification reveal the benefit of semantic consistency and iterative
feedback, outperforming existing methods on six zero-shot learning benchmarks.
Source code at https://github.com/akshitac8/tfvaegan.Comment: Accepted for publication at ECCV 202
D2-Net: Weakly-Supervised Action Localization via Discriminative Embeddings and Denoised Activations
This work proposes a weakly-supervised temporal action localization
framework, called D2-Net, which strives to temporally localize actions using
video-level supervision. Our main contribution is the introduction of a novel
loss formulation, which jointly enhances the discriminability of latent
embeddings and robustness of the output temporal class activations with respect
to foreground-background noise caused by weak supervision. The proposed
formulation comprises a discriminative and a denoising loss term for enhancing
temporal action localization. The discriminative term incorporates a
classification loss and utilizes a top-down attention mechanism to enhance the
separability of latent foreground-background embeddings. The denoising loss
term explicitly addresses the foreground-background noise in class activations
by simultaneously maximizing intra-video and inter-video mutual information
using a bottom-up attention mechanism. As a result, activations in the
foreground regions are emphasized whereas those in the background regions are
suppressed, thereby leading to more robust predictions. Comprehensive
experiments are performed on two benchmarks: THUMOS14 and ActivityNet1.2. Our
D2-Net performs favorably in comparison to the existing methods on both
datasets, achieving gains as high as 3.6% in terms of mean average precision on
THUMOS14
Generative Multi-Label Zero-Shot Learning
Multi-label zero-shot learning strives to classify images into multiple
unseen categories for which no data is available during training. The test
samples can additionally contain seen categories in the generalized variant.
Existing approaches rely on learning either shared or label-specific attention
from the seen classes. Nevertheless, computing reliable attention maps for
unseen classes during inference in a multi-label setting is still a challenge.
In contrast, state-of-the-art single-label generative adversarial network (GAN)
based approaches learn to directly synthesize the class-specific visual
features from the corresponding class attribute embeddings. However,
synthesizing multi-label features from GANs is still unexplored in the context
of zero-shot setting. In this work, we introduce different fusion approaches at
the attribute-level, feature-level and cross-level (across attribute and
feature-levels) for synthesizing multi-label features from their corresponding
multi-label class embedding. To the best of our knowledge, our work is the
first to tackle the problem of multi-label feature synthesis in the
(generalized) zero-shot setting. Comprehensive experiments are performed on
three zero-shot image classification benchmarks: NUS-WIDE, Open Images and MS
COCO. Our cross-level fusion-based generative approach outperforms the
state-of-the-art on all three datasets. Furthermore, we show the generalization
capabilities of our fusion approach in the zero-shot detection task on MS COCO,
achieving favorable performance against existing methods. The source code is
available at https://github.com/akshitac8/Generative_MLZSL.Comment: 10 pages, source code is available at
https://github.com/akshitac8/Generative_MLZS