304 research outputs found
Structure propagation for zero-shot learning
The key of zero-shot learning (ZSL) is how to find the information transfer
model for bridging the gap between images and semantic information (texts or
attributes). Existing ZSL methods usually construct the compatibility function
between images and class labels with the consideration of the relevance on the
semantic classes (the manifold structure of semantic classes). However, the
relationship of image classes (the manifold structure of image classes) is also
very important for the compatibility model construction. It is difficult to
capture the relationship among image classes due to unseen classes, so that the
manifold structure of image classes often is ignored in ZSL. To complement each
other between the manifold structure of image classes and that of semantic
classes information, we propose structure propagation (SP) for improving the
performance of ZSL for classification. SP can jointly consider the manifold
structure of image classes and that of semantic classes for approximating to
the intrinsic structure of object classes. Moreover, the SP can describe the
constrain condition between the compatibility function and these manifold
structures for balancing the influence of the structure propagation iteration.
The SP solution provides not only unseen class labels but also the relationship
of two manifold structures that encode the positive transfer in structure
propagation. Experimental results demonstrate that SP can attain the promising
results on the AwA, CUB, Dogs and SUN databases
DPF: Learning Dense Prediction Fields with Weak Supervision
Nowadays, many visual scene understanding problems are addressed by dense
prediction networks. But pixel-wise dense annotations are very expensive (e.g.,
for scene parsing) or impossible (e.g., for intrinsic image decomposition),
motivating us to leverage cheap point-level weak supervision. However, existing
pointly-supervised methods still use the same architecture designed for full
supervision. In stark contrast to them, we propose a new paradigm that makes
predictions for point coordinate queries, as inspired by the recent success of
implicit representations, like distance or radiance fields. As such, the method
is named as dense prediction fields (DPFs). DPFs generate expressive
intermediate features for continuous sub-pixel locations, thus allowing outputs
of an arbitrary resolution. DPFs are naturally compatible with point-level
supervision. We showcase the effectiveness of DPFs using two substantially
different tasks: high-level semantic parsing and low-level intrinsic image
decomposition. In these two cases, supervision comes in the form of
single-point semantic category and two-point relative reflectance,
respectively. As benchmarked by three large-scale public datasets
PASCALContext, ADE20K and IIW, DPFs set new state-of-the-art performance on all
of them with significant margins.
Code can be accessed at https://github.com/cxx226/DPF
See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data
Zero-shot point cloud segmentation aims to make deep models capable of
recognizing novel objects in point cloud that are unseen in the training phase.
Recent trends favor the pipeline which transfers knowledge from seen classes
with labels to unseen classes without labels. They typically align visual
features with semantic features obtained from word embedding by the supervision
of seen classes' annotations. However, point cloud contains limited information
to fully match with semantic features. In fact, the rich appearance information
of images is a natural complement to the textureless point cloud, which is not
well explored in previous literature. Motivated by this, we propose a novel
multi-modal zero-shot learning method to better utilize the complementary
information of point clouds and images for more accurate visual-semantic
alignment. Extensive experiments are performed in two popular benchmarks, i.e.,
SemanticKITTI and nuScenes, and our method outperforms current SOTA methods
with 52% and 49% improvement on average for unseen class mIoU, respectively.Comment: Accepted by ICCV 202
Transductive Zero-Shot Action Recognition by Word-Vector Embedding
The number of categories for action recognition is growing rapidly and it has
become increasingly hard to label sufficient training data for learning
conventional models for all categories. Instead of collecting ever more data
and labelling them exhaustively for all categories, an attractive alternative
approach is zero-shot learning" (ZSL). To that end, in this study we construct
a mapping between visual features and a semantic descriptor of each action
category, allowing new categories to be recognised in the absence of any visual
training data. Existing ZSL studies focus primarily on still images, and
attribute-based semantic representations. In this work, we explore word-vectors
as the shared semantic space to embed videos and category labels for ZSL action
recognition. This is a more challenging problem than existing ZSL of still
images and/or attributes, because the mapping between video spacetime features
of actions and the semantic space is more complex and harder to learn for the
purpose of generalising over any cross-category domain shift. To solve this
generalisation problem in ZSL action recognition, we investigate a series of
synergistic strategies to improve upon the standard ZSL pipeline. Most of these
strategies are transductive in nature which means access to testing data in the
training phase.Comment: Accepted by IJC
- …