19,048 research outputs found
Perspective Plane Program Induction from a Single Image
We study the inverse graphics problem of inferring a holistic representation
for natural images. Given an input image, our goal is to induce a
neuro-symbolic, program-like representation that jointly models camera poses,
object locations, and global scene structures. Such high-level, holistic scene
representations further facilitate low-level image manipulation tasks such as
inpainting. We formulate this problem as jointly finding the camera pose and
scene structure that best describe the input image. The benefits of such joint
inference are two-fold: scene regularity serves as a new cue for perspective
correction, and in turn, correct perspective correction leads to a simplified
scene structure, similar to how the correct shape leads to the most regular
texture in shape from texture. Our proposed framework, Perspective Plane
Program Induction (P3I), combines search-based and gradient-based algorithms to
efficiently solve the problem. P3I outperforms a set of baselines on a
collection of Internet images, across tasks including camera pose estimation,
global structure inference, and down-stream image manipulation tasks.Comment: CVPR 2020. First two authors contributed equally. Project page:
http://p3i.csail.mit.edu
What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification
Matching pedestrians across disjoint camera views, known as person
re-identification (re-id), is a challenging problem that is of importance to
visual recognition and surveillance. Most existing methods exploit local
regions within spatial manipulation to perform matching in local
correspondence. However, they essentially extract \emph{fixed} representations
from pre-divided regions for each image and perform matching based on the
extracted representation subsequently. For models in this pipeline, local finer
patterns that are crucial to distinguish positive pairs from negative ones
cannot be captured, and thus making them underperformed. In this paper, we
propose a novel deep multiplicative integration gating function, which answers
the question of \emph{what-and-where to match} for effective person re-id. To
address \emph{what} to match, our deep network emphasizes common local patterns
by learning joint representations in a multiplicative way. The network
comprises two Convolutional Neural Networks (CNNs) to extract convolutional
activations, and generates relevant descriptors for pedestrian matching. This
thus, leads to flexible representations for pair-wise images. To address
\emph{where} to match, we combat the spatial misalignment by performing
spatially recurrent pooling via a four-directional recurrent neural network to
impose spatial dependency over all positions with respect to the entire image.
The proposed network is designed to be end-to-end trainable to characterize
local pairwise feature interactions in a spatially aligned manner. To
demonstrate the superiority of our method, extensive experiments are conducted
over three benchmark data sets: VIPeR, CUHK03 and Market-1501.Comment: Published at Pattern Recognition, Elsevie
Data-Driven Shape Analysis and Processing
Data-driven methods play an increasingly important role in discovering
geometric, structural, and semantic relationships between 3D shapes in
collections, and applying this analysis to support intelligent modeling,
editing, and visualization of geometric data. In contrast to traditional
approaches, a key feature of data-driven approaches is that they aggregate
information from a collection of shapes to improve the analysis and processing
of individual shapes. In addition, they are able to learn models that reason
about properties and relationships of shapes without relying on hard-coded
rules or explicitly programmed instructions. We provide an overview of the main
concepts and components of these techniques, and discuss their application to
shape classification, segmentation, matching, reconstruction, modeling and
exploration, as well as scene analysis and synthesis, through reviewing the
literature and relating the existing works with both qualitative and numerical
comparisons. We conclude our report with ideas that can inspire future research
in data-driven shape analysis and processing.Comment: 10 pages, 19 figure
P{\O}DA: Prompt-driven Zero-shot Domain Adaptation
Domain adaptation has been vastly investigated in computer vision but still
requires access to target images at train time, which might be intractable in
some uncommon conditions. In this paper, we propose the task of `Prompt-driven
Zero-shot Domain Adaptation', where we adapt a model trained on a source domain
using only a single general textual description of the target domain, i.e., a
prompt. First, we leverage a pretrained contrastive vision-language model
(CLIP) to optimize affine transformations of source features, steering them
towards target text embeddings, while preserving their content and semantics.
Second, we show that augmented features can be used to perform zero-shot domain
adaptation for semantic segmentation. Experiments demonstrate that our method
significantly outperforms CLIP-based style transfer baselines on several
datasets for the downstream task at hand. Our prompt-driven approach even
outperforms one-shot unsupervised domain adaptation on some datasets, and gives
comparable results on others. Our code is available at
https://github.com/astra-vision/PODA.Comment: Project page: https://astra-vision.github.io/PODA
- …