127 research outputs found
Cross-Ray Neural Radiance Fields for Novel-view Synthesis from Unconstrained Image Collections
Neural Radiance Fields (NeRF) is a revolutionary approach for rendering
scenes by sampling a single ray per pixel and it has demonstrated impressive
capabilities in novel-view synthesis from static scene images. However, in
practice, we usually need to recover NeRF from unconstrained image collections,
which poses two challenges: 1) the images often have dynamic changes in
appearance because of different capturing time and camera settings; 2) the
images may contain transient objects such as humans and cars, leading to
occlusion and ghosting artifacts. Conventional approaches seek to address these
challenges by locally utilizing a single ray to synthesize a color of a pixel.
In contrast, humans typically perceive appearance and objects by globally
utilizing information across multiple pixels. To mimic the perception process
of humans, in this paper, we propose Cross-Ray NeRF (CR-NeRF) that leverages
interactive information across multiple rays to synthesize occlusion-free novel
views with the same appearances as the images. Specifically, to model varying
appearances, we first propose to represent multiple rays with a novel cross-ray
feature and then recover the appearance by fusing global statistics, i.e.,
feature covariance of the rays and the image appearance. Moreover, to avoid
occlusion introduced by transient objects, we propose a transient objects
handler and introduce a grid sampling strategy for masking out the transient
objects. We theoretically find that leveraging correlation across multiple rays
promotes capturing more global information. Moreover, extensive experimental
results on large real-world datasets verify the effectiveness of CR-NeRF.Comment: ICCV 2023 Ora
Generating Visually Aligned Sound from Videos
We focus on the task of generating sound from natural videos, and the sound
should be both temporally and content-wise aligned with visual signals. This
task is extremely challenging because some sounds generated \emph{outside} a
camera can not be inferred from video content. The model may be forced to learn
an incorrect mapping between visual content and these irrelevant sounds. To
address this challenge, we propose a framework named REGNET. In this framework,
we first extract appearance and motion features from video frames to better
distinguish the object that emits sound from complex background information. We
then introduce an innovative audio forwarding regularizer that directly
considers the real sound as input and outputs bottlenecked sound features.
Using both visual and bottlenecked sound features for sound prediction during
training provides stronger supervision for the sound prediction. The audio
forwarding regularizer can control the irrelevant sound component and thus
prevent the model from learning an incorrect mapping between video frames and
sound emitted by the object that is out of the screen. During testing, the
audio forwarding regularizer is removed to ensure that REGNET can produce
purely aligned sound only from visual features. Extensive evaluations based on
Amazon Mechanical Turk demonstrate that our method significantly improves both
temporal and content-wise alignment. Remarkably, our generated sound can fool
the human with a 68.12% success rate. Code and pre-trained models are publicly
available at https://github.com/PeihaoChen/regnetComment: Published in IEEE Transactions on Image Processing, 2020. Code,
pre-trained models and demo video: https://github.com/PeihaoChen/regne
Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment
Text-to-image synthesis has made encouraging progress and attracted lots of
public attention recently. However, popular evaluation metrics in this area,
like the Inception Score and Fr'echet Inception Distance, incur several issues.
First of all, they cannot explicitly assess the perceptual quality of generated
images and poorly reflect the semantic alignment of each text-image pair. Also,
they are inefficient and need to sample thousands of images to stabilise their
evaluation results. In this paper, we propose to evaluate text-to-image
generation performance by directly estimating the likelihood of the generated
images using a pre-trained likelihood-based text-to-image generative model,
i.e., a higher likelihood indicates better perceptual quality and better
text-image alignment. To prevent the likelihood of being dominated by the
non-crucial part of the generated image, we propose several new designs to
develop a credit assignment strategy based on the semantic and perceptual
significance of the image patches. In the experiments, we evaluate the proposed
metric on multiple popular text-to-image generation models and datasets in
accessing both the perceptual quality and the text-image alignment. Moreover,
it can successfully assess the generation ability of these models with as few
as a hundred samples, making it very efficient in practice
Internal Wasserstein Distance for Adversarial Attack and Defense
Deep neural networks (DNNs) are vulnerable to adversarial examples that can
trigger misclassification of DNNs but may be imperceptible to human perception.
Adversarial attack has been an important way to evaluate the robustness of
DNNs. Existing attack methods on the construction of adversarial examples use
such distance as a similarity metric to perturb samples. However, this
kind of metric is incompatible with the underlying real-world image formation
and human visual perception. In this paper, we first propose an internal
Wasserstein distance (IWD) to measure image similarity between a sample and its
adversarial example. We apply IWD to perform adversarial attack and defense.
Specifically, we develop a novel attack method by capturing the distribution of
patches in original samples. In this case, our approach is able to generate
semantically similar but diverse adversarial examples that are more difficult
to defend by existing defense methods. Relying on IWD, we also build a new
defense method that seeks to learn robust models to defend against unseen
adversarial examples. We provide both thorough theoretical and empirical
evidence to support our methods
- …