1,645 research outputs found
An Internal Learning Approach to Video Inpainting
We propose a novel video inpainting algorithm that simultaneously
hallucinates missing appearance and motion (optical flow) information, building
upon the recent 'Deep Image Prior' (DIP) that exploits convolutional network
architectures to enforce plausible texture in static images. In extending DIP
to video we make two important contributions. First, we show that coherent
video inpainting is possible without a priori training. We take a generative
approach to inpainting based on internal (within-video) learning without
reliance upon an external corpus of visual data to train a one-size-fits-all
model for the large space of general videos. Second, we show that such a
framework can jointly generate both appearance and flow, whilst exploiting
these complementary modalities to ensure mutual consistency. We show that
leveraging appearance statistics specific to each video achieves visually
plausible results whilst handling the challenging problem of long-term
consistency.Comment: Accepted by ICCV 2019. Website:
https://cs.stanford.edu/~haotianz/publications/video_inpainting
Infusion: Internal Diffusion for Video Inpainting
Video inpainting is the task of filling a desired region in a video in a
visually convincing manner. It is a very challenging task due to the high
dimensionality of the signal and the temporal consistency required for
obtaining convincing results. Recently, diffusion models have shown impressive
results in modeling complex data distributions, including images and videos.
Diffusion models remain nonetheless very expensive to train and perform
inference with, which strongly restrict their application to video. We show
that in the case of video inpainting, thanks to the highly auto-similar nature
of videos, the training of a diffusion model can be restricted to the video to
inpaint and still produce very satisfying results. This leads us to adopt an
internal learning approch, which also allows for a greatly reduced network
size. We call our approach "Infusion": an internal learning algorithm for video
inpainting through diffusion. Due to our frugal network, we are able to propose
the first video inpainting approach based purely on diffusion. Other methods
require supporting elements such as optical flow estimation, which limits their
performance in the case of dynamic textures for example. We introduce a new
method for efficient training and inference of diffusion models in the context
of internal learning. We split the diffusion process into different learning
intervals which greatly simplifies the learning steps. We show qualititative
and quantitative results, demonstrating that our method reaches
state-of-the-art performance, in particular in the case of dynamic backgrounds
and textures.Comment: 12 pages, 8 figure
- …