187 research outputs found
The Video Mesh: A Data Structure for Image-based Three-dimensional Video Editing
This paper introduces the video mesh, a data structure for representing video as 2.5D âpaper cutouts.â The video mesh allows interactive editing of moving objects and modeling of depth, which enables 3D effects and post-exposure camera control. The video mesh sparsely encodes optical flow as well as depth, and handles occlusion using local layering and alpha mattes. Motion is described by a sparse set of points tracked over time. Each point also stores a depth value. The video mesh is a triangulation over this point set and per-pixel information is obtained by interpolation. The user rotoscopes occluding contours and we introduce an algorithm to cut the video mesh along them. Object boundaries are refined with per-pixel alpha values. The video mesh is at its core a set of texture mapped triangles, we leverage graphics hardware to enable interactive editing and rendering of a variety of effects. We demonstrate the effectiveness of our representation with special effects such as 3D viewpoint changes, object insertion, depth-of-field manipulation, and 2D to 3D video conversion
Self-supervised Outdoor Scene Relighting
Outdoor scene relighting is a challenging problem that requires good
understanding of the scene geometry, illumination and albedo. Current
techniques are completely supervised, requiring high quality synthetic
renderings to train a solution. Such renderings are synthesized using priors
learned from limited data. In contrast, we propose a self-supervised approach
for relighting. Our approach is trained only on corpora of images collected
from the internet without any user-supervision. This virtually endless source
of training data allows training a general relighting solution. Our approach
first decomposes an image into its albedo, geometry and illumination. A novel
relighting is then produced by modifying the illumination parameters. Our
solution capture shadow using a dedicated shadow prediction map, and does not
rely on accurate geometry estimation. We evaluate our technique subjectively
and objectively using a new dataset with ground-truth relighting. Results show
the ability of our technique to produce photo-realistic and physically
plausible results, that generalizes to unseen scenes.Comment: Published in ECCV '20,
http://gvv.mpi-inf.mpg.de/projects/SelfRelight
Scalable, Detailed and Mask-Free Universal Photometric Stereo
In this paper, we introduce SDM-UniPS, a groundbreaking Scalable, Detailed,
Mask-free, and Universal Photometric Stereo network. Our approach can recover
astonishingly intricate surface normal maps, rivaling the quality of 3D
scanners, even when images are captured under unknown, spatially-varying
lighting conditions in uncontrolled environments. We have extended previous
universal photometric stereo networks to extract spatial-light features,
utilizing all available information in high-resolution input images and
accounting for non-local interactions among surface points. Moreover, we
present a new synthetic training dataset that encompasses a diverse range of
shapes, materials, and illumination scenarios found in real-world scenes.
Through extensive evaluation, we demonstrate that our method not only surpasses
calibrated, lighting-specific techniques on public benchmarks, but also excels
with a significantly smaller number of input images even without object masks.Comment: CVPR 2023 (Highlight). The source code will be available at
https://github.com/satoshi-ikehata/SDM-UniPS-CVPR202
- âŠ