2 research outputs found
SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
We present SPSG, a novel approach to generate high-quality, colored 3D models
of scenes from RGB-D scan observations by learning to infer unobserved scene
geometry and color in a self-supervised fashion. Our self-supervised approach
learns to jointly inpaint geometry and color by correlating an incomplete RGB-D
scan with a more complete version of that scan. Notably, rather than relying on
3D reconstruction losses to inform our 3D geometry and color reconstruction, we
propose adversarial and perceptual losses operating on 2D renderings in order
to achieve high-resolution, high-quality colored reconstructions of scenes.
This exploits the high-resolution, self-consistent signal from individual raw
RGB-D frames, in contrast to fused 3D reconstructions of the frames which
exhibit inconsistencies from view-dependent effects, such as color balancing or
pose inconsistencies. Thus, by informing our 3D scene generation directly
through 2D signal, we produce high-quality colored reconstructions of 3D
scenes, outperforming state of the art on both synthetic and real data.Comment: Video: https://youtu.be/1cj962m9zq
SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
We present a novel approach that converts partial and noisy RGB-D scans into
high-quality 3D scene reconstructions by inferring unobserved scene geometry.
Our approach is fully self-supervised and can hence be trained solely on
real-world, incomplete scans. To achieve self-supervision, we remove frames
from a given (incomplete) 3D scan in order to make it even more incomplete;
self-supervision is then formulated by correlating the two levels of
partialness of the same scan while masking out regions that have never been
observed. Through generalization across a large training set, we can then
predict 3D scene completion without ever seeing any 3D scan of entirely
complete geometry. Combined with a new 3D sparse generative neural network
architecture, our method is able to predict highly-detailed surfaces in a
coarse-to-fine hierarchical fashion, generating 3D scenes at 2cm resolution,
more than twice the resolution of existing state-of-the-art methods as well as
outperforming them by a significant margin in reconstruction quality.Comment: CVPR 2020; Video: https://youtu.be/rN6D3QmMNu