Scalable Inside-Out Image-Based Rendering

International audienceOur aim is to give users real-time free-viewpoint rendering of real indoor scenes, captured with off-the-shelf equipment such as a high-quality color camera and a commodity depth sensor. Image-based Rendering (IBR) can provide the realistic imagery required at real-time speed. For indoor scenes however, two challenges are especially prominent. First, the reconstructed 3D geometry must be compact, but faithful enough to respect occlusion relationships when viewed up close. Second, man-made materials call for view-dependent tex-turing, but using too many input photographs reduces performance. We customize a typical RGB-D 3D surface reconstruction pipeline to produce a coarse global 3D surface, and local, per-view geometry for each input image. Our tiled IBR preserves quality by economizing on the expected contributions that entire groups of input pixels make to a final image. The two components are designed to work together, giving real-time performance, while hardly sacrificing quality. Testing on a variety of challenging scenes shows that our inside-out IBR scales favorably with the number of input images

Neural View-Interpolation for Sparse Light Field Video

We suggest representing light field (LF) videos as "one-off" neural networks (NN), i.e., a learned mapping from view-plus-time coordinates to high-resolution color values, trained on sparse views. Initially, this sounds like a bad idea for three main reasons: First, a NN LF will likely have less quality than a same-sized pixel basis representation. Second, only few training data, e.g., 9 exemplars per frame are available for sparse LF videos. Third, there is no generalization across LFs, but across view and time instead. Consequently, a network needs to be trained for each LF video. Surprisingly, these problems can turn into substantial advantages: Other than the linear pixel basis, a NN has to come up with a compact, non-linear i.e., more intelligent, explanation of color, conditioned on the sparse view and time coordinates. As observed for many NN however, this representation now is interpolatable: if the image output for sparse view coordinates is plausible, it is for all intermediate, continuous coordinates as well. Our specific network architecture involves a differentiable occlusion-aware warping step, which leads to a compact set of trainable parameters and consequently fast learning and fast execution

Rendu basé image de fonds photographiques historiques massifs

This paper states an overview of my dissertation research centered on the continuous immersive visualization and navigation through time and space of large sets of historical photographs. The research aims for: (i) the treatment of scientific obstacles (e.g. data volume, heterogeneity, distortions, and uncertainties) that appear when old pictures are placed in today's environment; (ii) the visualization (saliently and spatially) of these photos. The main model of the study is image-based rendering IBR, because of its capacity to use imprecise or non-existent geometry (i.e. since a modern 3D scene may differ from a historical one, due to environmental changes over time). The findings of this work may contribute significantly by extending the current IBR models and providing a new innovative way to examine these massive and heterogeneous datasets.Cet article donne un aperçu de ma thèse de doctorat centrée sur la visualisation immersive continue et la navigation dans le temps et dans l’espace de grands ensembles de photographies historiques. Les objectifs de la recherche sont les suivants: (i) traitement des obstacles scientifiques (volume de données, hétérogénéité, distorsions et incertitudes, par exemple) qui apparaissent lorsque de vieilles images sont placées dans l’environnement actuel; (ii) la visualisation (de manière saillante et spatiale) de ces photos. Le modèle principal de l’étude est le rendu IBR basé sur l’image, en raison de sa capacité à utiliser une géométrie imprécise ou inexistante (c’est-à-dire qu’une scène 3D moderne peut différer de l’historique, en raison des changements environnementaux survenus au fil du temps). Les résultats de ce travail pourraient contribuer de manière significative en étendant les modèles IBR actuels et en fournissant un nouveau moyen innovant d’examiner ces ensembles de données massifs et hétérogènes

Learning to Predict Image-based Rendering Artifacts with Respect to a Hidden Reference Image

Image metrics predict the perceived per-pixel difference between a reference image and its degraded (e. g., re-rendered) version. In several important applications, the reference image is not available and image metrics cannot be applied. We devise a neural network architecture and training procedure that allows predicting the MSE, SSIM or VGG16 image difference from the distorted image alone while the reference is not observed. This is enabled by two insights: The first is to inject sufficiently many un-distorted natural image patches, which can be found in arbitrary amounts and are known to have no perceivable difference to themselves. This avoids false positives. The second is to balance the learning, where it is carefully made sure that all image errors are equally likely, avoiding false negatives. Surprisingly, we observe, that the resulting no-reference metric, subjectively, can even perform better than the reference-based one, as it had to become robust against mis-alignments. We evaluate the effectiveness of our approach in an image-based rendering context, both quantitatively and qualitatively. Finally, we demonstrate two applications which reduce light field capture time and provide guidance for interactive depth adjustment.Comment: 13 pages, 11 figure

Interactive Free-Viewpoint Video Generation

Background Free-viewpoint video (FVV) is processed video content in which viewers can freely select the viewing position and angle. FVV delivers an improved visual experience and can also help synthesize special effects and virtual reality content. In this paper, a complete FVV system is proposed to interactively control the viewpoints of video relay programs through multimedia terminals such as computers and tablets. Methods The hardware of the FVV generation system is a set of synchronously controlled cameras, and the software generates videos in novel viewpoints from the captured video using view interpolation. The interactive interface is designed to visualize the generated video in novel viewpoints and enable the viewpoint to be changed interactively. Results Experiments show that our system can synthesize plausible videos in intermediate viewpoints with a view range of up to 180°