Point-DynRF: Point-based Dynamic Radiance Fields from a Monocular Video

Dynamic radiance fields have emerged as a promising approach for generating novel views from a monocular video. However, previous methods enforce the geometric consistency to dynamic radiance fields only between adjacent input frames, making it difficult to represent the global scene geometry and degenerates at the viewpoint that is spatio-temporally distant from the input camera trajectory. To solve this problem, we introduce point-based dynamic radiance fields (\textbf{Point-DynRF}), a novel framework where the global geometric information and the volume rendering process are trained by neural point clouds and dynamic radiance fields, respectively. Specifically, we reconstruct neural point clouds directly from geometric proxies and optimize both radiance fields and the geometric proxies using our proposed losses, allowing them to complement each other. We validate the effectiveness of our method with experiments on the NVIDIA Dynamic Scenes Dataset and several causally captured monocular video clips.Comment: WACV202

DynIBaR: Neural Dynamic Image-Based Rendering

We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. State-of-the-art methods based on temporally varying Neural Radiance Fields (aka dynamic NeRFs) have shown impressive results on this task. However, for long videos with complex object motions and uncontrolled camera trajectories, these methods can produce blurry or inaccurate renderings, hampering their use in real-world applications. Instead of encoding the entire dynamic scene within the weights of an MLP, we present a new approach that addresses these limitations by adopting a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views in a scene-motion-aware manner. Our system retains the advantages of prior methods in its ability to model complex scenes and view-dependent effects, but also enables synthesizing photo-realistic novel views from long videos featuring complex scene dynamics with unconstrained camera trajectories. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets, and also apply our approach to in-the-wild videos with challenging camera and object motion, where prior methods fail to produce high-quality renderings. Our project webpage is at dynibar.github.io.Comment: Project page: dynibar.github.i

HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video

We introduce HOSNeRF, a novel 360{\deg} free-viewpoint rendering method that reconstructs neural radiance fields for dynamic human-object-scene from a single monocular in-the-wild video. Our method enables pausing the video at any frame and rendering all scene details (dynamic humans, objects, and backgrounds) from arbitrary viewpoints. The first challenge in this task is the complex object motions in human-object interactions, which we tackle by introducing the new object bones into the conventional human skeleton hierarchy to effectively estimate large object deformations in our dynamic human-object model. The second challenge is that humans interact with different objects at different times, for which we introduce two new learnable object state embeddings that can be used as conditions for learning our human-object representation and scene representation, respectively. Extensive experiments show that HOSNeRF significantly outperforms SOTA approaches on two challenging datasets by a large margin of 40% ~ 50% in terms of LPIPS. The code, data, and compelling examples of 360{\deg} free-viewpoint renderings from single videos will be released in https://showlab.github.io/HOSNeRF.Comment: Project page: https://showlab.github.io/HOSNeR

Raum-Zeit Interpolationstechniken

The photo-realistic modeling and animation of complex scenes in 3D requires a lot of work and skill of artists even with modern acquisition techniques. This is especially true if the rendering should additionally be performed in real-time. In this thesis we follow another direction in computer graphics to generate photo-realistic results based on recorded video sequences of one or multiple cameras. We propose several methods to handle scenes showing natural phenomena and also multi-view footage of general complex 3D scenes. In contrast to other approaches, we make use of relaxed geometric constraints and focus especially on image properties important to create perceptually plausible in-between images. The results are novel photo-realistic video sequences rendered in real-time allowing for interactive manipulation or to interactively explore novel view and time points.Das Modellieren und die Animation von 3D Szenen in fotorealistischer Qualität ist sehr arbeitsaufwändig, auch wenn moderne Verfahren benutzt werden. Wenn die Bilder in Echtzeit berechnet werden sollen ist diese Aufgabe um so schwieriger zu lösen. In dieser Dissertation verfolgen wir einen alternativen Ansatz der Computergrafik, um neue photorealistische Ergebnisse aus einer oder mehreren aufgenommenen Videosequenzen zu gewinnen. Es werden mehrere Methoden entwickelt die für natürlicher Phänomene und für generelle Szenen einsetzbar sind. Im Unterschied zu anderen Verfahren nutzen wir abgeschwächte geometrische Einschränkungen und berechnen eine genaue Lösung nur dort wo sie wichtig für die menschliche Wahrnehmung ist. Die Ergebnisse sind neue fotorealistische Videosequenzen, die in Echtzeit berechnet und interaktiv manipuliert, oder in denen neue Blick- und Zeitpunkte der Szenen frei erkundet werden können

Flexible Stereoscopic 3D Content Creation of Real World Scenes

We propose an alternative over current approaches to stereoscopic 3D video content creation based on a free-viewpoint video. Acquisition and editing is greatly simplified. Our method is suitable for arbitrary real-world scenes. From unsynchronized multi-view video footage, our approach renders high-quality stereo sequences without the need to explicitly reconstruct any scene depth or geometry. By allowing to freely edit viewpoint, slow motion, freeze-rotate shots, depth-of-field, and many more effects, the presented approach extends the possibilities in stereo 3D movie creation.In diesem Report schlagen wir eine Alternative zu gegenwärtig in der Produktion von stereoskopischen Filmen verwendeten Techniken vor. Unser Ansatz basiert auf der Verwendung eines Systems zur Blickpunktnavigation. Die Aufnahme und die Editierung der Stereodaten wird dadurch erheblich vereinfacht. Unsere Methode generiert qualitativ hochwertige Stereosequenzen ohne dabei Szenengeometrie oder Szenetiefe explizit zu rekonstruieren. Das Verfahren ermöglicht es den Blickpunkt und die Wiedergabegeschwindigkeit zu ändern und visuelle Effekte zu integrieren, wodurch neue künstlerische Möglichkeiten in der stereoskopischen Filmproduktion erschlossen werden

Duality based optical flow algorithms with applications

We consider the popular TV-L1 optical flow formulation, and the so-called dual-ity based algorithm for minimizing the TV-L1 energy. The original formulation is extended to allow for vector valued images, and minimization results are given. In addition we consider di↵erent definitions of total variation regulariza-tion, and related formulations of the optical flow problem that may be used with a duality based algorithm. We present a highly optimized algorithmic setup to estimate optical flows, and give five novel applications. The first application is registration of medical images, where X-ray images of di↵erent hands, taken using di↵erent imaging devices are registered using a TV-L1 optical flow algo-rithm. We propose to regularize the input images, using sparsity enhancing regularization of the image gradient to improve registration results. The second application is registration of 2D chromatograms, where registration only have to be done in one of the two dimensions, resulting in a vector valued registration problem with values having several hundred dimensions. We propose a nove

Spacetime Tetrahedra: Bildbasierte Blickpunktnavigation durch Raum und Zeit

We present a purely image-based rendering system to viewpoint-navigate through space and time of arbitrary dynamic scenes. Unlike previous methods, our approach does not rely on synchronized and calibrated multi-video footage as input. Instead of estimating scene depth or reconstructing 3D geometry, our approach is based on dense image correspondences, treating view interpolation equally in space and time. In a nutshell, we tetrahedrally partition the volume spanned by camera directions and time, determine the warp field along each tetrahedral edge, and warp-blend-interpolate any viewpoint inside a tetrahedron from the four video frames representing its vertices. Besides fast and easy acquisition to make outdoor recordings feasible, our space-time symmetric approach allows for smooth interpolation of view perspective and time, i.e., for simultaneous free-viewpoint and slow motion rendering