Neural Texture Puppeteer: A Framework for Neural Geometry and Texture Rendering of Articulated Shapes, Enabling Re-Identification at Interactive Speed

In this paper, we present a neural rendering pipeline for textured articulated shapes that we call Neural Texture Puppeteer. Our method separates geometry and texture encoding. The geometry pipeline learns to capture spatial relationships on the surface of the articulated shape from ground truth data that provides this geometric information. A texture auto-encoder makes use of this information to encode textured images into a global latent code. This global texture embedding can be efficiently trained separately from the geometry, and used in a downstream task to identify individuals. The neural texture rendering and the identification of individuals run at interactive speeds. To the best of our knowledge, we are the first to offer a promising alternative to CNN- or transformer-based approaches for re-identification of articulated individuals based on neural rendering. Realistic looking novel view and pose synthesis for different synthetic cow textures further demonstrate the quality of our method. Restricted by the availability of ground truth data for the articulated shape's geometry, the quality for real-world data synthesis is reduced. We further demonstrate the flexibility of our model for real-world data by applying a synthetic to real-world texture domain shift where we reconstruct the texture from a real-world 2D RGB image. Thus, our method can be applied to endangered species where data is limited. Our novel synthetic texture dataset NePuMoo is publicly available to inspire further development in the field of neural rendering-based re-identification

SupeRVol: Super-Resolution Shape and Reflectance Estimation in Inverse Volume Rendering

We propose an end-to-end inverse rendering pipeline called SupeRVol that allows us to recover 3D shape and material parameters from a set of color images in a super-resolution manner. To this end, we represent both the bidirectional reflectance distribution function (BRDF) and the signed distance function (SDF) by multi-layer perceptrons. In order to obtain both the surface shape and its reflectance properties, we revert to a differentiable volume renderer with a physically based illumination model that allows us to decouple reflectance and lighting. This physical model takes into account the effect of the camera's point spread function thereby enabling a reconstruction of shape and material in a super-resolution quality. Experimental validation confirms that SupeRVol achieves state of the art performance in terms of inverse rendering quality. It generates reconstructions that are sharper than the individual input images, making this method ideally suited for 3D modeling from low-resolution imagery

Sparse Views, Near Light: A Practical Paradigm for Uncalibrated Point-light Photometric Stereo

Neural approaches have shown a significant progress on camera-based reconstruction. But they require either a fairly dense sampling of the viewing sphere, or pre-training on an existing dataset, thereby limiting their generalizability. In contrast, photometric stereo (PS) approaches have shown great potential for achieving high-quality reconstruction under sparse viewpoints. Yet, they are impractical because they typically require tedious laboratory conditions, are restricted to dark rooms, and often multi-staged, making them subject to accumulated errors. To address these shortcomings, we propose an end-to-end uncalibrated multi-view PS framework for reconstructing high-resolution shapes acquired from sparse viewpoints in a real-world environment. We relax the dark room assumption, and allow a combination of static ambient lighting and dynamic near LED lighting, thereby enabling easy data capture outside the lab. Experimental validation confirms that it outperforms existing baseline approaches in the regime of sparse viewpoints by a large margin. This allows to bring high-accuracy 3D reconstruction from the dark room to the real world, while maintaining a reasonable data capture complexity.Comment: Accepted in CVPR 202

An Introduction to Optimization Techniques in Computer Graphics

International audienceBackground: Many students in Computer Science do not have a sufficient background in applied mathematics to employ state-of-the-art optimization techniques and to judge the outcome of such techniques critically (e.g. regarding the stability/quality/accuracy of their output). At the same time, the use of optimization techniques in computer graphics is becoming ubiquitous. Treating optimization algorithms as a black box yields sub-optimal results at best. At worst, stability issues and convergence problems may prevent the solution of a problem or impede the general application of a method to a wide range of input, i.e. beyond the set of examples shown in a paper. The course will draw attention to these aspects and to current best practices. This will enable participants to judge articles that use optimization schemes critically and improve their own skill set

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Markerless methods for animal posture tracking have been developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple-views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For correspondence matching, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain correspondences accross views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator for Root Mean Square Error (RMSE) and Percentage of Correct Keypoints (PCK). We also showcase a novel use case where our model trained with data of single pigeons provides comparable results on data containing multiple pigeons. This can simplify the domain shift to new species because annotating single animal data is less labour intensive than multi-animal data. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 10 fps in 2D and 1.5 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we show that 3D-MuPPET also works in natural environments without model fine-tuning on additional annotations. To the best of our knowledge we are the first to present a framework for 2D/3D posture and trajectory tracking that works in both indoor and outdoor environments

Introducing total curvature for image processing

We introduce the novel continuous regularizer total curvature (TC) for images u: Ω → R. It is defined as the Menger-Melnikov curvature of the Radon measure |Du|, which can be understood as a measure theoretic formulation of curvature mathematically related to mean curvature. The functional is not convex, therefore we define a convex relaxation which yields a close approximation. Similar to the total variation, the relaxation can be written as the support functional of a convex set, which means that there are stable and efficient minimization algorithms available when it is used as a regularizer in image processing problems. Our current implementation can handle general inverse problems, inpainting and segmentation. We demonstrate in experiments and comparisons how the regularizer performs in practice. 1

SPACETIME-CONTINUOUS GEOMETRY MESHES FROM MULTI-VIEW VIDEO SEQUENCES

We reconstruct geometry for a time-varying scene given by a number of video sequences. The dynamic geometry is represented by a 3D hypersurface embedded in space-time. The intersection of the hypersurface with planes of constant time then yields the geometry at a single time instant. In this paper, we model the hypersurface with a collection of triangle meshes, one for each time frame. The photo-consistency error is measured by an error functional defined as an integral over the hypersurface. It can be minimized using a PDE driven surface evolution, which simultaneously optimizes space-time continuity as well. Compared to our previous implementation based on level sets, triangle meshes yield more accurate results, while requiring less memory and computation time. Meshes are also directly compatible with triangle-based rendering algorithms, so no additional post-processing is required. 1

An approach to vectorial total variation based on geometric measure theory

We analyze a previously unexplored generalization of the scalar total variation to vector-valued functions, which is motivated by geometric measure theory. A complete mathematical characterization is given, which proves important invariance properties as well as existence of solutions of the vectorial ROF model. As an important feature, there exists a dual formulation for the proposed vectorial total variation, which leads to a fast and stable minimization algorithm. The main difference to previous approaches with similar properties is that we penalize across a common edge direction for all channels, which is a major theoretical advantage. Experiments show that this leads to a significiantly better restoration of color edges in practice. 1

Space-Time Isosurface Evolution for Temporally Coherent 3D Reconstruction

We model the dynamic geometry of a time-varying scene as a 3D isosurface in space-time. The intersection of the isosurface with planes of constant time yields the geometry at a single time instant. An optimal fit of our model to multiple video sequences is defined as the minimum of an energy functional. This functional is given by an integral over the entire hypersurface, which is designed to optimize photo-consistency. A PDE-based evolution derived from the Euler-Lagrange equation maximizes consistency with all of the given video data simultaneously. The result is a 3D model of the scene which varies smoothly over time. The geometry reconstructed by this scheme is significantly better than results obtained by space-carving approaches that do not enforce temporal coherence

Superresolution texture maps for multiview reconstruction

We study the scenario of a multiview setting, where several calibrated views of a textured object with known surface geometry are available. The objective is to estimate a diffuse texture map as precisely as possible. A superresolution image formation model based on the camera properties leads to a total variation energy for the desired texture map, which can be recovered as the minimizer of the functional by solving the Euler-Lagrange equation on the surface. The PDE is transformed to planar texture space via an automatically created conformal atlas, where it can be solved using total variation deblurring. The proposed approach allows to recover a high-resolution, high-quality texture map even from lower-resolution photographs, which is of interest for a variety of image-based modeling applications. 1