A Generative Model of People in Clothing

We present the first image-based generative model of people in clothing for the full body. We sidestep the commonly used complex graphics rendering pipeline and the need for high-quality 3D scans of dressed people. Instead, we learn generative models from a large image database. The main challenge is to cope with the high variance in human pose, shape and appearance. For this reason, pure image-based approaches have not been considered so far. We show that this challenge can be overcome by splitting the generating process in two parts. First, we learn to generate a semantic segmentation of the body and clothing. Second, we learn a conditional model on the resulting segments that creates realistic images. The full model is differentiable and can be conditioned on pose, shape or color. The result are samples of people in different clothing items and styles. The proposed model can generate entirely new people with realistic clothing. In several experiments we present encouraging results that suggest an entirely data-driven approach to people generation is possible

Combining Data-Driven 2D and 3D Human Appearance Models

Detailed 2D and 3D body estimation of humans has many applications in our everyday life: interaction with machines, virtual try-on of fashion or product adjustments based on a body size estimate are just some examples. Two key components of such systems are: (1) detailed pose and shape estimation and (2) generation of images. Ideally, they should use 2D images as input signal so that they can be applied easily and on arbitrary digital images. Due to the high complexity of human appearance and the depth ambiguities in 2D space, data driven models are the tool at hand to design such methods. In this work, we consider two aspects of such systems: in the first part, we propose general optimization and implementation techniques for machine learning models and make them available in the form of software packages. In the second part, we present in multiple steps, how the detailed analysis and generation of human appearance based on digital 2D images can be realized. We work with two machine learning methods: Decision Forests and Artificial Neural Networks. The contribution of this thesis to the theory of Decision Forests consists of the introduction of a generalized entropy function that is efficient to evaluate and tunable to specific tasks and allows us to establish relations to frequently used heuristics. For both, Decision Forests and Neural Networks, we present methods for implementation and a software package. Existing methods for 3D body estimation from images usually estimate the 14 most important, pose defining points in 2D and convert them to a 3D `skeleton'. In this work we show that a carefully crafted energy function is sufficient to recover a full 3D body shape automatically from the keypoints. In this way, we devise the first fully automatic method estimating 3D body pose and shape from a 2D image. While this method successfully recovers a coarse 3D pose and shape, it is still a challenge to recover details such as body part rotations. However, for more detailed models, it would be necessary to annotate data with a very rich set of cues. This approach does not scale to large datasets, since the effort per image as well as the required quality could not be reached due to how hard it is to estimate the position of keypoints on the body surface. To solve this problem, we develop a method that can alternate between optimizing the 2D and 3D models, improving them iteratively. The labeling effort for humans remains low. At the same time, we create 2D models reasoning about factors more items than existing methods and we extend the 3D pose and body shape estimation to rotation and body extent. To generate images of people, existing methods usually work with 3D models that are hard to adjust and to use. In contrast, we develop a method that builds on the possibilities for automatic 3D body estimation: we use it to create a dataset of 3D bodies together with 2D clothes and cloth segments. With this information, we develop a data driven model directly producing 2D images of people. Only the broad interplay of 2D and 3D body and appearance models in different forms makes it possible to achieve a high level of detail for analysis and generation of human appearance. The developed techniques can in principle also be used for the analysis and generation of images of other creatures and objects.Detaillierte 2D und 3D Körperschätzung von Menschen hat vielfältige Anwendungen in unser aller Alltag: Interaktion mit Maschinen, virtuelle "Anprobe" von Kleidung oder direkte Produktanpassungen durch Schätzung der Körpermaße sind nur einige Beispiele. Dazu sind Methoden zur (1) detaillierten Posen- und Körpermaßschätzung und (2) Körperdarstellung notwendig. Idealerweise sollten sie digitale 2D Bilder als Ein- und Ausgabemedium verwenden, damit die einfache und allgemeine Anwendbarkeit gewährleistet bleibt. Aufgrund der hohen Komplexität des menschlichen Erscheinungsbilds und der Tiefenmehrdeutigkeit im 2D Raum sind datengetriebene Modelle ein naheliegendes Werkzeug, um solche Methoden zu entwerfen. In dieser Arbeit betrachten wir zwei Aspekte solcher Systeme: im ersten Teil entwickeln wir allgemein anwendbare Techniken für die Optimierung und Implementierung maschineller Lernmethoden und stellen diese in Form von Softwarepaketen bereit. Im zweiten Teil präsentieren wir in mehreren Schritten, wie die detaillierte Analyse und Darstellung von Menschen basierend auf digitalen 2D Bildern bewerkstelligt werden kann. Wir arbeiten dabei mit zwei Methoden zum maschinellen Lernen: Entscheidungswäldern und Künstlichen Neuronalen Netzen. Der Beitrag dieser Dissertation zur Theorie der Entscheidungswälder besteht in der Einführung einer verallgemeinerten Entropiefunktion, die effizient auswertbar und anpassbar ist und es ermöglicht, häufig verwendete Heuristiken besser einzuordnen. Für Entscheidungswälder und für Neuronale Netze beschreiben wir Methoden zur Implementierung und stellen jeweils ein Softwarepaket bereit, welches diese umsetzt. Die bisherigen Methoden zur 3D Körperschätzung aus Bildern beschränken sich auf die automatische Bestimmung der 14 wichtigsten 2D Punkte, welche die Pose definieren und deren Konvertierung in ein 3D "Skelett" Wir zeigen, dass durch die Optimierung einer fein abgestimmten Energiefunktion auch ein voller 3D Körper, nicht nur dessen Skelett, aus automatisch bestimmten 14 Punkten geschätzt werden kann. Damit beschreiben wir die erste vollautomatische Methode, die einen 3D Körper aus einem digitalen 2D Bild schätzt. Die detaillierte 3D Pose, beispielsweise mit Rotationen der Körperteile und die Beschaffenheit des untersuchten Körpers, ist damit noch nicht bestimmbar. Um detailliertere Modelle zu erstellen wäre es notwendig, Daten mit einem hohen Detailgrad zu annotieren. Dies skaliert jedoch nicht zu großen Datenmengen, da sowohl der Zeitaufwand pro Bild, als auch die notwendige Qualität aufgrund der schwierig einzuschätzenden exakten Positionen von Punkten auf der Körperoberfläche nicht erreicht werden können. Um dieses Problem zu lösen entwickeln wir eine Methode, die zwischen der Optimierung der 2D und 3D Modelle alterniert und diese wechselseitig verbessert. Dabei bleibt der Annotationsaufwand für Menschen gering. Gleichzeitig gelingt es, 2D Modelle mit einem Vielfachen an Details bisheriger Methoden zu erstellen und die Schätzung der 3D Pose und des Körpers auf Rotationen und Körperumfang zu erweitern. Um Bilder von Menschen zu generieren, beschränken sich existierende Methoden auf 3D Modelle, die schwer anzupassen und zu verwenden sind. Im Gegensatz dazu nutzen wir in dieser Arbeit einen Ansatz, der auf den Möglichkeiten zur automatischen 3D Posenschätzung basiert: wir nutzen sie, um einen Datensatz aus 3D Körpern mit dazugehörigen 2D Kleidungen und Kleidungssegmenten zu erstellen. Dies erlaubt es uns, ein datengetriebenes Modell zu entwickeln, welches direkt 2D Bilder von Menschen erzeugt. Erst das vielfältige Zusammenspiel von 2D und 3D Körper- und Erscheinungsmodellen in verschiedenen Formen ermöglicht es uns, einen hohen Detailgrad sowohl bei der Analyse als auch der Generierung menschlicher Erscheinung zu erzielen. Die hierfür entwickelten Techniken sind prinzipiell auch für die Analyse und Generierung von Bildern anderer Lebewesen und Objekte anwendbar

Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation

Direct prediction of 3D body pose and shape remains a challenge even for highly parameterized deep learning models. Mapping from the 2D image space to the prediction space is difficult: perspective ambiguities make the loss function noisy and training data is scarce. In this paper, we propose a novel approach (Neural Body Fitting (NBF)). It integrates a statistical body model within a CNN, leveraging reliable bottom-up semantic body part segmentation and robust top-down body model constraints. NBF is fully differentiable and can be trained using 2D and 3D annotations. In detailed experiments, we analyze how the components of our model affect performance, especially the use of part segmentations as an explicit intermediate representation, and present a robust, efficiently trainable framework for 3D human pose estimation from 2D images with competitive results on standard benchmarks. Code will be made available at http://github.com/mohomran/neural_body_fittingComment: 3DV 201

Norm-Induced Entropies for Decision Forests

Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works mus

Neural Lens Modeling

Recent methods for 3D reconstruction and rendering increasingly benefit from end-to-end optimization of the entire image formation process. However, this approach is currently limited: effects of the optical hardware stack and in particular lenses are hard to model in a unified way. This limits the quality that can be achieved for camera calibration and the fidelity of the results of 3D reconstruction. In this paper, we propose NeuroLens, a neural lens model for distortion and vignetting that can be used for point projection and ray casting and can be optimized through both operations. This means that it can (optionally) be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction, e.g., while optimizing a radiance field. To evaluate the performance of our proposed model, we create a comprehensive dataset assembled from the Lensfun database with a multitude of lenses. Using this and other real-world datasets, we show that the quality of our proposed lens model outperforms standard packages as well as recent approaches while being much easier to use and extend. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.Comment: To be presented at CVPR 2023, Project webpage: https://neural-lens.github.i

Neural Assets: Volumetric Object Capture and Rendering for Interactive Environments

Creating realistic virtual assets is a time-consuming process: it usually involves an artist designing the object, then spending a lot of effort on tweaking its appearance. Intricate details and certain effects, such as subsurface scattering, elude representation using real-time BRDFs, making it impossible to fully capture the appearance of certain objects. Inspired by the recent progress of neural rendering, we propose an approach for capturing real-world objects in everyday environments faithfully and fast. We use a novel neural representation to reconstruct volumetric effects, such as translucent object parts, and preserve photorealistic object appearance. To support real-time rendering without compromising rendering quality, our model uses a grid of features and a small MLP decoder that is transpiled into efficient shader code with interactive framerates. This leads to a seamless integration of the proposed neural assets with existing mesh environments and objects. Thanks to the use of standard shader code rendering is portable across many existing hardware and software systems