3D Garment Modelling - Creation of a Virtual Mannequin of the Human Body

This work presents the modelling and numerical simulation of mannequins as well as clothes in a 3D virtual environment. The paper explains garment modelling, with the objective of defining the development of a 3D body model that is useful and based on the demand of the garment industry. For this purpose a development strategy should be defined. Many scientists are working on the creative process for virtual garment development to create a garment directly on a 3D model of the human body, also called “ virtual tailoring”.The first strategic issue in this context that we present is that the model must necessarily incorporate the ease garment model associatively. These parameters define the priority concepts that are the draping and proper fit of the garment. The second point is that the transition between the 3D and 2D patterns, known as flattening 3D patterns, must be associative, precise and must take into account the real deformation of the fabric.Project entitled “Development of the research infrastructure of innovative techniques and technologies of the textile garment industry” CLO-2IN-TEX, financed by Operational Program Innovative Economy, 2007-2013, Action 2.

Virtual Try-On With Generative Adversarial Networks: A Taxonomical Survey

This chapter elaborates on using generative adversarial networks (GAN) for virtual try-on applications. It presents the first comprehensive survey on this topic. Virtual try-on represents a practical application of GANs and pixel translation, which improves on the techniques of virtual try-on prior to these new discoveries. This survey details the importance of virtual try-on systems and the history of virtual try-on; shows how GANs, pixel translation, and perceptual losses have influenced the field; and summarizes the latest research in creating virtual try-on systems. Additionally, the authors present the future directions of research to improve virtual try-on systems by making them usable, faster, more effective. By walking through the steps of virtual try-on from start to finish, the chapter aims to expose readers to key concepts shared by many GAN applications and to give readers a solid foundation to pursue further topics in GANs

Modelling 3D humans : pose, shape, clothing and interactions

Digital humans are increasingly becoming a part of our lives with applications like animation, gaming, virtual try-on, Metaverse and much more. In recent years there has been a great push to make our models of digital humans as real as possible. In this thesis we present methodologies to model two key characteristics of real humans, their appearance and actions. This thesis covers four innovations: (i) MGN, the first approach to reconstruct 3D garments and body shape underneath, as separate meshes, from a few RGB images of a person. This allows, for the first time, real world applications like texture transfer, garment transfer and virtual try-on in 3D, using just images. (ii) IPNet, a neural network, that leverages implicit functions for detailed reconstruction and registers the reconstructed mesh with the parametric SMPL model to make it controllable for real world tasks like animation and editing. (iii) LoopReg, a novel formulation that makes 3D registration task end-to-end differentiable for the first time. Semi-supervised LoopReg outperforms contemporary supervised methods using ∼100x less supervised data. (iv) BEHAVE the first dataset and method to track full body real interactions between humans and movable objects. All our code, MGN digital wardrobe and BEHAVE dataset are publicly available for further research.Digital humans are increasingly becoming a part of our lives with applications like animation, gaming, virtual try-on, Metaverse and much more. In recent years there has been a great push to make our models of digital humans as real as possible. In this thesis we present methodologies to model two key characteristics of real humans, their appearance and actions. This thesis covers four innovations: (i) MGN, the first approach to reconstruct 3D garments and body shape underneath, as separate meshes, from a few RGB images of a person. This allows, for the first time, real world applications like texture transfer, garment transfer and virtual try-on in 3D, using just images. (ii) IPNet, a neural network, that leverages implicit functions for detailed reconstruction and registers the reconstructed mesh with the parametric SMPL model to make it controllable for real world tasks like animation and editing. (iii) LoopReg, a novel formulation that makes 3D registration task end-to-end differentiable for the first time. Semi-supervised LoopReg outperforms contemporary supervised methods using ∼100x less supervised data. (iv) BEHAVE the first dataset and method to track full body real interactions between humans and movable objects. All our code, MGN digital wardrobe and BEHAVE dataset are publicly available for further research.Der digitale Mensch wird immer mehr zu einem Teil unseres Lebens mit Anwendungen wie Animation, Spielen, virtuellem Ausprobieren, Metaverse und vielem mehr. In den letzten Jahren wurden große Anstrengungen unternommen, um unsere Modelle digitaler Menschen so real wie möglich zu gestalten. In dieser Arbeit stellen wir Methoden zur Modellierung von zwei Schlüsseleigenschaften echter Menschen vor: ihr Aussehen und ihre Handlungen. Wir schlagen MGN vor, den ersten Ansatz zur Rekonstruktion von 3D-Kleidungsstücken und der darunter liegenden Körperform als separate Netze aus einigen wenigen RGB-Bildern einer Person. Wir erweitern das weit verbreitete SMPL-Körpermodell, das nur unbekleidete Formen darstellt, um auch Kleidungsstücke zu erfassen (SMPL+G). SMPL+G kann mit Kleidungsstücken bekleidet werden, die entsprechend dem SMPL-Modell posiert und geformt werden können. Dies ermöglicht zum ersten Mal reale Anwendungen wie Texturübertragung, Kleidungsübertragung und virtuelle Anprobe in 3D, wobei nur Bilder verwendet werden. Wir unterstreichen auch die entscheidende Einschränkung der netzbasierten Darstellung für digitale Menschen, nämlich die Fähigkeit, hochfrequente Details darzustellen. Daher untersuchen wir die neue implizite funktionsbasierte Darstellung als Alternative zur netzbasierten Darstellung (einschließlich parametrischer Modelle wie SMPL) für digitale Menschen. Typischerweise mangelt es den Methoden, die auf letzteren basieren, an Details, während ersteren die Kontrolle fehlt. Wir schlagen IPNet vor, ein neuronales Netzwerk, das implizite Funktionen für eine detaillierte Rekonstruktion nutzt und das rekonstruierte Netz mit dem parametrischen SMPL-Modell registriert, um es kontrollierbar zu machen. Auf diese Weise wird das Beste aus beiden Welten genutzt. Wir untersuchen den Prozess der Registrierung eines parametrischen Modells, wie z. B. SMPL, auf ein 3D-Netz. Dieses jahrzehntealte Problem im Bereich der Computer Vision und der Graphik erfordert in der Regel einen zweistufigen Prozess: i) Herstellung von Korrespondenzen zwischen dem Modell und dem Netz, und ii) Optimierung des Modells, um den Abstand zwischen den entsprechenden Punkten zu minimieren. Dieser zweistufige Prozess ist nicht durchgängig differenzierbar. Wir schlagen LoopReg vor, das eine neue, auf impliziten Funktionen basierende Darstellung des Modells verwendet und die Registrierung differenzierbar macht. Semi-überwachtes LoopReg übertrifft aktuelle überwachte Methoden mit ∼100x weniger überwachten Daten. Die Modellierung des menschlichen Aussehens ist notwendig, aber nicht ausreichend, um realistische digitale Menschen zu schaffen. Wir müssen nicht nur modellieren, wie Menschen aussehen, sondern auch, wie sie mit ihren umgebenden Objekten interagieren. Zu diesem Zweck präsentieren wir mit BEHAVE den ersten Datensatz von realen Ganzkörper-Interaktionen zwischen Menschen und beweglichen Objekten. Wir stellen segmentierte Multiview-RGBDFrames zusammen mit registrierten SMPL- und Objekt-Fits sowie Kontaktannotationen in 3D zur Verfügung. Der BEHAVE-Datensatz enthält ∼15k Frames und seine Erweiterung enthält ∼400k Frames mit Pseudo-Ground-Truth-Annotationen. Unsere BEHAVE-Methode verwendet diesen Datensatz, um ein neuronales Netz zu trainieren, das die Person, das Objekt und die Kontakte zwischen ihnen gemeinsam verfolgt. In dieser Arbeit untersuchen wir die oben genannten Ideen und bieten eine eingehende Analyse unserer Schlüsselideen und Designentscheidungen. Wir erörtern auch die Grenzen unserer Ideen und schlagen künftige Arbeiten vor, um nicht nur diese Grenzen anzugehen, sondern auch die Forschung weiter auszubauen. Unser gesamter Code, die digitale Garderobe und der Datensatz sind für weitere Forschungen öffentlich zugänglich

Significance of Skeleton-based Features in Virtual Try-On

The idea of \textit{Virtual Try-ON} (VTON) benefits e-retailing by giving an user the convenience of trying a clothing at the comfort of their home. In general, most of the existing VTON methods produce inconsistent results when a person posing with his arms folded i.e., bent or crossed, wants to try an outfit. The problem becomes severe in the case of long-sleeved outfits. As then, for crossed arm postures, overlap among different clothing parts might happen. The existing approaches, especially the warping-based methods employing \textit{Thin Plate Spline (TPS)} transform can not tackle such cases. To this end, we attempt a solution approach where the clothing from the source person is segmented into semantically meaningful parts and each part is warped independently to the shape of the person. To address the bending issue, we employ hand-crafted geometric features consistent with human body geometry for warping the source outfit. In addition, we propose two learning-based modules: a synthesizer network and a mask prediction network. All these together attempt to produce a photo-realistic, pose-robust VTON solution without requiring any paired training data. Comparison with some of the benchmark methods clearly establishes the effectiveness of the approach

Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis

Deep person generation has attracted extensive research attention due to its wide applications in virtual agents, video conferencing, online shopping and art/movie production. With the advancement of deep learning, visual appearances (face, pose, cloth) of a person image can be easily generated or manipulated on demand. In this survey, we first summarize the scope of person generation, and then systematically review recent progress and technical trends in deep person generation, covering three major tasks: talking-head generation (face), pose-guided person generation (pose) and garment-oriented person generation (cloth). More than two hundred papers are covered for a thorough overview, and the milestone works are highlighted to witness the major technical breakthrough. Based on these fundamental tasks, a number of applications are investigated, e.g., virtual fitting, digital human, generative data augmentation. We hope this survey could shed some light on the future prospects of deep person generation, and provide a helpful foundation for full applications towards digital human

LiveCap: Real-time Human Performance Capture from Monocular Video

We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting non-linear optimization problems per-frame are solved with specially-tailored data-parallel Gauss-Newton solvers. In order to achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture. Our method yields comparable accuracy with off-line performance capture techniques, while being orders of magnitude faster

Computer Vision in Human Analysis: From Face and Body to Clothes

For decades, researchers of different areas, ranging from artificial intelligence to computer vision, have intensively investigated human-centered data, i.e., data in which the human plays a significant role, acquired through a non-invasive approach, such as cameras. This interest has been largely supported by the highly informative nature of this kind of data, which provides a variety of information with which it is possible to understand many aspects including, for instance, the human body or the outward appearance. Some of the main tasks related to human analysis are focused on the body (e.g., human pose estimation and anthropocentric measurement estimation), the hands (e.g., gesture detection and recognition), the head (e.g., head pose estimation), or the face (e.g., emotion and expression recognition). Additional tasks are based on non-corporal elements, such as motion (e.g., action recognition and human behavior understanding) and clothes (e.g., garment-based virtual try-on and attribute recognition). Unfortunately, privacy issues severely limit the usage and the diffusion of this kind of data, making the exploitation of learning approaches challenging. In particular, privacy issues behind the acquisition and the use of human-centered data must be addressed by public and private institutions and companies. Thirteen high-quality papers have been published in this Special Issue and are summarized in the following: four of them are focused on the human face (facial geometry, facial landmark detection, and emotion recognition), two on eye image analysis (eye status classification and 3D gaze estimation), five on the body (pose estimation, conversational gesture analysis, and action recognition), and two on the outward appearance (transferring clothing styles and fashion-oriented image captioning). These numbers confirm the high interest in human-centered data and, in particular, the variety of real-world applications that it is possible to develop