Reconstructing head models from photographs for individualized 3D-audio processing

International audienceVisual fidelity and interactivity are the main goals in Computer Graphics research, but recently also audio is assuming an important role. Binaural rendering can provide extremely pleasing and realistic three-dimensional sound, but to achieve best results it's necessary either to measure or to estimate individual Head Related Transfer Function (HRTF). This function is strictly related to the peculiar features of ears and face of the listener. Recent sound scattering simulation techniques can calculate HRTF starting from an accurate 3D model of a human head. Hence, the use of binaural rendering on large scale (i.e. video games, entertainment) could depend on the possibility to produce a sufficiently accurate 3D model of a human head, starting from the smallest possible input. In this paper we present a completely automatic system, which produces a 3D model of a head starting from simple input data (five photos and some key-points indicated by user). The geometry is generated by extracting information from images and accordingly deforming a 3D dummy to reproduce user head features. The system proves to be fast, automatic, robust and reliable: geometric validation and preliminary assessments show that it can be accurate enough for HRTF calculation

State of the Art in Example-based Texture Synthesis

International audienceRecent years have witnessed significant progress in example-based texture synthesis algorithms. Given an example texture, these methods produce a larger texture that is tailored to the user's needs. In this state-of-the-art report, we aim to achieve three goals: (1) provide a tutorial that is easy to follow for readers who are not already familiar with the subject, (2) make a comprehensive survey and comparisons of different methods, and (3) sketch a vision for future work that can help motivate and guide readers that are interested in texture synthesis research. We cover fundamental algorithms as well as extensions and applications of texture synthesis

Self-Supervised Shape and Appearance Modeling via Neural Differentiable Graphics

Inferring 3D shape and appearance from natural images is a fundamental challenge in computer vision. Despite recent progress using deep learning methods, a key limitation is the availability of annotated training data, as acquisition is often very challenging and expensive, especially at a large scale. This thesis proposes to incorporate physical priors into neural networks that allow for self-supervised learning. As a result, easy-to-access unlabeled data can be used for model training. In particular, novel algorithms in the context of 3D reconstruction and texture/material synthesis are introduced, where only image data is available as supervisory signal. First, a method that learns to reason about 3D shape and appearance solely from unstructured 2D images, achieved via differentiable rendering in an adversarial fashion, is proposed. As shown next, learning from videos significantly improves 3D reconstruction quality. To this end, a novel ray-conditioned warp embedding is proposed that aggregates pixel-wise features from multiple source images. Addressing the challenging task of disentangling shape and appearance, first a method that enables 3D texture synthesis independent of shape or resolution is presented. For this purpose, 3D noise fields of different scales are transformed into stationary textures. The method is able to produce 3D textures, despite only requiring 2D textures for training. Lastly, the surface characteristics of textures under different illumination conditions are modeled in the form of material parameters. Therefore, a self-supervised approach is proposed that has no access to material parameters but only flash images. Similar to the previous method, random noise fields are reshaped to material parameters, which are conditioned to replicate the visual appearance of the input under matching light

Patient-specific anatomical illustration via model-guided texture synthesis

Medical illustrations can make powerful use of textures to attractively, effectively, and efficiently visualize the appearance of the surface or cut surface of anatomic structures. It can do this by implying the anatomic structure's physical composition and clarifying its identity and 3-D shape. Current visualization methods are only capable of conveying detailed information about the orientation, internal structure, and other local properties of the anatomical objects for a typical individual, not for a particular patient. Although one can derive the shape of the individual patient's object from CT or MRI, it is important to apply these illustrative techniques to those particular shapes. In this research patient-specific anatomical illustrations are created by model-guided texture synthesis (MGTS). Given 2D exemplar textures and model-based guidance information as input, MGTS uses exemplar-based texture synthesis techniques to create patient-specific surface and solid textures. It consists of three main components. The first component includes a novel texture metamorphosis approach for creating interpolated exemplar textures given two exemplar textures. This component uses an energy optimization scheme derived from optimal control principles that utilizes intensity and structure information in obtaining the transformation. The second component consists of creating the model-based guidance information, such as directions and layers, for that specific model. This component uses coordinates implied by discrete medial 3D anatomical models (m-reps). The last component accomplishes exemplar-based texture synthesis by textures whose characteristics are spatially variant on and inside the 3D models. It considers the exemplar textures from the first component and guidance information from the second component in synthesizing high-quality, high-resolution solid and surface textures. Patient-specific illustrations with a variety of textures for different anatomical models, such as muscles and bones, are shown to be useful for our clinician to comprehend the shape of the models under radiation dose and to distinguish the models from one another

Uses of uncalibrated images to enrich 3D models information

The decrease in costs of semi-professional digital cameras has led to the possibility for everyone to acquire a very detailed description of a scene in a very short time. Unfortunately, the interpretation of the images is usually quite hard, due to the amount of data and the lack of robust and generic image analysis methods. Nevertheless, if a geometric description of the depicted scene is available, it gets much easier to extract information from 2D data. This information can be used to enrich the quality of the 3D data in several ways. In this thesis, several uses of sets of unregistered images for the enrichment of 3D models are shown. In particular, two possible fields of application are presented: the color acquisition, projection and visualization and the geometry modification. Regarding color management, several practical and cheap solutions to overcome the main issues in this field are presented. Moreover, some real applications, mainly related to Cultural Heritage, show that provided methods are robust and effective. In the context of geometry modification, two approaches are presented to modify already existing 3D models. In the first one, information extracted from images is used to deform a dummy model to obtain accurate 3D head models, used for simulation in the context of three-dimensional audio rendering. The second approach presents a method to fill holes in 3D models, with the use of registered images depicting a pattern projected on the real object. Finally, some useful indications about the possible future work in all the presented fields are given, in order to delineate the developments of this promising direction of research

Modélisation de fruits, de leur structure interne et de leurs défauts

One of the largest areas of research in computer graphics deals with natural phenomena representation. Over the years, as our understanding of nature grew, researchers started to propose new ways of simulating the various natural phenomena that we can observe in our everyday life. In this thesis, we focused on the representation of fruits. The fruit is a complex object. Depending on the desired accuracy, modeling a 3D fruit using classic 3D modeling software can become very tedious. We propose a model for generating vast varieties of fruits as well as their internal structure, thanks to the use of a single formal grammar. Each fruit that will be generated using our method will have global features that characterize its species, but it will also have local variations that are specific to it. The second part of ou work involves the representation of geometrical imperfections along the fruit. The fruit is the result of a series of physiological processes that strongly interact with each other. When one of these processes does not work the way it should, this dysfunction is materialized in the form of a shape defect. Our model introduces a simple approach, based on the use of grammars, which will allow us to apply variations on fruits in order to generate various categories of shape defects.La représentation de phénomènes naturels fait partie des domaines les plus complexes et les plus actifs de la recherche en informatique graphique. Notre compréhension de la nature s'améliorant au fil des années, les chercheurs ne cessent de proposer des nouveaux modèles, toujours plus pertinents les uns que les autres, et permettant de reproduire les différents phénomènes naturels que nous pouvons observer autour de nous, dans la vie de tous les jours. Nous nous sommes intéressés à la représentation du fruit et des différents éléments qui le caractérisent. Le fruit est un objet complexe et, en fonction de la précision requise, sa conception à l'aide de logiciels de modélisation 3D peut très vite devenir compliquée. Notre modèle permet de générer une grande variété de fruits de formes différentes ainsi que les différents éléments de leur structure interne et ce, grâce à l'utilisation d'une seule grammaire. Au sein d'une même espèce, les fruits générés seront tous différents au niveau de leur forme, tout en restant semblables. La seconde partie de nos travaux porte quant à elle sur la représentation des imperfections géométriques qui sont propres aux fruits. Les fruits sont le résultat de l'enchaînement d'un très grand nombre de processus physiologiques complexes qui interagissent fortement entre eux. Lorsque le bon fonctionnement de l'un de ces processus est compromis, cette anomalie se matérialise sur le fruit par l'apparition de défauts au niveau de sa forme. Notre modèle propose une approche simple, basée sur l'utilisation de grammaires, qui permettent d'altérer soit la forme générale d'un fruit soit des parties de sa surface

User-appropriate viewer for high resolution interactive engagement with 3D digital cultural artefacts.

The core mission of museums and cultural institutions is the preservation, study and presentation of cultural heritage content. In this technological age, the creation of digital datasets and archives has been widely adopted as one way of seeking to achieve some or all of these goals. However, there are many challenges with the use of these data, and in particular the large numbers of 3D digital artefacts that have been produced using methods such as non- contact laser scanning. As public expectation for more open access to information and innovative digital media increases, there are many issues that need to be rapidly addressed. The novel nature of 3D datasets and their visualisation presenting unique issues that impede use and dissemination. Key questions include the legal issues associated with 3D datasets created from cultural artefacts; the complex needs of users who are interacting with them; a lack of knowledge to texture and assess the visual quality of the datasets; and how the visual quality of the presented dataset relates to the perceptual experience of the user. This engineering doctorate, based on an industrial partnership with the National Museums of Liverpool and Conservation Technologies, investigates these questions and offers new ways of working with 3D cultural heritage datasets. The research outcomes in the thesis provide an improved understanding of the complexity of intellectual property law in relation to 3D cultural heritage datasets and how this impacts dissemination of these types of data. It also provides tools and techniques that can be used to understand the needs of a user when interacting with 3D cultural content. Additionally, the results demonstrate the importance of the relationship between texture and polygonal resolution and how this can affect the perceived visual experience of a visitor. It finds that there is an acceptable cost to texture and polygonal resolution to offer the best perceptual experience with 3D digital cultural heritage. The results also demonstrate that a non-textured mesh may be as highly received as a high resolution textured mesh. The research presented provides methodologies and guidelines to improve upon the dissemination and visualisation of 3D cultural content; enhancing and communicating the significance of their 3D collections to their physical and virtual visitors. Future opportunities and challenges for disseminating and visualising 3D cultural content are also discussed