3D Topology Transformation with Generative Adversarial Networks

Generation and transformation of images and videos using artificial intelligence have flourished over the past few years. Yet, there are only a few works aiming to produce creative 3D shapes, such as sculptures. Here we show a novel 3D-to-3D topology transformation method using Generative Adversarial Networks (GAN). We use a modified pix2pix GAN, which we call Vox2Vox, to transform the volumetric style of a 3D object while retaining the original object shape. In particular, we show how to transform 3D models into two new volumetric topologies - the 3D Network and the Ghirigoro. We describe how to use our approach to construct customized 3D representations. We believe that the generated 3D shapes are novel and inspirational. Finally, we compare the results between our approach and a baseline algorithm that directly convert the 3D shapes, without using our GAN

Data-driven approaches for interactive appearance editing

This thesis proposes several techniques for interactive editing of digital content and fast rendering of virtual 3D scenes. Editing of digital content - such as images or 3D scenes - is difficult, requires artistic talent and technical expertise. To alleviate these difficulties, we exploit data-driven approaches that use the easily accessible Internet data (e. g., images, videos, materials) to develop new tools for digital content manipulation. Our proposed techniques allow casual users to achieve high-quality editing by interactively exploring the manipulations without the need to understand the underlying physical models of appearance. First, the thesis presents a fast algorithm for realistic image synthesis of virtual 3D scenes. This serves as the core framework for a new method that allows artists to fine tune the appearance of a rendered 3D scene. Here, artists directly paint the final appearance and the system automatically solves for the material parameters that best match the desired look. Along this line, an example-based material assignment approach is proposed, where the 3D models of a virtual scene can be "materialized" simply by giving a guidance source (image/video). Next, the thesis proposes shape and color subspaces of an object that are learned from a collection of exemplar images. These subspaces can be used to constrain image manipulations to valid shapes and colors, or provide suggestions for manipulations. Finally, data-driven color manifolds which contain colors of a specific context are proposed. Such color manifolds can be used to improve color picking performance, color stylization, compression or white balancing.Diese Dissertation stellt Techniken zum interaktiven Editieren von digitalen Inhalten und zum schnellen Rendering von virtuellen 3D Szenen vor. Digitales Editieren - seien es Bilder oder dreidimensionale Szenen - ist kompliziert, benötigt künstlerisches Talent und technische Expertise. Um diese Schwierigkeiten zu relativieren, nutzen wir datengesteuerte Ansätze, die einfach zugängliche Internetdaten, wie Bilder, Videos und Materialeigenschaften, nutzen um neue Werkzeuge zur Manipulation von digitalen Inhalten zu entwickeln. Die von uns vorgestellten Techniken erlauben Gelegenheitsnutzern das Editieren in hoher Qualität, indem Manipulationsmöglichkeiten interaktiv exploriert werden können ohne die zugrundeliegenden physikalischen Modelle der Bildentstehung verstehen zu müssen. Zunächst stellen wir einen effizienten Algorithmus zur realistischen Bildsynthese von virtuellen 3D Szenen vor. Dieser dient als Kerngerüst einer Methode, die Nutzern die Feinabstimmung des finalen Aussehens einer gerenderten dreidimensionalen Szene erlaubt. Hierbei malt der Künstler direkt das beabsichtigte Aussehen und das System errechnet automatisch die zugrundeliegenden Materialeigenschaften, die den beabsichtigten Eigenschaften am nahesten kommen. Zu diesem Zweck wird ein auf Beispielen basierender Materialzuordnungsansatz vorgestellt, für den das 3D Model einer virtuellen Szene durch das simple Anführen einer Leitquelle (Bild, Video) in Materialien aufgeteilt werden kann. Als Nächstes schlagen wir Form- und Farbunterräume von Objektklassen vor, die aus einer Sammlung von Beispielbildern gelernt werden. Diese Unterräume können genutzt werden um Bildmanipulationen auf valide Formen und Farben einzuschränken oder Manipulationsvorschläge zu liefern. Schließlich werden datenbasierte Farbmannigfaltigkeiten vorgestellt, die Farben eines spezifischen Kontexts enthalten. Diese Mannigfaltigkeiten ermöglichen eine Leistungssteigerung bei Farbauswahl, Farbstilisierung, Komprimierung und Weißabgleich

Contrôle artistique du rendu en synthèse d'images

Les images de synthèse sont désormais omniprésentes dans les domaines de création artistique, en particulier celui du cinéma. Dans ce cadre, un infographiste modélise une scène virtuelle, qui est ensuite rendue par un algorithme spécialisé, appelé moteur de rendu, pour produire une image résultat, le rendu. La description de la scène et l'algorithme de rendu reposent tous deux sur des modèles physiques afin d'assurer un résultat réaliste. Cependant, l'infographiste ne recherche pas seulement un résultat réaliste mais aussi un objectif artistique, relatif à l'oeuvre concernée. Cet objectif artistique peut être en opposition avec le réalisme physique du rendu, de la même manière qu'en peinture, des techniques comme le clair-obscur ne correspondent pas à une réalité physique mais répondent à un objectif artistique précis. Dans cette thèse, nous commençons par définir la problèmatique de contrôle artistique du rendu en synthèse d'images et établissons trois critères d'évaluation des méthodes d'édition du rendu. Nous proposons une taxonomie caractérisant les différents paradigmes d'approches permettant de contrôler le rendu. Dans nos travaux, nous nous intéressons aux approches comportementales permettant de répondre équitabement à tous les critères d'évaluation retenus. Ces méthodes agissent sur le comportement du transport lumineux. Suivant cette approche, nous proposons un formalisme théorique d'édition du rendu par téléportation du transport lumineux, que nous intégrons aux équations de radiométrie, bases des algorithmes de rendu. Nous proposons ensuite une implémentation pratique, appelée portails, dans laquelle le contrôle du transport se fait à l'aide d'un couple de surfaces, entrée et sortie, représentant la téléportation dans l'espace 3D du transport lumineux. Nous intégrons les portails à différents moteurs courants, et analysons les résultats obtenus, à la fois nouveaux et en comparaison des travaux précèdents. Finalement, nous proposons une analyse préliminaire de la structure de l'espace des chemins et faisons un tour d'horizon des travaux futurs.Nowadays, computer-generated imagery (CGI) is a standard in digital content creation, as in the context of film production. Visual artists design a model of the virtual scene. The renderer, a specific software, then produces the resulting image, called rendering. Both the 3D model and the renderer rely on physical laws in order to give photorealism. However, for artistic purposes, the visual artist does not necessarily seeks photorealism but is more inclined to care about artistic goals, mostly based on the related work of art. These artistic goals may be in opposition to physically-based rendering, in a similar way to techniques such as chiaroscuro in painting which do not yield a photorealistic result but rather aim at a precise artistic goal. In this PhD thesis, we first define the question of artistic control in rendering and specify three criteria to evaluate editing techniques. We propose a taxonomy to characterise the suitable paradigms to edit rendering. In our work, we focus on behavioural approaches, which properly suit all the evaluation criteria we defined. These methods act on the behaviour of light transport. Following this approach, we propose a theoretical formalism to edit rendering through the teleportation of light transport. We integrate this formalism in the underlying concept of rendering techniques, the rendering equation. We propose a practical tool we call \emph{RayPortals}. Using RayPortals, the rendering is edited using a pair of surfaces, input and output, defining the teleportation in 3D space of ligth transport. We integrate RayPortals in usual rendering techniques. We compare to previous work and also show some novel results. Ultimately, we propose a preliminary analysis of the path-space structure and summarize the future works

Cross-dimensional Analysis for Improved Scene Understanding

Visual data have taken up an increasingly large role in our society. Most people have instant access to a high quality camera in their pockets, and we are taking more pictures than ever before. Meanwhile, through the advent of better software and hardware, the prevalence of 3D data is also rapidly expanding, and demand for data and analysis methods is burgeoning in a wide range of industries. The amount of information about the world implicitly contained in this stream of data is staggering. However, as these images and models are created in uncontrolled circumstances, the extraction of any structured information from the unstructured pixels and vertices is highly non-trivial. To aid this process, we note that the 2D and 3D data modalities are similar in content, but intrinsically different in form. Exploiting their complementary nature, we can investigate certain problems in a cross-dimensional fashion - for example, where 2D lacks expressiveness, 3D can supplement it; where 3D lacks quality, 2D can provide it. In this thesis, we explore three analysis tasks with this insight as our point of departure. First, we show that by considering the tasks of 2D and 3D retrieval jointly we can improve performance of 3D retrieval while simultaneously enabling interesting new ways of exploring 2D retrieval results. Second, we discuss a compact representation of indoor scenes called a "scene map", which represents the objects in a scene using a top-down map of object locations. We propose a method for automatically extracting such scene maps from single 2D images using a database of 3D models for training. Finally, we seek to convert single 2D images to full 3D scenes using a database of 3D models as input. Occlusion is handled by modelling object context explicitly, allowing us to identify and pose objects that would otherwise be too occluded to make inferences about. For all three tasks, we show the utility of our cross-dimensional insight by evaluating each method extensively and showing favourable performance over baseline methods

3D Material Style Transfer Chuong H. Nguyen 1

This work proposes a technique to transfer the material style or mood from a guide source such as an image or video onto a target 3D scene. It formulates the problem as a combinatorial optimization of assigning discrete materials extracted from the guide source to discrete objects in the target 3D scene. The assignment is optimized to fulfill multiple goals: overall image mood based on several image statistics; spatial material organization and grouping as well as geometric similarity between objects that were assigned to similar materials. To be able to use common uncalibrated images and videos with unknown geometry and lighting as guides, a material estimation derives perceptually plausible reflectance, specularity, glossiness, and texture. Finally, results produced by our method are compared to manual material assignments in a perceptual study. 1