Transport-Based Neural Style Transfer for Smoke Simulations

Artistically controlling fluids has always been a challenging task. Optimization techniques rely on approximating simulation states towards target velocity or density field configurations, which are often handcrafted by artists to indirectly control smoke dynamics. Patch synthesis techniques transfer image textures or simulation features to a target flow field. However, these are either limited to adding structural patterns or augmenting coarse flows with turbulent structures, and hence cannot capture the full spectrum of different styles and semantically complex structures. In this paper, we propose the first Transport-based Neural Style Transfer (TNST) algorithm for volumetric smoke data. Our method is able to transfer features from natural images to smoke simulations, enabling general content-aware manipulations ranging from simple patterns to intricate motifs. The proposed algorithm is physically inspired, since it computes the density transport from a source input smoke to a desired target configuration. Our transport-based approach allows direct control over the divergence of the stylization velocity field by optimizing incompressible and irrotational potentials that transport smoke towards stylization. Temporal consistency is ensured by transporting and aligning subsequent stylized velocities, and 3D reconstructions are computed by seamlessly merging stylizations from different camera viewpoints.Comment: ACM Transaction on Graphics (SIGGRAPH ASIA 2019), additional materials: http://www.byungsoo.me/project/neural-flow-styl

Stereological techniques for synthesizing solid textures from images of aggregate materials

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, February 2005.Includes bibliographical references (leaves 121-130).When creating photorealistic digital scenes, textures are commonly used to depict complex variation in surface appearance. For materials that have spatial variation in three dimensions, such as wood or marble, solid textures offer a natural representation. Unlike 2D textures, which can be easily captured with a photograph, it can be difficult to obtain a 3D material volume. This thesis addresses the challenge of extrapolating tileable 3D solid textures from images of aggregate materials, such as concrete, asphalt, terrazzo or granite. The approach introduced here is inspired by and builds on prior work in stereology--the study of 3D properties of a material based on 2D observations. Unlike ad hoc methods for texture synthesis, this approach has rigorous mathematical foundations that allow for reliable, accurate material synthesis with well-defined assumptions. The algorithm is also driven by psychophysical constraints to insure that slices through the synthesized volume have a perceptually similar appearance to the input image. The texture synthesis algorithm uses a variety of techniques to independently solve for the shape, distribution, and color of the embedded particles, as well as the residual noise. To approximate particle shape, I consider four methods-including two algorithms of my own contribution. I compare these methods under a variety of input conditions using automated, perceptually-motivated metrics as well as a carefully controlled psychophysical experiment. In addition to assessing the relative performance of the four algorithms, I also evaluate the reliability of the automated metrics in predicting the results of the user study. To solve for the particle distribution, I apply traditional stereological methods.(cont.) I first illustrate this approach for aggregate materials of spherical particles and then extend the technique to apply to particles of arbitrary shapes. The particle shape and distribution are used in conjunction to create an explicit 3D material volume using simulated annealing. Particle colors are assigned using a stochastic method, and high-frequency noise is replicated with the assistance of existing algorithms. The data representation is suitable for high-fidelity rendering and physical simulation. I demonstrate the effectiveness of the approach with side-by-side comparisons of real materials and their synthetic counterparts derived from the application of these techniques.by Robert Carl Jagnow.Ph.D

A Shape-Aware Model for Discrete Texture Synthesis

International audienceWe present a novel shape-aware method for synthesizing 2D and 3D discrete element textures consisting of collections of distinct vector graphics objects. Extending the long-proven point process framework, we propose a shape process, a novel stochastic model based on spatial measurements that fully take into account the geometry of the elements. We demonstrate that our approach is well-suited for discrete texture synthesis by example. Our modelenables for both robust statistical parameter estimation and reliable output generation by Monte Carlo sampling. Our numerous experiments show that contrary to current state-of-the-art techniques, our algorithm manages to capture anisotropic element distributions and systematically prevents undesirable collisions between objects

Bridging the gap between reconstruction and synthesis

Aplicat embargament des de la data de defensa fins el 15 de gener de 20223D reconstruction and image synthesis are two of the main pillars in computer vision. Early works focused on simple tasks such as multi-view reconstruction and texture synthesis. With the spur of Deep Learning, the field has rapidly progressed, making it possible to achieve more complex and high level tasks. For example, the 3D reconstruction results of traditional multi-view approaches are currently obtained with single view methods. Similarly, early pattern based texture synthesis works have resulted in techniques that allow generating novel high-resolution images. In this thesis we have developed a hierarchy of tools that cover all these range of problems, lying at the intersection of computer vision, graphics and machine learning. We tackle the problem of 3D reconstruction and synthesis in the wild. Importantly, we advocate for a paradigm in which not everything should be learned. Instead of applying Deep Learning naively we propose novel representations, layers and architectures that directly embed prior 3D geometric knowledge for the task of 3D reconstruction and synthesis. We apply these techniques to problems including scene/person reconstruction and photo-realistic rendering. We first address methods to reconstruct a scene and the clothed people in it while estimating the camera position. Then, we tackle image and video synthesis for clothed people in the wild. Finally, we bridge the gap between reconstruction and synthesis under the umbrella of a unique novel formulation. Extensive experiments conducted along this thesis show that the proposed techniques improve the performance of Deep Learning models in terms of the quality of the reconstructed 3D shapes / synthesised images, while reducing the amount of supervision and training data required to train them. In summary, we provide a variety of low, mid and high level algorithms that can be used to incorporate prior knowledge into different stages of the Deep Learning pipeline and improve performance in tasks of 3D reconstruction and image synthesis.La reconstrucció 3D i la síntesi d'imatges són dos dels pilars fonamentals en visió per computador. Els estudis previs es centren en tasques senzilles com la reconstrucció amb informació multi-càmera i la síntesi de textures. Amb l'aparició del "Deep Learning", aquest camp ha progressat ràpidament, fent possible assolir tasques molt més complexes. Per exemple, per obtenir una reconstrucció 3D, tradicionalment s'utilitzaven mètodes multi-càmera, en canvi ara, es poden obtenir a partir d'una sola imatge. De la mateixa manera, els primers treballs de síntesi de textures basats en patrons han donat lloc a tècniques que permeten generar noves imatges completes en alta resolució. En aquesta tesi, hem desenvolupat una sèrie d'eines que cobreixen tot aquest ventall de problemes, situats en la intersecció entre la visió per computador, els gràfics i l'aprenentatge automàtic. Abordem el problema de la reconstrucció i la síntesi 3D en el món real. És important destacar que defensem un paradigma on no tot s'ha d'aprendre. Enlloc d'aplicar el "Deep Learning" de forma naïve, proposem representacions novedoses i arquitectures que incorporen directament els coneixements geomètrics ja existents per a aconseguir la reconstrucció 3D i la síntesi d'imatges. Nosaltres apliquem aquestes tècniques a problemes com ara la reconstrucció d'escenes/persones i a la renderització d'imatges fotorealistes. Primer abordem els mètodes per reconstruir una escena, les persones vestides que hi ha i la posició de la càmera. A continuació, abordem la síntesi d'imatges i vídeos de persones vestides en situacions quotidianes. I finalment, aconseguim, a través d'una nova formulació única, connectar la reconstrucció amb la síntesi. Els experiments realitzats al llarg d'aquesta tesi demostren que les tècniques proposades milloren el rendiment dels models de "Deepp Learning" pel que fa a la qualitat de les reconstruccions i les imatges sintetitzades alhora que redueixen la quantitat de dades necessàries per entrenar-los. En resum, proporcionem una varietat d'algoritmes de baix, mitjà i alt nivell que es poden utilitzar per incorporar els coneixements previs a les diferents etapes del "Deep Learning" i millorar el rendiment en tasques de reconstrucció 3D i síntesi d'imatges.Postprint (published version

Design of decorative 3D models: from geodesic ornaments to tangible assemblies

L'obiettivo di questa tesi è sviluppare strumenti utili per creare opere d'arte decorative digitali in 3D. Uno dei processi decorativi più comunemente usati prevede la creazione di pattern decorativi, al fine di abbellire gli oggetti. Questi pattern possono essere dipinti sull'oggetto di base o realizzati con l'applicazione di piccoli elementi decorativi. Tuttavia, la loro realizzazione nei media digitali non è banale. Da un lato, gli utenti esperti possono eseguire manualmente la pittura delle texture o scolpire ogni decorazione, ma questo processo può richiedere ore per produrre un singolo pezzo e deve essere ripetuto da zero per ogni modello da decorare. D'altra parte, gli approcci automatici allo stato dell'arte si basano sull'approssimazione di questi processi con texturing basato su esempi o texturing procedurale, o con sistemi di riproiezione 3D. Tuttavia, questi approcci possono introdurre importanti limiti nei modelli utilizzabili e nella qualità dei risultati. Il nostro lavoro sfrutta invece i recenti progressi e miglioramenti delle prestazioni nel campo dell'elaborazione geometrica per creare modelli decorativi direttamente sulle superfici. Presentiamo una pipeline per i pattern 2D e una per quelli 3D, e dimostriamo come ognuna di esse possa ricreare una vasta gamma di risultati con minime modifiche dei parametri. Inoltre, studiamo la possibilità di creare modelli decorativi tangibili. I pattern 3D generati possono essere stampati in 3D e applicati a oggetti realmente esistenti precedentemente scansionati. Discutiamo anche la creazione di modelli con mattoncini da costruzione, e la possibilità di mescolare mattoncini standard e mattoncini custom stampati in 3D. Ciò consente una rappresentazione precisa indipendentemente da quanto la voxelizzazione sia approssimativa. I principali contributi di questa tesi sono l'implementazione di due diverse pipeline decorative, un approccio euristico alla costruzione con mattoncini e un dataset per testare quest'ultimo.The aim of this thesis is to develop effective tools to create digital decorative 3D artworks. Real-world art often involves the use of decorative patterns to enrich objects. These patterns can be painted on the base or might be realized with the application of small decorative elements. However, their creation in digital media is not trivial. On the one hand, users can manually perform texture paint or sculpt each decoration, in a process that can take hours to produce a single piece and needs to be repeated from the ground up for every model that needs to be decorated. On the other hand, automatic approaches in state of the art rely on approximating these processes with procedural or by-example texturing or with 3D reprojection. However, these approaches can introduce significant limitations in the models that can be used and in the quality of the results. Instead, our work exploits the recent advances and performance improvements in the geometry processing field to create decorative patterns directly on surfaces. We present a pipeline for 2D and one for 3D patterns and demonstrate how each of them can recreate a variety of results with minimal tweaking of the parameters. Furthermore, we investigate the possibility of creating decorative tangible models. The 3D patterns we generate can be 3D printed and applied to previously scanned real-world objects. We also discuss the creation of models with standard building bricks and the possibility of mixing standard and custom 3D-printed bricks. This allows for a precise representation regardless of the coarseness of the voxelization. The main contributions of this thesis are the implementation of two different decorative pipelines, a heuristic approach to brick construction, and a dataset to test the latter