Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications

We present an overview and evaluation of a new, systematic approach for generation of highly realistic, annotated synthetic data for training of deep neural networks in computer vision tasks. The main contribution is a procedural world modeling approach enabling high variability coupled with physically accurate image synthesis, and is a departure from the hand-modeled virtual worlds and approximate image synthesis methods used in real-time applications. The benefits of our approach include flexible, physically accurate and scalable image synthesis, implicit wide coverage of classes and features, and complete data introspection for annotations, which all contribute to quality and cost efficiency. To evaluate our approach and the efficacy of the resulting data, we use semantic segmentation for autonomous vehicles and robotic navigation as the main application, and we train multiple deep learning architectures using synthetic data with and without fine tuning on organic (i.e. real-world) data. The evaluation shows that our approach improves the neural network's performance and that even modest implementation efforts produce state-of-the-art results.Comment: The project web page at http://vcl.itn.liu.se/publications/2017/TKWU17/ contains a version of the paper with high-resolution images as well as additional materia

Improving Usability in Procedural Modeling

This work presents new approaches and algorithms for procedural modeling geared towards user convenience and improving usability, in order to increase artists’ productivity. Procedural models create geometry for 3D models from sets of rules. Existing approaches that allow to model trees, buildings, and terrain are reviewed and possible improvements are discussed. A new visual programming language for procedural modeling is discussed, where the user connects operators to visual programs called model graphs. These operators create geometry with textures, assign or evaluate variables or control the sequence of operations. When the user moves control points using the mouse in 3D space, the model graph is executed to change the geometry interactively. Thus, model graphs combine the creativity of freehand modeling with the power of programmed modeling while displaying the program structure more clearly than textbased approaches. Usability is increased as a result of these advantages. Also, an interactive editor for botanical trees is demonstrated. In contrast to previous tree modeling systems, we propose linking rules, parameters and geometry to semantic entities. This has the advantage that problems of associating parameters and instances are completely avoided. When an entity is clicked in the viewport, its parameters are displayed immediately, changes are applied to selected entities, and viewport editing operations are reflected in the parameter set. Furthermore, we store the entities in a hierarchical data structure and allow the user to activate recursive traversal via selection options for all editing operations. The user may choose to apply viewport or parameter changes to a single entity or many entities at once, and only the geometry for the affected entities needs to be updated. The proposed user interface simplifies the modeling process and increases productivity. Interactive editing approaches for 3D models often allow more precise control over a model than a global set of parameters that is used to generate a shape. However, usually scripted procedural modeling generates shapes directly from a fixed set of parameters, and interactive editing mostly uses a fixed set of tools. We propose to use scripts not only to generate models, but also for manipulating the models. A base script would set up the state of an object, and tool scripts would modify that state. The base script and the tool scripts generate geometry when necessary. Together, such a collection of scripts forms a template, and templates can be created for various types of objects. We examine how templates simplify the procedural modeling workflow by allowing for editing operations that are context-sensitive, flexible and powerful at the same time. Many algorithms have been published that produce geometry for fictional landscapes. There are algorithms which produce terrain with minimal setup time, allowing to adapt the level of detail as the user zooms into the landscape. However, these approaches lack plausible river networks, and algorithms that create eroded terrain with river networks require a user to supervise creation and minutes or hours of computation. In contrast to that, this work demonstrates an algorithm that creates terrain with plausible river networks and adaptive level of detail with no more than a few seconds of preprocessing. While the system can be configured using parameters, this text focuses on the algorithm that produces the rivers. However, integrating more tools for user-controlled editing of terrain would be possible.Verbesserung der Usability bei prozeduraler Modellierung Ziel der vorliegenden Arbeit ist es, prozedurale Modellierung durch neue neue Ansätze und Algorithmen einfacher, bequemer und anwendungsfreundlicher zu machen, und damit die Produktivität der Künstler zu erhöhen. Diese Anforderungen werden häufig unter dem Stichwort Usability zusammengefasst. Prozedurale Modelle spezifizieren 3D-Modelle über Regeln. Existierende Ansätze für Bäume, Gebäude und Terrain werden untersucht und es werden mögliche Verbesserungen diskutiert. Eine neue visuelle Programmiersprache für prozedurale Modelle wird vorgestellt, bei der Operatoren zu Modellgraphen verschaltet werden. Die Operatoren erzeugen texturierte Geometrie, weisen Variablen zu und werten sie aus, oder sie steuern den Ablauf der Operationen. Wenn der Benutzer Kontrollpunkte im Viewport mit der Maus verschiebt, wird der Modellgraph ausgeführt, um interaktiv neue Geometrie für das Modell zu erzeugen. Modellgraphen kombinieren die kreativen Möglichkeiten des freihändigen Editierens mit der Mächtigkeit der prozeduralen Modellierung. Darüber hinaus sind Modellgraphen eine visuelle Programmiersprache und stellen die Struktur der Algorithmen deutlicher dar als textbasierte Programmiersprachen. Als Resultat dieser Verbesserungen erhöht sich die Usability. Ein interaktiver Editor für botanische Bäume wird ebenfalls vorgestellt. Im Gegensatz zu früheren Ansätzen schlagen wir vor, Regeln, Parameter und Geometrie zu semantischen Entitäten zu verschmelzen. Auf diese Weise werden Zuordnungsprobleme zwischen Parametern und deren Instanzen komplett vermieden. Wenn im Viewport eine Instanz angeklickt wird, werden sofort ihre Parameter angezeigt, alle Änderungen wirken sich direkt auf die betroffenen Instanzen aus, und Änderungen im Viewport werden sofort in den Parametern reflektiert. Darüber hinaus werden die Entitäten in einer hierarchischen Datenstruktur gespeichert und alle Änderungen können rekursiv auf der Hierarchie ausgeführt werden. Dem Benutzer werden Selektionsoptionen zur Verfügung gestellt, über die er Änderungen an den Parametern oder Änderungen im Viewport an einzelnen oder vielen Instanzen gleichzeitig vornehmen kann. Anschließend muss das System nur die Geometrie der betroffenen Instanzen aktualisieren. Auch hier ist das Ziel, das User Interface möglichst an den Bedürfnissen des Benutzers auszurichten, um Vereinfachungen und eine Erhöhung der Produktivität zu erreichen. Interaktive Editieransätze für 3D-Modelle erlauben häufig eine präzisere Kontrolle über ein Modell als ein globaler Parametersatz, der für die Erzeugung des Modells genutzt wird. Trotzdem erzeugen prozedurale Modellierskripte ihre Modelle meist direkt aus einem festen Parametersatz, während interaktive Tools meist mit hartkodierten Operationen arbeiten. Wir schlagen vor, Skripte nicht nur zur Erzeugung der Modelle zu verwenden, sondern auch um die erzeugten Modelle zu editieren. Ein Basisskript soll die Statusinformationen eines Objekts anlegen, während weitere Skripte diesen Status verändern und passende Geometrie erzeugen. Diese Skripte bilden dann ein Template zum Erzeugen einer Klasse von Objekten. Verschiedene Objekttypen können jeweils ihr eigenes Template haben. Wir zeigen, wie Templates den Workflow mit prozeduralen Modellen vereinfachen können, indem Operationen geschaffen werden, die gleichzeitig kontext-sensitiv, mächtig und flexibel sind. Es existiert eine Reihe von Verfahren, um Geometrie für synthetische Landschaften zu erzeugen. Ein Teil der Algorithmen erzeugt Geometrie mit minimaler Vorberechnung und erlaubt es, den Detailgrad der Landschaft interaktiv an die Perspektive anzupassen. Leider fehlen den so erzeugten Landschaften plausible Flussnetze. Algorithmen, die erodiertes Terrain mit Flussnetzen erzeugen, müssen aufwendig vom Benutzer überwacht werden und brauchen Minuten oder Stunden Rechenzeit. Im Gegensatz dazu stellen wir einen Algorithmus vor, der plausible Flussnetze erzeugt, während sich der Betrachter interaktiv durch die Szene bewegt. Das System kann über Parameter gesteuert werden, aber der Fokus liegt auf dem Algorithmus zur Erzeugung der Flüsse. Dennoch wäre es möglich, Tools zum benutzergesteuerten Editieren von Terrain zu integrieren

Assessing the Utility of Procedural Modeling for the Urban Planning Discipline: CityEngine and Missing Middle Housing

In recent years, procedural modeling techniques have been introduced to the urban planning discipline. By enabling the quick generation of design alternatives, these methods have the potential to expedite the public participation process. In this study, we explore ArcGIS CityEngine, an Esri procedural modeling software, and its application for depicting new missing middle housing developments in San Luis Obispo, California. Missing middle housing is an abstract planning concept and the public may benefit from 3D visualizations of the typology contextualized within their own neighborhood. To this end, we procedurally model two existing missing middle developments in San Luis Obispo and transfer them to three alternative contexts. Our team consists of two planning students with no programming background, granting us a relevant perspective on the experience for planning professionals. Through this exercise, we determined that procedural modeling can accelerate the design process when applied to a fitting scenario after the initial training period is complete

Automated 3D generation and rendering of stylized building designs

Thesis (S.B. in Art and Design)--Massachusetts Institute of Technology, Dept. of Architecture, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 29).The entertainment industry relies fairly heavily on computer-generated imagery to depict built environments in current films, video games, and other forms of simulated reality. These often involve highly complex geometries that take a long time to hand-model and are too difficult or costly for many productions' rendering capacities, both in computational costs as well as time. Procedural modeling and the automation of these geometries is one option to solve these problems. Many modeling programs involve a script or procedural modeling component. This thesis explores the use of CityEngine, a commercially available software that is specialized to generate geometries for buildings in urban environments. By using the CGA Shape Grammar built into CityEngine, this project seeks to generate geometries based on complex architectural precedents using a procedural modeling system based on shape grammar and computational design principles. Results are generated and discussed, as well as applications and future work.by Anisha V. Deshmane.S.B.in Art and Desig

Procedural modeling historical buildings for serious games

[EN] In this paper we target the goal of obtaining detailed historical virtual buildings, like a castle or a city old town, through a methodology that facilitates their reconstruction. We allow having in a short time an approximation model that is flexible for being explored, analyzed and eventually modified. This is crucial for serious game development pipelines, whose objective is focused not only on accuracy and realism, but also on transmitting a sense of immersion to the player.[ES] En este trabajo apuntamos al objetivo de obtener edificios históricos virtuales con alto nivel de detalle, como por ejemplo un castillo o un barrio de una ciudad antigua, a través de una metodología que facilite su reconstrucción. Las herramientas que proponemos nos permiten obtener rápidamente un modelo aproximado y flexible para ser explorado, analizado o eventualmente modificado. Modelos con estas características son esenciales para el desarrollo en plataformas de juegos serios, cuyo objetivo fundamental se centra no sólo en la precisión y el realismo, sino también en transmitir una sensación de inmersión para el jugador.We would like to thank Adriana Desplans for help with the Carcassonne prototype. This work was partially funded with grant TIN2010-20590-C02-02 from Ministerio de Ciencia e Innovación, Spain.Besuievsky, G.; Patow, G. (2013). Procedural modeling historical buildings for serious games. Virtual Archaeology Review. 4(9):160-166. https://doi.org/10.4995/var.2013.426816016649P. Müller, P. Wonka, S. Haegler, A. Ulmer, and L. Van Gool. Procedural modeling of buildings. ACM Trans. Graph., 25(3):614-623, 2006. http://dx.doi.org/10.1145/1141911.1141931P.Musialski, P. Wonka, D. Aliaga, M. Wimmer, L. van Gool and W. Purgathofer. A Survey of Urban Reconstruction. EUROGRAPHICS 2012 State of the Art ReportsS. Haegler, P.Mller and L. Van Gool. Procedural Modeling for Digital Cultural Heritage. EURASIP J. Image and Video Processing V 2009.R. Ridorsa, G. Patow, The skylineEngine System, in XX Congreso Español de Iinformàtica Gràfica, CEIG2010 (2010), pp. 207-216L. Krecklau, C. Manthei, L. Kobbelt. Procedural Interpolation of Historical City Maps. Vol. 31 (2012), Number 2G. Patow, User-Friendly Graph Editing for Procedural Buildings.IEEE Computer Graphics and Applications, vol. 32, no. 2, Mar./Apr. 2012, pp. 66--75; http://dx.doi.org/10.1109/MCG.2010.104B.Watson, P. Müller, O. VERYOVKA, A.Filler., P. Wonka and C.Sexton. 2008. Procedural urban modeling in practice. IEEE Computer Graphics and Applications 28, 18-26. http://dx.doi.org/10.1109/MCG.2008.57CityEngine 2011. www.procedural.comG. Besuievsky and G. Patow, A Procedural Modelling Approach for Automatic Generation of LoD Building Models, Proceedings of the CISBAT 2011, pp. 993-998, Lausanne, Switzerland.O. Pueyo, G. Patow. Structuring Urban Data, Tech. Rep. IMA12-01-RR, Dept. IMA, Universitat de Girona (2012)S.Barroso, G.Patow. Recreación de estructuras arquitectónicas mediante modelaje

House generation using procedural modeling with rules

This is a work dedicated to the development of a software for creating 3D models of houses with modern design. The main goal is to generate a tool that makes use of procedural modeling techniques to generate a wide variety of models. The result is an application capable of generating realistic model renderings with an input consisting of a set of rules. The most important part of this project will be the development of grammar and of all each operation will start to be simple and basic in the form of 3D. Combinations of operations such as extrusions, subdivisions or prefabrications will then be done to increase the complexity of the whole model that makes it look like a modern house. Thispart was carried out with some tests based on a correct performance of the same algorithm, as this is the core of the whole application.Aquest és un treball dedicat al desenvolupament d'un software de creació de models 3D de cases amb disseny modern. L'objectiu principal és generar una eina que faci ús de tècniques de modelat procedural per generar una gran varietat de models. El resultat és una aplicació capaç de generar renders de models realistes amb una entrada que consisteix en un conjunt de regles. La part més important d’aquest projecte consistirà en el desenvolupament de la gramàtica i cada operació començarà a ser senzilla i bàsica en forma de 3D. A continuació, es faran combinacions d’operacions com extrusions, subdivisions o prefabricacions augmenta la complexitat de tot el model que fa que sembli una casa actual. Aquesta part es va dur a terme amb algunes proves basades en un correcte rendiment del mateix algorisme, ja que aquest és el nucli de tota l'aplicació.Este es un trabajo dedicado al desarrollo de un software de creación de modelos 3D de casas con diseño moderno. El objetivo principal es generar una herramienta que haga uso de técnicas de modelado procedural para generar una gran variedad de modelos. El resultado es una aplicación capaz de generar renders de modelos realistas con una entrada que consiste en un conjunto de reglas. La parte más importante de este proyecto consistirá en el desarrollo de la gramática y cada operación empezará a ser sencilla y básica en forma de 3D. A continuación, se harán combinaciones de operaciones como extrusiones, subdivisiones o prefabricaciones aumenta la complejidad de todo el modelo que hace que parezca una casa actual. Esta parte se llevó a cabo con algunas pruebas basadas en un correcto rendimiento del mismo algoritmo, puesto que este es el núcleo de toda la aplicación

Interactive translucent volume rendering and procedural modeling

Journal ArticleDirect volume rendering is a commonly used technique in visualization applications. Many of these applications require sophisticated shading models to capture subtle lighting effects and characteristics of volume metric data and materials. Many common objects and natural phenomena exhibit visual quality that cannot be captured using simple lighting models or cannot be solved at interactive rates using more sophisticated methods. We present a simple yet effective interactive shading model which captures volumetric light attenuation effects to produce volumetric shadows and the subtle appearance of translucency. We also present a technique for volume displacement or perturbation that allows realistic interactive modeling of high frequency detail for real and synthetic volumetric data