Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications

We present an overview and evaluation of a new, systematic approach for generation of highly realistic, annotated synthetic data for training of deep neural networks in computer vision tasks. The main contribution is a procedural world modeling approach enabling high variability coupled with physically accurate image synthesis, and is a departure from the hand-modeled virtual worlds and approximate image synthesis methods used in real-time applications. The benefits of our approach include flexible, physically accurate and scalable image synthesis, implicit wide coverage of classes and features, and complete data introspection for annotations, which all contribute to quality and cost efficiency. To evaluate our approach and the efficacy of the resulting data, we use semantic segmentation for autonomous vehicles and robotic navigation as the main application, and we train multiple deep learning architectures using synthetic data with and without fine tuning on organic (i.e. real-world) data. The evaluation shows that our approach improves the neural network's performance and that even modest implementation efforts produce state-of-the-art results.Comment: The project web page at http://vcl.itn.liu.se/publications/2017/TKWU17/ contains a version of the paper with high-resolution images as well as additional materia

Acquisition, Modeling, and Augmentation of Reflectance for Synthetic Optical Flow Reference Data

This thesis is concerned with the acquisition, modeling, and augmentation of material reflectance to simulate high-fidelity synthetic data for computer vision tasks. The topic is covered in three chapters: I commence with exploring the upper limits of reflectance acquisition. I analyze state-of-the-art BTF reflectance field renderings and show that they can be applied to optical flow performance analysis with closely matching performance to real-world images. Next, I present two methods for fitting efficient BRDF reflectance models to measured BTF data. Both methods combined retain all relevant reflectance information as well as the surface normal details on a pixel level. I further show that the resulting synthesized images are suited for optical flow performance analysis, with a virtually identical performance for all material types. Finally, I present a novel method for augmenting real-world datasets with physically plausible precipitation effects, including ground surface wetting, water droplets on the windshield, and water spray and mists. This is achieved by projecting the realworld image data onto a reconstructed virtual scene, manipulating the scene and the surface reflectance, and performing unbiased light transport simulation of the precipitation effects

Understanding the Influence of Rendering Parameters in Synthetic Datasets for Neural Semantic Segmentation Tasks

Cursos e Congresos , C-155[Abstract] Deep neural networks are well known for demanding large amounts of training data, motivating the appearance of multiple synthetic datasets covering multiple domains. However, synthetic datasets have not yet outperformed real data for autonomous driving applications, particularly for semantic segmentation tasks. Thus, a deeper comprehension about how the parameters involved in synthetic data generation could help in creating better synthetic datasets. This work provides a summary review of prior research covering how image noise, camera noise and rendering photorealism could affect learning tasks. Furthermore, we presents novel experiments aimed at advancing our understanding around generating synthetic data for autonomous driving neural networks aimed at semantic segmentationXunta de Galicia; ED431F 2021/11This work has been supported by the Spanish Ministry of Science and Innovation (AEI/PID2020-115734RB-C22). We also want to acknowledge Side Effects Software Inc. for their support to this work. J.A. Iglesias-Guitian also acknowledges the UDC-Inditex InTalent programme, the Ministry of Science and Innovation (AEI/RYC2018-025385-I) and Xunta de Galicia (ED431F 2021/11). CITIC is funded by the Xunta de Galicia through the collaboration agreement between the Consellería de Cultura, Educación, Formación Profesional e Universidades and the Galician universities for the reinforcement of the research centres of the Galician University System (CIGUS

Physically informed car engine sound synthesis for virtual and augmented environments

The richness of crossmodal feedback in car driving makes it an engaging, complex, yet “natural” activity. Audition plays an important role, as the engine sound, perceived in the cabin, conveys relevant cues about the vehicle motion. In this paper, we introduce a procedural and physically informed model for synthetic combustion engine sound, as an effective, flexible and computationally efficient alternative to sample-based and analysis/resynthesis approaches. The sound model, currently being developed as Max/MSP external, has been integrated in GeneCars, a driving simulator environment for industrial sound design, and SkAT Studio, a demonstration framework for the rapid creation of audio processing workflows

Modelling and Visualisation of the Optical Properties of Cloth

Cloth and garment visualisations are widely used in fashion and interior design, entertaining, automotive and nautical industry and are indispensable elements of visual communication. Modern appearance models attempt to offer a complete solution for the visualisation of complex cloth properties. In the review part of the chapter, advanced methods that enable visualisation at micron resolution, methods used in three-dimensional (3D) visualisation workflow and methods used for research purposes are presented. Within the review, those methods offering a comprehensive approach and experiments on explicit clothes attributes that present specific optical phenomenon are analysed. The review of appearance models includes surface and image-based models, volumetric and explicit models. Each group is presented with the representative authors’ research group and the application and limitations of the methods. In the final part of the chapter, the visualisation of cloth specularity and porosity with an uneven surface is studied. The study and visualisation was performed using image data obtained with photography. The acquisition of structure information on a large scale namely enables the recording of structure irregularities that are very common on historical textiles, laces and also on artistic and experimental pieces of cloth. The contribution ends with the presentation of cloth visualised with the use of specular and alpha maps, which is the result of the image processing workflow