Compression, Modeling, and Real-Time Rendering of Realistic Materials and Objects

The realism of a scene basically depends on the quality of the geometry, the illumination and the materials that are used. Whereas many sources for the creation of three-dimensional geometry exist and numerous algorithms for the approximation of global illumination were presented, the acquisition and rendering of realistic materials remains a challenging problem. Realistic materials are very important in computer graphics, because they describe the reflectance properties of surfaces, which are based on the interaction of light and matter. In the real world, an enormous diversity of materials can be found, comprising very different properties. One important objective in computer graphics is to understand these processes, to formalize them and to finally simulate them. For this purpose various analytical models do already exist, but their parameterization remains difficult as the number of parameters is usually very high. Also, they fail for very complex materials that occur in the real world. Measured materials, on the other hand, are prone to long acquisition time and to huge input data size. Although very efficient statistical compression algorithms were presented, most of them do not allow for editability, such as altering the diffuse color or mesostructure. In this thesis, a material representation is introduced that makes it possible to edit these features. This makes it possible to re-use the acquisition results in order to easily and quickly create deviations of the original material. These deviations may be subtle, but also substantial, allowing for a wide spectrum of material appearances. The approach presented in this thesis is not based on compression, but on a decomposition of the surface into several materials with different reflection properties. Based on a microfacette model, the light-matter interaction is represented by a function that can be stored in an ordinary two-dimensional texture. Additionally, depth information, local rotations, and the diffuse color are stored in these textures. As a result of the decomposition, some of the original information is inevitably lost, therefore an algorithm for the efficient simulation of subsurface scattering is presented as well. Another contribution of this work is a novel perception-based simplification metric that includes the material of an object. This metric comprises features of the human visual system, for example trichromatic color perception or reduced resolution. The proposed metric allows for a more aggressive simplification in regions where geometric metrics do not simplif

Neural Radiance Fields: Past, Present, and Future

The various aspects like modeling and interpreting 3D environments and surroundings have enticed humans to progress their research in 3D Computer Vision, Computer Graphics, and Machine Learning. An attempt made by Mildenhall et al in their paper about NeRFs (Neural Radiance Fields) led to a boom in Computer Graphics, Robotics, Computer Vision, and the possible scope of High-Resolution Low Storage Augmented Reality and Virtual Reality-based 3D models have gained traction from res with more than 1000 preprints related to NeRFs published. This paper serves as a bridge for people starting to study these fields by building on the basics of Mathematics, Geometry, Computer Vision, and Computer Graphics to the difficulties encountered in Implicit Representations at the intersection of all these disciplines. This survey provides the history of rendering, Implicit Learning, and NeRFs, the progression of research on NeRFs, and the potential applications and implications of NeRFs in today's world. In doing so, this survey categorizes all the NeRF-related research in terms of the datasets used, objective functions, applications solved, and evaluation criteria for these applications.Comment: 413 pages, 9 figures, 277 citation

A fully automated three-stage procedure for spatio-temporal leaf segmentation with regard to the B-spline-based phenotyping of cucumber plants

Plant phenotyping deals with the metrological acquisition of plants in order to investigate the impact of environmental factors and a plant’s genotype on its appearance. Phenotyping methods that are used as standard in crop science are often invasive or even destructive. Due to the increase of automation within geodetic measurement systems and with the development of quasi-continuous measurement techniques, geodetic techniques are perfectly suitable for performing automated and non-invasive phenotyping and, hence, are an alternative to standard phenotyping methods. In this contribution, sequentially acquired point clouds of cucumber plants are used to determine the plants’ phenotypes in terms of their leaf areas. The focus of this contribution is on the spatio-temporal segmentation of the acquired point clouds, which automatically groups and tracks those sub point clouds that describe the same leaf. The application on example data sets reveals a successful segmentation of 93% of the leafs. Afterwards, the segmented leaves are approximated by means of B-spline surfaces, which provide the basis for the subsequent determination of the leaf areas. In order to validate the results, the determined leaf areas are compared to results obtained by means of standard methods used in crop science. The investigations reveal consistency of the results with maximal deviations in the determined leaf areas of up to 5

Dynamic texture synthesis in image and video processing.

Xu, Leilei.Thesis submitted in: October 2007.Thesis (M.Phil.)--Chinese University of Hong Kong, 2008.Includes bibliographical references (leaves 78-84).Abstracts in English and Chinese.Abstract --- p.iAcknowledgement --- p.iiiChapter 1 --- Introduction --- p.1Chapter 1.1 --- Texture and Dynamic Textures --- p.1Chapter 1.2 --- Related work --- p.4Chapter 1.3 --- Thesis Outline --- p.7Chapter 2 --- Image/Video Processing --- p.8Chapter 2.1 --- Bayesian Analysis --- p.8Chapter 2.2 --- Markov Property --- p.10Chapter 2.3 --- Graph Cut --- p.12Chapter 2.4 --- Belief Propagation --- p.13Chapter 2.5 --- Expectation-Maximization --- p.15Chapter 2.6 --- Principle Component Analysis --- p.15Chapter 3 --- Linear Dynamic System --- p.17Chapter 3.1 --- System Model --- p.18Chapter 3.2 --- Degeneracy and Canonical Model Realization --- p.19Chapter 3.3 --- Learning of Dynamic Textures --- p.19Chapter 3.4 --- Synthesizing Dynamic Textures --- p.21Chapter 3.5 --- Summary --- p.21Chapter 4 --- Dynamic Color Texture Synthesis --- p.25Chapter 4.1 --- Related Work --- p.25Chapter 4.2 --- System Model --- p.26Chapter 4.2.1 --- Laplacian Pyramid-based DCTS Model --- p.28Chapter 4.2.2 --- RBF-based DCTS Model --- p.28Chapter 4.3 --- Experimental Results --- p.32Chapter 4.4 --- Summary --- p.42Chapter 5 --- Dynamic Textures using Multi-resolution Analysis --- p.43Chapter 5.1 --- System Model --- p.44Chapter 5.2 --- Multi-resolution Descriptors --- p.46Chapter 5.2.1 --- Laplacian Pyramids --- p.47Chapter 5.2.2 --- Haar Wavelets --- p.48Chapter 5.2.3 --- Steerable Pyramid --- p.49Chapter 5.3 --- Experimental Results --- p.51Chapter 5.4 --- Summary --- p.55Chapter 6 --- Motion Transfer --- p.59Chapter 6.1 --- Problem formulation --- p.60Chapter 6.1.1 --- Similarity on Appearance --- p.61Chapter 6.1.2 --- Similarity on Dynamic Behavior --- p.62Chapter 6.1.3 --- The Objective Function --- p.65Chapter 6.2 --- Further Work --- p.66Chapter 7 --- Conclusions --- p.67Chapter A --- List of Publications --- p.68Chapter B --- Degeneracy in LDS Model --- p.70Chapter B.l --- Equivalence Class --- p.70Chapter B.2 --- The Choice of the Matrix Q --- p.70Chapter B.3 --- Swapping the Column of C and A --- p.71Chapter C --- Probability Density Functions --- p.74Chapter C.1 --- Probability Distribution --- p.74Chapter C.2 --- Joint Probability Distributions --- p.75Bibliography --- p.7

City-Scaled Digital Documentation: A Comparative Analysis of Digital Documentation Technologies for Recording Architectural Heritage

The historic preservation field, enabled by advances in technology, has demonstrated an increased interest in digitizing cultural heritage sites and historic structures. Increases in software capabilities as well as greater affordability has fostered augmented use of digital documentation technologies for architectural heritage applications. Literature establishes four prominent categories of digital documentation tools for preservation: laser scanning, photogrammetry, multimedia geographic information systems (GIS) and three-dimensional modeling. Thoroughly explored through published case studies, the documentation techniques for recording heritage are most often integrated. Scholarly literature does not provide a parallel comparison of the four technologies. A comparative analysis of the four techniques, as presented in this thesis, makes it possible for cities to understand the most applicable technique for their preservation objectives. The thesis analyzes four cases studies that employ applications of the technologies: New Orleans Laser Scanning, University of Maryland Photogrammetry, Historic Columbia Maps Project and the Virtual Historic Savannah Project. Following this, the thesis undertakes a trial of each documentation technology – laser scanning, photogrammetry, multimedia GIS and three-dimensional modeling – utilizing a block on Church Street between Queen and Chalmers streets within the Charleston Historic District. The apparent outcomes of each of the four techniques is analyzed according to a series of parameters including: audience, application, efficacy in recordation, refinement, expertise required, manageability of the product, labor intensity and necessary institutional capacity. A concluding matrix quantifies the capability of each of the technologies in terms of the parameters. This method furnishes a parallel comparison of the techniques and their efficacy in architectural heritage documentation within mid-sized cities

Material Recognition Meets 3D Reconstruction : Novel Tools for Efficient, Automatic Acquisition Systems

For decades, the accurate acquisition of geometry and reflectance properties has represented one of the major objectives in computer vision and computer graphics with many applications in industry, entertainment and cultural heritage. Reproducing even the finest details of surface geometry and surface reflectance has become a ubiquitous prerequisite in visual prototyping, advertisement or digital preservation of objects. However, today's acquisition methods are typically designed for only a rather small range of material types. Furthermore, there is still a lack of accurate reconstruction methods for objects with a more complex surface reflectance behavior beyond diffuse reflectance. In addition to accurate acquisition techniques, the demand for creating large quantities of digital contents also pushes the focus towards fully automatic and highly efficient solutions that allow for masses of objects to be acquired as fast as possible. This thesis is dedicated to the investigation of basic components that allow an efficient, automatic acquisition process. We argue that such an efficient, automatic acquisition can be realized when material recognition "meets" 3D reconstruction and we will demonstrate that reliably recognizing the materials of the considered object allows a more efficient geometry acquisition. Therefore, the main objectives of this thesis are given by the development of novel, robust geometry acquisition techniques for surface materials beyond diffuse surface reflectance, and the development of novel, robust techniques for material recognition. In the context of 3D geometry acquisition, we introduce an improvement of structured light systems, which are capable of robustly acquiring objects ranging from diffuse surface reflectance to even specular surface reflectance with a sufficient diffuse component. We demonstrate that the resolution of the reconstruction can be increased significantly for multi-camera, multi-projector structured light systems by using overlappings of patterns that have been projected under different projector poses. As the reconstructions obtained by applying such triangulation-based techniques still contain high-frequency noise due to inaccurately localized correspondences established for images acquired under different viewpoints, we furthermore introduce a novel geometry acquisition technique that complements the structured light system with additional photometric normals and results in significantly more accurate reconstructions. In addition, we also present a novel method to acquire the 3D shape of mirroring objects with complex surface geometry. The aforementioned investigations on 3D reconstruction are accompanied by the development of novel tools for reliable material recognition which can be used in an initial step to recognize the present surface materials and, hence, to efficiently select the subsequently applied appropriate acquisition techniques based on these classified materials. In the scope of this thesis, we therefore focus on material recognition for scenarios with controlled illumination as given in lab environments as well as scenarios with natural illumination that are given in photographs of typical daily life scenes. Finally, based on the techniques developed in this thesis, we provide novel concepts towards efficient, automatic acquisition systems

Revealing the Invisible: On the Extraction of Latent Information from Generalized Image Data

The desire to reveal the invisible in order to explain the world around us has been a source of impetus for technological and scientific progress throughout human history. Many of the phenomena that directly affect us cannot be sufficiently explained based on the observations using our primary senses alone. Often this is because their originating cause is either too small, too far away, or in other ways obstructed. To put it in other words: it is invisible to us. Without careful observation and experimentation, our models of the world remain inaccurate and research has to be conducted in order to improve our understanding of even the most basic effects. In this thesis, we1 are going to present our solutions to three challenging problems in visual computing, where a surprising amount of information is hidden in generalized image data and cannot easily be extracted by human observation or existing methods. We are able to extract the latent information using non-linear and discrete optimization methods based on physically motivated models and computer graphics methodology, such as ray tracing, real-time transient rendering, and image-based rendering

Developing an Organisational Framework for Sustaining Virtual City Models

This research thesis presents an organisational framework for the management of virtual cities for hosts to adopt when seeking to produce and maintain a virtual city for use as a tool for urban planning related activities. The framework functions as an over-arching business model or structure, a general methodology for defining the organisational processes of virtual city enterprises. In achieving this aim, the research outlines standards and protocols for its creation, legal issues for its distribution and suggested processes for the update of 3D data. The diverse issues and needs of various stakeholders are addressed (Horne et al., 2006) in order to challenge the organisational issues and common concepts involved in creating, hosting and managing a city model (Voigt et al., 2004). Preliminary investigations showed that extensive research has been carried out on 3D and virtual city modelling techniques and their application, but the theoretical organisational and management issues for hosting 3D virtual city models needs to be addressed (Hamilton et al., 2005; Dokonal and Martens, 2001) through a ‘guiding source book’ for the creation and use of 3D city models(Bourdakis, 2004). This thesis explores the current state of virtual city modelling and its origins through literature research as well as an investigation into suitable business modelling practice. Pilot studies and an interview process with current virtual city hosts informed the research of current practice in the field. An organisational framework is subsequently put forward that combines elements from each of these investigations using a business model ‘canvas’ that can be adopted by current or prospective hosts and adapted to suit their circumstances, applications and users. The framework addresses the technical aspects of establishing a virtual city model, such as 3D data capture methods, spatial data infrastructure and modelling protocols in order to present a roadmap for virtual city enterprises. This correspondingly outlines a development from traditional and static datasets of geometry in ‘3D city models’ to more serviceable and user-centric ‘virtual city enterprises’. The organisational framework introduces 7 key areas that virtual city hosts should address for sustaining their enterprise that encompasses the technologies and expertise. Hence, this research makes significant contribution to knowledge by bringing together the many considerations that virtual city hosts must consider when creating a sustainable process to support urban planning

Photovoltaic potential in building façades

Tese de doutoramento, Sistemas Sustentáveis de Energia, Universidade de Lisboa, Faculdade de Ciências, 2018Consistent reductions in the costs of photovoltaic (PV) systems have prompted interest in applications with less-than-optimum inclinations and orientations. That is the case of building façades, with plenty of free area for the deployment of solar systems. Lower sun heights benefit vertical façades, whereas rooftops are favoured when the sun is near the zenith, therefore the PV potential in urban environments can increase twofold when the contribution from building façades is added to that of the rooftops. This complementarity between façades and rooftops is helpful for a better match between electricity demand and supply. This thesis focuses on: i) the modelling of façade PV potential; ii) the optimization of façade PV yields; and iii) underlining the overall role that building façades will play in future solar cities. Digital surface and solar radiation modelling methodologies were reviewed. Special focus is given to the 3D LiDAR-based model SOL and the CAD/plugin models DIVA and LadyBug. Model SOL was validated against measurements from the BIPV system in the façade of the Solar XXI building (Lisbon), and used to evaluate façade PV potential in different urban sites in Lisbon and Geneva. The plugins DIVA and LadyBug helped assessing the potential for PV glare from façade integrated photovoltaics in distinct urban blocks. Technologies for PV integration in façades were also reviewed. Alternative façade designs, including louvers, geometric forms and balconies, were explored and optimized for the maximization of annual solar irradiation using DIVA. Partial shading impacts on rooftops and façades were addressed through SOL simulations and the interconnections between PV modules were optimized using a custom Multi-Objective Genetic Algorithm. The contribution of PV façades to the solar potential of two dissimilar neighbourhoods in Lisbon was quantified using SOL, considering local electricity consumption. Cost-efficient rooftop/façade PV mixes are proposed based on combined payback times. Impacts of larger scale PV deployment on the spare capacity of power distribution transformers were studied through LadyBug and SolarAnalyst simulations. A new empirical solar factor was proposed to account for PV potential in future upgrade interventions. The combined effect of aggregating building demand, photovoltaic generation and storage on the self-consumption of PV and net load variance was analysed using irradiation results from DIVA, metered distribution transformer loads and custom optimization algorithms. SOL is shown to be an accurate LiDAR-based model (nMBE ranging from around 7% to 51%, nMAE from 20% to 58% and nRMSE from 29% to 81%), being the isotropic diffuse radiation algorithm its current main limitation. In addition, building surface material properties should be regarded when handling façades, for both irradiance simulation and PV glare evaluation. The latter appears to be negligible in comparison to glare from typical glaze/mirror skins used in high-rises. Irradiation levels in the more sunlit façades reach about 50-60% of the rooftop levels. Latitude biases the potential towards the vertical surfaces, which can be enhanced when the proportion of diffuse radiation is high. Façade PV potential can be increased in about 30% if horizontal folded louvers becomes a more common design and in another 6 to 24% if the interconnection of PV modules are optimized. In 2030, a mix of PV systems featuring around 40% façade and 60% rooftop occupation is shown to comprehend a combined financial payback time of 10 years, if conventional module efficiencies reach 20%. This will trigger large-scale PV deployment that might overwhelm current grid assets and lead to electricity grid instability. This challenge can be resolved if the placement of PV modules is optimized to increase self-sufficiency while keeping low net load variance. Aggregated storage within solar communities might help resolving the conflicting interests between prosumers and grid, although the former can achieve self-sufficiency levels above 50% with storage capacities as small as 0.25kWh/kWpv. Business models ought to adapt in order to create conditions for both parts to share the added value of peak power reduction due to optimized solar façades.Fundação para a Ciência e a Tecnologia (FCT), SFRH/BD/52363/201