Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects

We introduce a method based on the deflectometry principle for the reconstruction of specular objects exhibiting significant size and geometric complexity. A key feature of our approach is the deployment of an Automatic Virtual Environment (CAVE) as pattern generator. To unfold the full power of this extraordinary experimental setup, an optical encoding scheme is developed which accounts for the distinctive topology of the CAVE. Furthermore, we devise an algorithm for detecting the object of interest in raw deflectometric images. The segmented foreground is used for single-view reconstruction, the background for estimation of the camera pose, necessary for calibrating the sensor system. Experiments suggest a significant gain of coverage in single measurements compared to previous methods. To facilitate research on specular surface reconstruction, we will make our data set publicly available

On-site surface reflectometry

The rapid development of Augmented Reality (AR) and Virtual Reality (VR) applications over the past years has created the need to quickly and accurately scan the real world to populate immersive, realistic virtual environments for the end user to enjoy. While geometry processing has already gone a long way towards that goal, with self-contained solutions commercially available for on-site acquisition of large scale 3D models, capturing the appearance of the materials that compose those models remains an open problem in general uncontrolled environments. The appearance of a material is indeed a complex function of its geometry, intrinsic physical properties and furthermore depends on the illumination conditions in which it is observed, thus traditionally limiting the scope of reflectometry to highly controlled lighting conditions in a laboratory setup. With the rapid development of digital photography, especially on mobile devices, a new trend in the appearance modelling community has emerged, that investigates novel acquisition methods and algorithms to relax the hard constraints imposed by laboratory-like setups, for easy use by digital artists. While arguably not as accurate, we demonstrate the ability of such self-contained methods to enable quick and easy solutions for on-site reflectometry, able to produce compelling, photo-realistic imagery. In particular, this dissertation investigates novel methods for on-site acquisition of surface reflectance based on off-the-shelf, commodity hardware. We successfully demonstrate how a mobile device can be utilised to capture high quality reflectance maps of spatially-varying planar surfaces in general indoor lighting conditions. We further present a novel methodology for the acquisition of highly detailed reflectance maps of permanent on-site, outdoor surfaces by exploiting polarisation from reflection under natural illumination. We demonstrate the versatility of the presented approaches by scanning various surfaces from the real world and show good qualitative and quantitative agreement with existing methods for appearance acquisition employing controlled or semi-controlled illumination setups.Open Acces

3D Acquisition of Mirroring Objects using Striped Patterns

Objects with mirroring optical characteristics are left out of the scope of most 3D scanning methods. We present here a new automatic acquisition approach, shape-from-distortion, that focuses on that category of objects, requires only a still camera and a color monitor, and produces range scans (plus a normal and a reflectance map) of the target. Our technique consists of two steps: first, an improved environment matte is captured for the mirroring object, using the interference of patterns with different frequencies to obtain sub-pixel accuracy. Then, the matte is converted into a normal and a depth map by exploiting the self-coherence of a surface when integrating the normal map along different paths. The results show very high accuracy, capturing even smallest surface details. The acquired depth maps can be further processed using standard techniques to produce a complete 3D mesh of the object

Material Recognition Meets 3D Reconstruction : Novel Tools for Efficient, Automatic Acquisition Systems

For decades, the accurate acquisition of geometry and reflectance properties has represented one of the major objectives in computer vision and computer graphics with many applications in industry, entertainment and cultural heritage. Reproducing even the finest details of surface geometry and surface reflectance has become a ubiquitous prerequisite in visual prototyping, advertisement or digital preservation of objects. However, today's acquisition methods are typically designed for only a rather small range of material types. Furthermore, there is still a lack of accurate reconstruction methods for objects with a more complex surface reflectance behavior beyond diffuse reflectance. In addition to accurate acquisition techniques, the demand for creating large quantities of digital contents also pushes the focus towards fully automatic and highly efficient solutions that allow for masses of objects to be acquired as fast as possible. This thesis is dedicated to the investigation of basic components that allow an efficient, automatic acquisition process. We argue that such an efficient, automatic acquisition can be realized when material recognition "meets" 3D reconstruction and we will demonstrate that reliably recognizing the materials of the considered object allows a more efficient geometry acquisition. Therefore, the main objectives of this thesis are given by the development of novel, robust geometry acquisition techniques for surface materials beyond diffuse surface reflectance, and the development of novel, robust techniques for material recognition. In the context of 3D geometry acquisition, we introduce an improvement of structured light systems, which are capable of robustly acquiring objects ranging from diffuse surface reflectance to even specular surface reflectance with a sufficient diffuse component. We demonstrate that the resolution of the reconstruction can be increased significantly for multi-camera, multi-projector structured light systems by using overlappings of patterns that have been projected under different projector poses. As the reconstructions obtained by applying such triangulation-based techniques still contain high-frequency noise due to inaccurately localized correspondences established for images acquired under different viewpoints, we furthermore introduce a novel geometry acquisition technique that complements the structured light system with additional photometric normals and results in significantly more accurate reconstructions. In addition, we also present a novel method to acquire the 3D shape of mirroring objects with complex surface geometry. The aforementioned investigations on 3D reconstruction are accompanied by the development of novel tools for reliable material recognition which can be used in an initial step to recognize the present surface materials and, hence, to efficiently select the subsequently applied appropriate acquisition techniques based on these classified materials. In the scope of this thesis, we therefore focus on material recognition for scenarios with controlled illumination as given in lab environments as well as scenarios with natural illumination that are given in photographs of typical daily life scenes. Finally, based on the techniques developed in this thesis, we provide novel concepts towards efficient, automatic acquisition systems

Second Order Local Analysis for 3D Reconstruction of Specular Surfaces

We analyze the problem of recovering the shape of a mirror surface. A calibrated scene composed of lines passing through a point is assumed. The lines are reflected by the mirror surface onto the image plane of a calibrated camera, where the intersection, orientation and curvature of such reflections are measured. The relationship between the local geometry of the surface around the point of reflection and the measurements is analyzed. We extend the analysis in [13, 14], where we recovered positions and normals and second order local geometry of a specular surface up to one unknown parameter. We show that, provided that we work in a neighborhood of a surface whose third order surface terms can be neglected, the second order parameter ambiguity can be solved by equating the curvatures observed for the reflected lines with those computed from analytical differentiation followed by a perspective projection

Enhancing Mesh Deformation Realism: Dynamic Mesostructure Detailing and Procedural Microstructure Synthesis

Propomos uma solução para gerar dados de mapas de relevo dinâmicos para simular deformações em superfícies macias, com foco na pele humana. A solução incorpora a simulação de rugas ao nível mesoestrutural e utiliza texturas procedurais para adicionar detalhes de microestrutura estáticos. Oferece flexibilidade além da pele humana, permitindo a geração de padrões que imitam deformações em outros materiais macios, como couro, durante a animação. As soluções existentes para simular rugas e pistas de deformação frequentemente dependem de hardware especializado, que é dispendioso e de difícil acesso. Além disso, depender exclusivamente de dados capturados limita a direção artística e dificulta a adaptação a mudanças. Em contraste, a solução proposta permite a síntese dinâmica de texturas que se adaptam às deformações subjacentes da malha de forma fisicamente plausível. Vários métodos foram explorados para sintetizar rugas diretamente na geometria, mas sofrem de limitações como auto-interseções e maiores requisitos de armazenamento. A intervenção manual de artistas na criação de mapas de rugas e mapas de tensão permite controle, mas pode ser limitada em deformações complexas ou onde maior realismo seja necessário. O nosso trabalho destaca o potencial dos métodos procedimentais para aprimorar a geração de padrões de deformação dinâmica, incluindo rugas, com maior controle criativo e sem depender de dados capturados. A incorporação de padrões procedimentais estáticos melhora o realismo, e a abordagem pode ser estendida além da pele para outros materiais macios.We propose a solution for generating dynamic heightmap data to simulate deformations for soft surfaces, with a focus on human skin. The solution incorporates mesostructure-level wrinkles and utilizes procedural textures to add static microstructure details. It offers flexibility beyond human skin, enabling the generation of patterns mimicking deformations in other soft materials, such as leater, during animation. Existing solutions for simulating wrinkles and deformation cues often rely on specialized hardware, which is costly and not easily accessible. Moreover, relying solely on captured data limits artistic direction and hinders adaptability to changes. In contrast, our proposed solution provides dynamic texture synthesis that adapts to underlying mesh deformations. Various methods have been explored to synthesize wrinkles directly to the geometry, but they suffer from limitations such as self-intersections and increased storage requirements. Manual intervention by artists using wrinkle maps and tension maps provides control but may be limited to the physics-based simulations. Our research presents the potential of procedural methods to enhance the generation of dynamic deformation patterns, including wrinkles, with greater creative control and without reliance on captured data. Incorporating static procedural patterns improves realism, and the approach can be extended to other soft-materials beyond skin

Measuring and simulating haemodynamics due to geometric changes in facial expression

The human brain has evolved to be very adept at recognising imperfections in human skin. In particular, observing someone’s facial skin appearance is important in recognising when someone is ill, or when finding a suitable mate. It is therefore a key goal of computer graphics research to produce highly realistic renderings of skin. However, the optical processes that give rise to skin appearance are complex and subtle. To address this, computer graphics research has incorporated more and more sophisticated models of skin reflectance. These models are generally based on static concentrations of skin chromophores; melanin and haemoglobin. However, haemoglobin concentrations are far from static, as blood flow is directly caused by both changes in facial expression and emotional state. In this thesis, we explore how blood flow changes as a consequence of changing facial expression with the aim of producing more accurate models of skin appearance. To build an accurate model of blood flow, we base it on real-world measurements of blood concentrations over time. We describe, in detail, the steps required to obtain blood concentrations from photographs of a subject. These steps are then used to measure blood concentration maps for a series of expressions that define a wide gamut of human expression. From this, we define a blending algorithm that allows us to interpolate these maps to generate concentrations for other expressions. This technique, however, requires specialist equipment to capture the maps in the first place. We try to rectify this problem by investigating a direct link between changes in facial geometry and haemoglobin concentrations. This requires building a unique capture device that captures both simultaneously. Our analysis hints a direct linear connection between the two, paving the way for further investigatio

A portable capturing system for image-based relighting.

Pang Wai Man.Thesis submitted in: July 2002.Thesis (M.Phil.)--Chinese University of Hong Kong, 2003.Includes bibliographical references (leaves 108-114).Abstracts in English and Chinese.Abstract --- p.iiAcknowledgments --- p.ivChapter 1 --- Introduction --- p.1Chapter 1.1 --- Image-based Rendering and Modeling --- p.1Chapter 1.1.1 --- Image-based versus Geometry-based --- p.5Chapter 1.2 --- Capturing for Graphics --- p.6Chapter 1.3 --- Organization of this Thesis --- p.8Chapter 2 --- Image-based Rendering and Relighting --- p.10Chapter 2.1 --- Theoretical Concepts --- p.11Chapter 2.1.1 --- Plenoptic Illumination Function --- p.11Chapter 2.1.2 --- Apparent BRDF --- p.13Chapter 2.1.3 --- Types of lighting --- p.14Chapter 2.1.4 --- Image superposition --- p.16Chapter 2.2 --- General Rendering Pipeline --- p.18Chapter 2.3 --- Rendering Techniques --- p.21Chapter 2.3.1 --- Nearest Neighbours and Interpolation --- p.21Chapter 2.3.2 --- Image Warping --- p.23Chapter 2.4 --- IBR Representations and applications --- p.26Chapter 2.4.1 --- Navigation --- p.28Chapter 2.4.2 --- Relighting Representations --- p.35Chapter 2.4.3 --- High Dynamic Range Imaging --- p.38Chapter 2.5 --- Chapter Summary --- p.42Chapter 3 --- Capturing Methods --- p.44Chapter 3.1 --- Spatial Tracking Approaches --- p.45Chapter 3.1.1 --- Mechanical based Method --- p.46Chapter 3.1.2 --- Electromagnetic based Method --- p.48Chapter 3.1.3 --- Vision based Method --- p.50Chapter 3.1.4 --- Comparison --- p.51Chapter 3.2 --- High Dynamic Range Imaging --- p.53Chapter 3.2.1 --- Successive Exposure Capturing --- p.53Chapter 3.2.2 --- Spatial Varing Filter --- p.53Chapter 3.2.3 --- Special Designed Hardware --- p.55Chapter 3.3 --- Chapter Summary --- p.56Chapter 4 --- System Design and Implementation --- p.58Chapter 4.1 --- System Overview --- p.58Chapter 4.2 --- The Setup --- p.60Chapter 4.3 --- Capturing Procedures --- p.61Chapter 4.3.1 --- Calibrations --- p.61Chapter 4.4 --- Vision based tracking --- p.64Chapter 4.4.1 --- The pin-hole camera model --- p.65Chapter 4.4.2 --- Basics of Camera Calibration --- p.66Chapter 4.5 --- Light Vector Tracking --- p.70Chapter 4.5.1 --- The Transformations --- p.70Chapter 4.5.2 --- Tracking Accuracy --- p.71Chapter 4.5.3 --- Tracking Range Enlargement --- p.72Chapter 4.6 --- Capturing Experiment --- p.74Chapter 4.7 --- Sampling Analysis --- p.74Chapter 4.8 --- Chapter Summary --- p.78Chapter 5 --- Data Postprocessing --- p.80Chapter 5.1 --- Scattered Data Fitting --- p.81Chapter 5.1.1 --- Spherical Delaunay Triangulation --- p.83Chapter 5.1.2 --- Interpolation on Sphere --- p.86Chapter 5.2 --- Compression --- p.88Chapter 5.3 --- Chapter Summary --- p.90Chapter 6 --- Relit Results --- p.91Chapter 6.1 --- Relighting with Multiple Directional Lights --- p.92Chapter 6.2 --- Relighting with Environmental Maps --- p.94Chapter 7 --- Conclusion --- p.101Chapter 7.1 --- Future Research Aspect --- p.102Chapter A --- System User Guide --- p.104Chapter A.1 --- Equipment Configuration --- p.104Chapter A.2 --- Operation Guide --- p.105Chapter A.3 --- Software Components --- p.106Chapter A.3.1 --- Image capturing - lightcap --- p.106Chapter A.3.2 --- Raw Frame Extraction ´ؤ lfprocess --- p.107Chapter A.3.3 --- Resampling and Compression - svscatterppm2urdf . --- p.107Bibliography --- p.10