Revisiting Gray Pixel for Statistical Illumination Estimation

We present a statistical color constancy method that relies on novel gray pixel detection and mean shift clustering. The method, called Mean Shifted Grey Pixel -- MSGP, is based on the observation: true-gray pixels are aligned towards one single direction. Our solution is compact, easy to compute and requires no training. Experiments on two real-world benchmarks show that the proposed approach outperforms state-of-the-art methods in the camera-agnostic scenario. In the setting where the camera is known, MSGP outperforms all statistical methods.Comment: updated and will appear in VISSAP 2019 (long paper

Measuring perceived gloss of rough surfaces

This thesis is concerned with the visual perception of glossy rough surfaces, specifically those characterised by 1/fB noise. Computer graphics were used to model these natural looking surfaces, which were generated and animated to provide realistic stimuli for observers. Different methods were employed to investigate the effects of varying surface roughness and reflection model parameters on perceived gloss. We first investigated how the perceived gloss of a matte Lambertian surface varies with RMS roughness. Then we estimated the perceived gloss of moderate RMS height surfaces rendered using a gloss reflection model. We found that adjusting parameters of the gloss reflection model on the moderate RMS height surfaces produces similar levels of gloss to the high RMS height Lambertian surfaces. More realistic stimuli were modelled using improvements in the reflection model, rendering technique, illumination and viewing conditions. In contrast with previous research, a non-monotonic relationship was found between perceived gloss and mesoscale roughness when microscale parameters were held constant. Finally, the joint effect of variations in mesoscale roughness (surface geometry) and microscale roughness (reflection model) on perceived gloss was investigated and tested against conjoint measurement models. It was concluded that perceived gloss of rough surfaces is significantly affected by surface roughness in both mesoscale and microscale and can be described by a full conjoint measurement model

Acquisition, Modeling, and Augmentation of Reflectance for Synthetic Optical Flow Reference Data

This thesis is concerned with the acquisition, modeling, and augmentation of material reflectance to simulate high-fidelity synthetic data for computer vision tasks. The topic is covered in three chapters: I commence with exploring the upper limits of reflectance acquisition. I analyze state-of-the-art BTF reflectance field renderings and show that they can be applied to optical flow performance analysis with closely matching performance to real-world images. Next, I present two methods for fitting efficient BRDF reflectance models to measured BTF data. Both methods combined retain all relevant reflectance information as well as the surface normal details on a pixel level. I further show that the resulting synthesized images are suited for optical flow performance analysis, with a virtually identical performance for all material types. Finally, I present a novel method for augmenting real-world datasets with physically plausible precipitation effects, including ground surface wetting, water droplets on the windshield, and water spray and mists. This is achieved by projecting the realworld image data onto a reconstructed virtual scene, manipulating the scene and the surface reflectance, and performing unbiased light transport simulation of the precipitation effects

Frequency Based Radiance Cache for Rendering Animations

International audienceWe propose a method to render animation sequences with direct distant lighting that only shades a fraction of the total pixels. We leverage frequency-based analyses of light transport to determine shading and image sampling rates across an animation using a samples cache. To do so, we derive frequency bandwidths that account for the complexity of distant lights, visibility, BRDF, and temporal coherence during animation. We finaly apply a cross-bilateral filter when rendering our final images from sparse sets of shading points placed according to our frequency-based oracles (generally < 25% of the pixels, per frame)

Intuitive and Accurate Material Appearance Design and Editing

Creating and editing high-quality materials for photorealistic rendering can be a difficult task due to the diversity and complexity of material appearance. Material design is the process by which artists specify the reflectance properties of a surface, such as its diffuse color and specular roughness. Even with the support of commercial software packages, material design can be a time-consuming trial-and-error task due to the counter-intuitive nature of the complex reflectance models. Moreover, many material design tasks require the physical realization of virtually designed materials as the final step, which makes the process even more challenging due to rendering artifacts and the limitations of fabrication. In this dissertation, we propose a series of studies and novel techniques to improve the intuitiveness and accuracy of material design and editing. Our goal is to understand how humans visually perceive materials, simplify user interaction in the design process and, and improve the accuracy of the physical fabrication of designs. Our first work focuses on understanding the perceptual dimensions for measured material data. We build a perceptual space based on a low-dimensional reflectance manifold that is computed from crowd-sourced data using a multi-dimensional scaling model. Our analysis shows the proposed perceptual space is consistent with the physical interpretation of the measured data. We also put forward a new material editing interface that takes advantage of the proposed perceptual space. We visualize each dimension of the manifold to help users understand how it changes the material appearance. Our second work investigates the relationship between translucency and glossiness in material perception. We conduct two human subject studies to test if subsurface scattering impacts gloss perception and examine how the shape of an object influences this perception. Based on our results, we discuss why it is necessary to include transparent and translucent media for future research in gloss perception and material design. Our third work addresses user interaction in the material design system. We present a novel Augmented Reality (AR) material design prototype, which allows users to visualize their designs against a real environment and lighting. We believe introducing AR technology can make the design process more intuitive and improve the authenticity of the results for both novice and experienced users. To test this assumption, we conduct a user study to compare our prototype with the traditional material design system with gray-scale background and synthetic lighting. The results demonstrate that with the help of AR techniques, users perform better in terms of objectively measured accuracy and time and they are subjectively more satisfied with their results. Finally, our last work turns to the challenge presented by the physical realization of designed materials. We propose a learning-based solution to map the virtually designed appearance to a meso-scale geometry that can be easily fabricated. Essentially, this is a fitting problem, but compared with previous solutions, our method can provide the fabrication recipe with higher reconstruction accuracy for a large fitting gamut. We demonstrate the efficacy of our solution by comparing the reconstructions with existing solutions and comparing fabrication results with the original design. We also provide an application of bi-scale material editing using the proposed method

Characterization and visualization of reflective properties of surfaces

Images play a vital role in several fields of natural science research, including biology, physics, astrophysics, and computer science. In the natural sciences, images are commonly used in measurements or documentation; such applications include images made with telescopes, optical microscopes, or electron microscopes. In the humanities, images also play an important role in research. In art history, for example, many different types of images, from photos of small objects to three-dimensional reconstructions of buildings, help art historians to develop theories, to discuss them with other scholars, and to document the current state of artworks, e.g. in the process of restoration. This is particularly useful if the object is not easily accessible, in which case a common solution is to work with photographs. Digital photography has simplified the process of visual representation, because digital images can be easily shared and made accessible. However, when it comes to more complex kinds of artworks like mosaics, these static and two-dimensional images are not able to reproduce the actual visual impression of the object. Similar considerations apply to a variety of other artifacts, such as early prints, books, parchments, and textiles. The challenge in the digitization of of these objects lies in their complex surface properties and reflection behavior. A promising way to solve those limitations is the use of Reflectance Transformation Imaging. RTI is a set of computational photographic methods that capture a subject’s surface shape and color, making it possible to interactively re-light the subject from any direction by means of a mathematical model. The major drawback of RTI is the limitation of the applied mathematical model. Other drawbacks are the RTI imaging workflow and the fact that display of RTI requires a particular stand-alone application. In this thesis, the author developed a data-driven scientific approach to reproduce surfaces composed of lambertian and glossy materials using the RTI technique with as few parameters as possible. This new approach has been called eRTI (enhanced Reflection Transformation Imaging). Furthermore the hardware needed to acquire RTI and eRTI has been improved, by collaborating with a local Swiss firm to develop a novel solution for image acquisition. Lastly a web-based viewer has been developed, to render eRTI images in any standard web browser, even on most mobile devices. The qualities of eRTI have been tested using a novel approach that includes a quantitative and a qualitative method. The results show agreement between the techniques

Light Fields Reconstructing Geometry and Reflectance Properties

Computer vision plays an important role in the progress of automation and digitalization of our society. One of the key challenges is the creation of accurate 3D representations of our environment. The rich information in light fields can enable highly accurate depth estimates, but requires the development of new algorithms. Especially specular reflections pose a challenge for many reconstruction algorithms. This is due to the violation of the brightness consistency assumption, which only holds for Lambertian surfaces. Most surfaces are to some extent specular and an appropriate handling is central to avoid erroneous depth maps. In this thesis we explore the potential of using specular highlights to determine the orientation of surfaces. To this end, we examine epipolar images in light field set ups. In light field data, reflectance properties can be characterized by intensity variations in the epipolar plane space. This space is analysed and compared to the expected reflectance, which is modelled using the render equation with different bidirectional reflection distribution functions. This approach allows us to infer highly accurate surface normals and depth estimates. Furthermore, it reveals material properties encoded in the reflectance by inspecting the intensity profile. Our results demonstrate the potential to increase the accuracy of the depth maps. Multiple cameras in a light field set up let us retrieve additional material properties encoded in the reflectance

Feature-based image patch classification for moving shadow detection

Moving object detection is a first step towards many computer vision applications, such as human interaction and tracking, video surveillance, and traffic monitoring systems. Accurate estimation of the target object’s size and shape is often required before higher-level tasks (e.g., object tracking or recog nition) can be performed. However, these properties can be derived only when the foreground object is detected precisely. Background subtraction is a common technique to extract foreground objects from image sequences. The purpose of background subtraction is to detect changes in pixel values within a given frame. The main problem with background subtraction and other related object detection techniques is that cast shadows tend to be misclassified as either parts of the foreground objects (if objects and their cast shadows are bonded together) or independent foreground objects (if objects and shadows are separated). The reason for this phenomenon is the presence of similar characteristics between the target object and its cast shadow, i.e., shadows have similar motion, attitude, and intensity changes as the moving objects that cast them. Detecting shadows of moving objects is challenging because of problem atic situations related to shadows, for example, chromatic shadows, shadow color blending, foreground-background camouflage, nontextured surfaces and dark surfaces. Various methods for shadow detection have been proposed in the liter ature to address these problems. Many of these methods use general-purpose image feature descriptors to detect shadows. These feature descriptors may be effective in distinguishing shadow points from the foreground object in a specific problematic situation; however, such methods often fail to distinguish shadow points from the foreground object in other situations. In addition, many of these moving shadow detection methods require prior knowledge of the scene condi tions and/or impose strong assumptions, which make them excessively restrictive in practice. The aim of this research is to develop an efficient method capable of addressing possible environmental problems associated with shadow detection while simultaneously improving the overall accuracy and detection stability. In this research study, possible problematic situations for dynamic shad ows are addressed and discussed in detail. On the basis of the analysis, a ro bust method, including change detection and shadow detection, is proposed to address these environmental problems. A new set of two local feature descrip tors, namely, binary patterns of local color constancy (BPLCC) and light-based gradient orientation (LGO), is introduced to address the identified problematic situations by incorporating intensity, color, texture, and gradient information. The feature vectors are concatenated in a column-by-column manner to con struct one dictionary for the objects and another dictionary for the shadows. A new sparse representation framework is then applied to find the nearest neighbor of the test image segment by computing a weighted linear combination of the reference dictionary. Image segment classification is then performed based on the similarity between the test image and the sparse representations of the two classes. The performance of the proposed framework on common shadow detec tion datasets is evaluated, and the method shows improved performance com pared with state-of-the-art methods in terms of the shadow detection rate, dis crimination rate, accuracy, and stability. By achieving these significant improve ments, the proposed method demonstrates its ability to handle various problems associated with image processing and accomplishes the aim of this thesis