Joint Material and Illumination Estimation from Photo Sets in the Wild

Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve

A Novel Framework for Highlight Reflectance Transformation Imaging

We propose a novel pipeline and related software tools for processing the multi-light image collections (MLICs) acquired in different application contexts to obtain shape and appearance information of captured surfaces, as well as to derive compact relightable representations of them. Our pipeline extends the popular Highlight Reflectance Transformation Imaging (H-RTI) framework, which is widely used in the Cultural Heritage domain. We support, in particular, perspective camera modeling, per-pixel interpolated light direction estimation, as well as light normalization correcting vignetting and uneven non-directional illumination. Furthermore, we propose two novel easy-to-use software tools to simplify all processing steps. The tools, in addition to support easy processing and encoding of pixel data, implement a variety of visualizations, as well as multiple reflectance-model-fitting options. Experimental tests on synthetic and real-world MLICs demonstrate the usefulness of the novel algorithmic framework and the potential benefits of the proposed tools for end-user applications.Terms: "European Union (EU)" & "Horizon 2020" / Action: H2020-EU.3.6.3. - Reflective societies - cultural heritage and European identity / Acronym: Scan4Reco / Grant number: 665091DSURF project (PRIN 2015) funded by the Italian Ministry of University and ResearchSardinian Regional Authorities under projects VIGEC and Vis&VideoLa

What Is Around The Camera?

How much does a single image reveal about the environment it was taken in? In this paper, we investigate how much of that information can be retrieved from a foreground object, combined with the background (i.e. the visible part of the environment). Assuming it is not perfectly diffuse, the foreground object acts as a complexly shaped and far-from-perfect mirror. An additional challenge is that its appearance confounds the light coming from the environment with the unknown materials it is made of. We propose a learning-based approach to predict the environment from multiple reflectance maps that are computed from approximate surface normals. The proposed method allows us to jointly model the statistics of environments and material properties. We train our system from synthesized training data, but demonstrate its applicability to real-world data. Interestingly, our analysis shows that the information obtained from objects made out of multiple materials often is complementary and leads to better performance.Comment: Accepted to ICCV. Project: http://homes.esat.kuleuven.be/~sgeorgou/multinatillum

Towards dynamic camera calibration for constrained flexible mirror imaging

Flexible mirror imaging systems consisting of a perspective camera viewing a scene reflected in a flexible mirror can provide direct control over image field-of-view and resolution. However, calibration of such systems is difficult due to the vast range of possible mirror shapes and the flexible nature of the system. This paper proposes the fundamentals of a dynamic calibration approach for flexible mirror imaging systems by examining the constrained case of single dimensional flexing. The calibration process consists of an initial primary calibration stage followed by in-service dynamic calibration. Dynamic calibration uses a linear approximation to initialise a non-linear minimisation step, the result of which is the estimate of the mirror surface shape. The method is easier to implement than existing calibration methods for flexible mirror imagers, requiring only two images of a calibration grid for each dynamic calibration update. Experimental results with both simulated and real data are presented that demonstrate the capabilities of the proposed approach

Estimation of forest variables using airborne laser scanning

Airborne laser scanning can provide three-dimensional measurements of the forest canopy with high efficiency and precision. There are presently a large number of airborne laser scanning instruments in operation. The aims of the studies reported in this thesis were, to develop and validate methods for estimation of forest variables using laser data, and to investigate the influence of laser system parameters on the estimates. All studies were carried out in hemi-boreal forest at a test area in southwestern Sweden (lat. 58°30’N, long. 13°40’ E). Forest variables were estimated using regression models. On plot level, the Root Mean Square Error (RMSE) for mean tree height estimations ranged between 6% and 11% of the average value for different datasets and methods. The RMSE for stem volume estimations ranged between 19% and 26% of the average value for different datasets and methods. On stand level (area 0.64 ha), the RMSE was 3% and 11% of the average value for mean tree height and stem volume estimations, respectively. A simulation model was used to investigate the effect of different scanning angles on laser measurement of tree height and canopy closure. The effect of different scanning angles was different within different simulated forest types, e.g., different tree species. High resolution laser data were used for detection of individual trees. In total, 71% of the field measurements were detected representing 91% of the total stem volume. Height and crown diameter of the detected trees could be estimated with a RMSE of 0.63 m and 0.61 m, respectively. The magnitude of the height estimation errors was similar to what is usually achieved using field inventory. Using different laser footprint diameters (0.26 to 3.68 m) gave similar estimation accuracies. The tree species Norway spruce (Picea abies L. Karst.) and Scots pine (Pinus sylvestris L.) were discriminated at individual tree level with an accuracy of 95%. The results in this thesis show that airborne laser scanners are useful as forest inventory tools. Forest variables can be estimated on tree level, plot level and stand level with similar accuracies as traditional field inventories