593 research outputs found
Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials
We present a method to capture both 3D shape and spatially varying
reflectance with a multi-view photometric stereo (MVPS) technique that works
for general isotropic materials. Our algorithm is suitable for perspective
cameras and nearby point light sources. Our data capture setup is simple, which
consists of only a digital camera, some LED lights, and an optional automatic
turntable. From a single viewpoint, we use a set of photometric stereo images
to identify surface points with the same distance to the camera. We collect
this information from multiple viewpoints and combine it with
structure-from-motion to obtain a precise reconstruction of the complete 3D
shape. The spatially varying isotropic bidirectional reflectance distribution
function (BRDF) is captured by simultaneously inferring a set of basis BRDFs
and their mixing weights at each surface point. In experiments, we demonstrate
our algorithm with two different setups: a studio setup for highest precision
and a desktop setup for best usability. According to our experiments, under the
studio setting, the captured shapes are accurate to 0.5 millimeters and the
captured reflectance has a relative root-mean-square error (RMSE) of 9%. We
also quantitatively evaluate state-of-the-art MVPS on a newly collected
benchmark dataset, which is publicly available for inspiring future research
Shape and Spatially-Varying Reflectance Estimation From Virtual Exemplars
This paper addresses the problem of estimating the shape of objects that
exhibit spatially-varying reflectance. We assume that multiple images of the
object are obtained under a fixed view-point and varying illumination, i.e.,
the setting of photometric stereo. At the core of our techniques is the
assumption that the BRDF at each pixel lies in the non-negative span of a known
BRDF dictionary.This assumption enables a per-pixel surface normal and BRDF
estimation framework that is computationally tractable and requires no
initialization in spite of the underlying problem being non-convex. Our
estimation framework first solves for the surface normal at each pixel using a
variant of example-based photometric stereo. We design an efficient multi-scale
search strategy for estimating the surface normal and subsequently, refine this
estimate using a gradient descent procedure. Given the surface normal estimate,
we solve for the spatially-varying BRDF by constraining the BRDF at each pixel
to be in the span of the BRDF dictionary, here, we use additional priors to
further regularize the solution. A hallmark of our approach is that it does not
require iterative optimization techniques nor the need for careful
initialization, both of which are endemic to most state-of-the-art techniques.
We showcase the performance of our technique on a wide range of simulated and
real scenes where we outperform competing methods.Comment: PAMI minor revision. arXiv admin note: substantial text overlap with
arXiv:1503.0426
A Dictionary-based Approach for Estimating Shape and Spatially-Varying Reflectance
We present a technique for estimating the shape and reflectance of an object
in terms of its surface normals and spatially-varying BRDF. We assume that
multiple images of the object are obtained under fixed view-point and varying
illumination, i.e, the setting of photometric stereo. Assuming that the BRDF at
each pixel lies in the non-negative span of a known BRDF dictionary, we derive
a per-pixel surface normal and BRDF estimation framework that requires neither
iterative optimization techniques nor careful initialization, both of which are
endemic to most state-of-the-art techniques. We showcase the performance of our
technique on a wide range of simulated and real scenes where we outperform
competing methods.Comment: IEEE Intl. Conf. Computational Photography, 201
SPLINE-Net: Sparse Photometric Stereo through Lighting Interpolation and Normal Estimation Networks
This paper solves the Sparse Photometric stereo through Lighting
Interpolation and Normal Estimation using a generative Network (SPLINE-Net).
SPLINE-Net contains a lighting interpolation network to generate dense lighting
observations given a sparse set of lights as inputs followed by a normal
estimation network to estimate surface normals. Both networks are jointly
constrained by the proposed symmetric and asymmetric loss functions to enforce
isotropic constrain and perform outlier rejection of global illumination
effects. SPLINE-Net is verified to outperform existing methods for photometric
stereo of general BRDFs by using only ten images of different lights instead of
using nearly one hundred images.Comment: Accepted to ICCV 201
Neural Inverse Rendering for General Reflectance Photometric Stereo
We present a novel convolutional neural network architecture for photometric
stereo (Woodham, 1980), a problem of recovering 3D object surface normals from
multiple images observed under varying illuminations. Despite its long history
in computer vision, the problem still shows fundamental challenges for surfaces
with unknown general reflectance properties (BRDFs). Leveraging deep neural
networks to learn complicated reflectance models is promising, but studies in
this direction are very limited due to difficulties in acquiring accurate
ground truth for training and also in designing networks invariant to
permutation of input images. In order to address these challenges, we propose a
physics based unsupervised learning framework where surface normals and BRDFs
are predicted by the network and fed into the rendering equation to synthesize
observed images. The network weights are optimized during testing by minimizing
reconstruction loss between observed and synthesized images. Thus, our learning
process does not require ground truth normals or even pre-training on external
images. Our method is shown to achieve the state-of-the-art performance on a
challenging real-world scene benchmark.Comment: To appear in International Conference on Machine Learning 2018 (ICML
2018). 10 pages + 20 pages (appendices
Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image
We propose a material acquisition approach to recover the spatially-varying
BRDF and normal map of a near-planar surface from a single image captured by a
handheld mobile phone camera. Our method images the surface under arbitrary
environment lighting with the flash turned on, thereby avoiding shadows while
simultaneously capturing high-frequency specular highlights. We train a CNN to
regress an SVBRDF and surface normals from this image. Our network is trained
using a large-scale SVBRDF dataset and designed to incorporate physical
insights for material estimation, including an in-network rendering layer to
model appearance and a material classifier to provide additional supervision
during training. We refine the results from the network using a dense CRF
module whose terms are designed specifically for our task. The framework is
trained end-to-end and produces high quality results for a variety of
materials. We provide extensive ablation studies to evaluate our network on
both synthetic and real data, while demonstrating significant improvements in
comparisons with prior works.Comment: submitted to European Conference on Computer Visio
A Differential Volumetric Approach to Multi-View Photometric Stereo
Highly accurate 3D volumetric reconstruction is still an open research topic
where the main difficulty is usually related to merging some rough estimations
with high frequency details. One of the most promising methods is the fusion
between multi-view stereo and photometric stereo images. Beside the intrinsic
difficulties that multi-view stereo and photometric stereo in order to work
reliably, supplementary problems arise when considered together.
In this work, we present a volumetric approach to the multi-view photometric
stereo problem. The key point of our method is the signed distance field
parameterisation and its relation to the surface normal. This is exploited in
order to obtain a linear partial differential equation which is solved in a
variational framework, that combines multiple images from multiple points of
view in a single system. In addition, the volumetric approach is naturally
implemented on an octree, which allows for fast ray-tracing that reliably
alleviates occlusions and cast shadows.
Our approach is evaluated on synthetic and real data-sets and achieves
state-of-the-art results
CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces
Most conventional photometric stereo algorithms inversely solve a BRDF-based
image formation model. However, the actual imaging process is often far more
complex due to the global light transport on the non-convex surfaces. This
paper presents a photometric stereo network that directly learns relationships
between the photometric stereo input and surface normals of a scene. For
handling unordered, arbitrary number of input images, we merge all the input
data to the intermediate representation called {\it observation map} that has a
fixed shape, is able to be fed into a CNN. To improve both training and
prediction, we take into account the rotational pseudo-invariance of the
observation map that is derived from the isotropic constraint. For training the
network, we create a synthetic photometric stereo dataset that is generated by
a physics-based renderer, therefore the global light transport is considered.
Our experimental results on both synthetic and real datasets show that our
method outperforms conventional BRDF-based photometric stereo algorithms
especially when scenes are highly non-convex.Comment: Accepted in ECCV 2018 (ECCV2018). Source code and supplementary are
available at https://github.com/satoshi-ikehata/CNN-P
Photometric Stereo by Hemispherical Metric Embedding
Photometric Stereo methods seek to reconstruct the 3d shape of an object from
motionless images obtained with varying illumination. Most existing methods
solve a restricted problem where the physical reflectance model, such as
Lambertian reflectance, is known in advance. In contrast, we do not restrict
ourselves to a specific reflectance model. Instead, we offer a method that
works on a wide variety of reflectances. Our approach uses a simple yet
uncommonly used property of the problem - the sought after normals are points
on a unit hemisphere. We present a novel embedding method that maps pixels to
normals on the unit hemisphere. Our experiments demonstrate that this approach
outperforms existing manifold learning methods for the task of hemisphere
embedding. We further show successful reconstructions of objects from a wide
variety of reflectances including smooth, rough, diffuse and specular surfaces,
even in the presence of significant attached shadows. Finally, we empirically
prove that under these challenging settings we obtain more accurate shape
reconstructions than existing methods
Deep Photometric Stereo for Non-Lambertian Surfaces
This paper addresses the problem of photometric stereo, in both calibrated
and uncalibrated scenarios, for non-Lambertian surfaces based on deep learning.
We first introduce a fully convolutional deep network for calibrated
photometric stereo, which we call PS-FCN. Unlike traditional approaches that
adopt simplified reflectance models to make the problem tractable, our method
directly learns the mapping from reflectance observations to surface normal,
and is able to handle surfaces with general and unknown isotropic reflectance.
At test time, PS-FCN takes an arbitrary number of images and their associated
light directions as input and predicts a surface normal map of the scene in a
fast feed-forward pass. To deal with the uncalibrated scenario where light
directions are unknown, we introduce a new convolutional network, named LCNet,
to estimate light directions from input images. The estimated light directions
and the input images are then fed to PS-FCN to determine the surface normals.
Our method does not require a pre-defined set of light directions and can
handle multiple images in an order-agnostic manner. Thorough evaluation of our
approach on both synthetic and real datasets shows that it outperforms
state-of-the-art methods in both calibrated and uncalibrated scenarios
- …