13,421 research outputs found
Joint Material and Illumination Estimation from Photo Sets in the Wild
Faithful manipulation of shape, material, and illumination in 2D Internet
images would greatly benefit from a reliable factorization of appearance into
material (i.e., diffuse and specular) and illumination (i.e., environment
maps). On the one hand, current methods that produce very high fidelity
results, typically require controlled settings, expensive devices, or
significant manual effort. To the other hand, methods that are automatic and
work on 'in the wild' Internet images, often extract only low-frequency
lighting or diffuse materials. In this work, we propose to make use of a set of
photographs in order to jointly estimate the non-diffuse materials and sharp
lighting in an uncontrolled setting. Our key observation is that seeing
multiple instances of the same material under different illumination (i.e.,
environment), and different materials under the same illumination provide
valuable constraints that can be exploited to yield a high-quality solution
(i.e., specular materials and environment illumination) for all the observed
materials and environments. Similar constraints also arise when observing
multiple materials in a single environment, or a single material across
multiple environments. The core of this approach is an optimization procedure
that uses two neural networks that are trained on synthetic images to predict
good gradients in parametric space given observation of reflected light. We
evaluate our method on a range of synthetic and real examples to generate
high-quality estimates, qualitatively compare our results against
state-of-the-art alternatives via a user study, and demonstrate
photo-consistent image manipulation that is otherwise very challenging to
achieve
SurReal: enhancing Surgical simulation Realism using style transfer
Surgical simulation is an increasingly important element of surgical
education. Using simulation can be a means to address some of the significant
challenges in developing surgical skills with limited time and resources. The
photo-realistic fidelity of simulations is a key feature that can improve the
experience and transfer ratio of trainees. In this paper, we demonstrate how we
can enhance the visual fidelity of existing surgical simulation by performing
style transfer of multi-class labels from real surgical video onto synthetic
content. We demonstrate our approach on simulations of cataract surgery using
real data labels from an existing public dataset. Our results highlight the
feasibility of the approach and also the powerful possibility to extend this
technique to incorporate additional temporal constraints and to different
applications
- …