7 research outputs found
End-to-end Projector Photometric Compensation
Projector photometric compensation aims to modify a projector input image
such that it can compensate for disturbance from the appearance of projection
surface. In this paper, for the first time, we formulate the compensation
problem as an end-to-end learning problem and propose a convolutional neural
network, named CompenNet, to implicitly learn the complex compensation
function. CompenNet consists of a UNet-like backbone network and an autoencoder
subnet. Such architecture encourages rich multi-level interactions between the
camera-captured projection surface image and the input image, and thus captures
both photometric and environment information of the projection surface. In
addition, the visual details and interaction information are carried to deeper
layers along the multi-level skip convolution layers. The architecture is of
particular importance for the projector compensation task, for which only a
small training dataset is allowed in practice. Another contribution we make is
a novel evaluation benchmark, which is independent of system setup and thus
quantitatively verifiable. Such benchmark is not previously available, to our
best knowledge, due to the fact that conventional evaluation requests the
hardware system to actually project the final results. Our key idea, motivated
from our end-to-end problem formulation, is to use a reasonable surrogate to
avoid such projection process so as to be setup-independent. Our method is
evaluated carefully on the benchmark, and the results show that our end-to-end
learning solution outperforms state-of-the-arts both qualitatively and
quantitatively by a significant margin.Comment: To appear in the 2019 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). Source code and dataset are available at
https://github.com/BingyaoHuang/compenne
CompenNet++: End-to-end Full Projector Compensation
Full projector compensation aims to modify a projector input image such that
it can compensate for both geometric and photometric disturbance of the
projection surface. Traditional methods usually solve the two parts separately,
although they are known to correlate with each other. In this paper, we propose
the first end-to-end solution, named CompenNet++, to solve the two problems
jointly. Our work non-trivially extends CompenNet, which was recently proposed
for photometric compensation with promising performance. First, we propose a
novel geometric correction subnet, which is designed with a cascaded
coarse-to-fine structure to learn the sampling grid directly from photometric
sampling images. Second, by concatenating the geometric correction subset with
CompenNet, CompenNet++ accomplishes full projector compensation and is
end-to-end trainable. Third, after training, we significantly simplify both
geometric and photometric compensation parts, and hence largely improves the
running time efficiency. Moreover, we construct the first setup-independent
full compensation benchmark to facilitate the study on this topic. In our
thorough experiments, our method shows clear advantages over previous arts with
promising compensation quality and meanwhile being practically convenient.Comment: To appear in ICCV 2019. High-res supplementary material:
https://www3.cs.stonybrook.edu/~hling/publication/CompenNet++_sup-high-res.pdf.
Code: https://github.com/BingyaoHuang/CompenNet-plusplu
MoSART: Mobile Spatial Augmented Reality for 3D Interaction With Tangible Objects
In this paper we introduce MoSART, a novel approach for Mobile Spatial Augmented Reality on Tangible objects. MoSART is dedicated to mobile interaction with tangible objects in single or collaborative situations. It is based on a novel “all-in-one” Head-Mounted Display (AMD) including a projector (for the SAR display) and cameras (for the scene registration). Equipped with the HMD the user is able to move freely around tangible objects and manipulate them at will. The system tracks the position and orientation of the tangible 3D objects and projects virtual content over them. The tracking is a feature-based stereo optical tracking providing high accuracy and low latency. A projection mapping technique is used for the projection on the tangible objects which can have a complex 3D geometry. Several interaction tools have also been designed to interact with the tangible and augmented content, such as a control panel and a pointer metaphor, which can benefit as well from the MoSART projection mapping and tracking features. The possibilities offered by our novel approach are illustrated in several use cases, in single or collaborative situations, such as for virtual prototyping, training or medical visualization