2 research outputs found
DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
Image-based relighting, projector compensation and depth/normal
reconstruction are three important tasks of projector-camera systems (ProCams)
and spatial augmented reality (SAR). Although they share a similar pipeline of
finding projector-camera image mappings, in tradition, they are addressed
independently, sometimes with different prerequisites, devices and sampling
images. In practice, this may be cumbersome for SAR applications to address
them one-by-one. In this paper, we propose a novel end-to-end trainable model
named DeProCams to explicitly learn the photometric and geometric mappings of
ProCams, and once trained, DeProCams can be applied simultaneously to the three
tasks. DeProCams explicitly decomposes the projector-camera image mappings into
three subprocesses: shading attributes estimation, rough direct light
estimation and photorealistic neural rendering. A particular challenge
addressed by DeProCams is occlusion, for which we exploit epipolar constraint
and propose a novel differentiable projector direct light mask. Thus, it can be
learned end-to-end along with the other modules. Afterwards, to improve
convergence, we apply photometric and geometric constraints such that the
intermediate results are plausible. In our experiments, DeProCams shows clear
advantages over previous arts with promising quality and meanwhile being fully
differentiable. Moreover, by solving the three tasks in a unified model,
DeProCams waives the need for additional optical devices, radiometric
calibrations and structured light.Comment: Source code and supplementary material at:
https://github.com/BingyaoHuang/DeProCam
End-to-end Full Projector Compensation
Full projector compensation aims to modify a projector input image to
compensate for both geometric and photometric disturbance of the projection
surface. Traditional methods usually solve the two parts separately and may
suffer from suboptimal solutions. In this paper, we propose the first
end-to-end differentiable solution, named CompenNeSt++, to solve the two
problems jointly. First, we propose a novel geometric correction subnet, named
WarpingNet, which is designed with a cascaded coarse-to-fine structure to learn
the sampling grid directly from sampling images. Second, we propose a novel
photometric compensation subnet, named CompenNeSt, which is designed with a
siamese architecture to capture the photometric interactions between the
projection surface and the projected images, and to use such information to
compensate the geometrically corrected images. By concatenating WarpingNet with
CompenNeSt, CompenNeSt++ accomplishes full projector compensation and is
end-to-end trainable. Third, to improve practicability, we propose a novel
synthetic data-based pre-training strategy to significantly reduce the number
of training images and training time. Moreover, we construct the first
setup-independent full compensation benchmark to facilitate future studies. In
thorough experiments, our method shows clear advantages over prior art with
promising compensation quality and meanwhile being practically convenient.Comment: Source code: https://github.com/BingyaoHuang/CompenNeSt-plusplus.
arXiv admin note: text overlap with arXiv:1908.06246, arXiv:1904.0433