6 research outputs found
Depth Assisted Full Resolution Network for Single Image-based View Synthesis
Researches in novel viewpoint synthesis majorly focus on interpolation from
multi-view input images. In this paper, we focus on a more challenging and
ill-posed problem that is to synthesize novel viewpoints from one single input
image. To achieve this goal, we propose a novel deep learning-based technique.
We design a full resolution network that extracts local image features with the
same resolution of the input, which contributes to derive high resolution and
prevent blurry artifacts in the final synthesized images. We also involve a
pre-trained depth estimation network into our system, and thus 3D information
is able to be utilized to infer the flow field between the input and the target
image. Since the depth network is trained by depth order information between
arbitrary pairs of points in the scene, global image features are also involved
into our system. Finally, a synthesis layer is used to not only warp the
observed pixels to the desired positions but also hallucinate the missing
pixels with recorded pixels. Experiments show that our technique performs well
on images of various scenes, and outperforms the state-of-the-art techniques
Softmax Splatting for Video Frame Interpolation
Differentiable image sampling in the form of backward warping has seen broad
adoption in tasks like depth estimation and optical flow prediction. In
contrast, how to perform forward warping has seen less attention, partly due to
additional challenges such as resolving the conflict of mapping multiple pixels
to the same target location in a differentiable way. We propose softmax
splatting to address this paradigm shift and show its effectiveness on the
application of frame interpolation. Specifically, given two input frames, we
forward-warp the frames and their feature pyramid representations based on an
optical flow estimate using softmax splatting. In doing so, the softmax
splatting seamlessly handles cases where multiple source pixels map to the same
target location. We then use a synthesis network to predict the interpolation
result from the warped representations. Our softmax splatting allows us to not
only interpolate frames at an arbitrary time but also to fine tune the feature
pyramid and the optical flow. We show that our synthesis approach, empowered by
softmax splatting, achieves new state-of-the-art results for video frame
interpolation.Comment: CVPR 2020, http://sniklaus.com/softspla