5,948 research outputs found
End-to-End Learning of Video Super-Resolution with Motion Compensation
Learning approaches have shown great success in the task of super-resolving
an image given a low resolution input. Video super-resolution aims for
exploiting additionally the information from multiple images. Typically, the
images are related via optical flow and consecutive image warping. In this
paper, we provide an end-to-end video super-resolution network that, in
contrast to previous works, includes the estimation of optical flow in the
overall network architecture. We analyze the usage of optical flow for video
super-resolution and find that common off-the-shelf image warping does not
allow video super-resolution to benefit much from optical flow. We rather
propose an operation for motion compensation that performs warping from low to
high resolution directly. We show that with this network configuration, video
super-resolution can benefit from optical flow and we obtain state-of-the-art
results on the popular test sets. We also show that the processing of whole
images rather than independent patches is responsible for a large increase in
accuracy.Comment: Accepted to GCPR201
Context-aware Synthesis for Video Frame Interpolation
Video frame interpolation algorithms typically estimate optical flow or its
variations and then use it to guide the synthesis of an intermediate frame
between two consecutive original frames. To handle challenges like occlusion,
bidirectional flow between the two input frames is often estimated and used to
warp and blend the input frames. However, how to effectively blend the two
warped frames still remains a challenging problem. This paper presents a
context-aware synthesis approach that warps not only the input frames but also
their pixel-wise contextual information and uses them to interpolate a
high-quality intermediate frame. Specifically, we first use a pre-trained
neural network to extract per-pixel contextual information for input frames. We
then employ a state-of-the-art optical flow algorithm to estimate bidirectional
flow between them and pre-warp both input frames and their context maps.
Finally, unlike common approaches that blend the pre-warped frames, our method
feeds them and their context maps to a video frame synthesis neural network to
produce the interpolated frame in a context-aware fashion. Our neural network
is fully convolutional and is trained end to end. Our experiments show that our
method can handle challenging scenarios such as occlusion and large motion and
outperforms representative state-of-the-art approaches.Comment: CVPR 2018, http://graphics.cs.pdx.edu/project/ctxsy
- …