Bits on the wire not only impact video quality delivered to customers but also drive the costs of video streaming services. This project aims at building state-of-the-art deep-learning-based video super-resolution (VSR) algorithms while addressing the compression artifact, and then integrate the VSR into the Scalable Extension of High Efficiency Video Coding (SHVC) by replacing the inter-layer upscaler with the VSR upscaler, and benchmark the codec performance. The discrete cosine transform upsampling filter in SHVC is applied to the base layer reconstructed video, there-fore the reference video for the enhancement layer has scaling artifacts and compression artifacts. The VSR model used to perform upsampling can provide a higher quality reference for the EL.
However, the traditional VSR model can’t be directly used on the SHVC as the low-resolution video used as input for the VSR upscaler is pristine but in SHVC the LR input to the upscaler is compressed by the base layer codec. High-frequency details are lost during the compression and artifacts are introduced by the block-based hybrid video coding framework. Therefore the video super-resolution models need to be modified as the compression process added artifacts to the input. In this work deartifact network (DANet) was introduced to perform the artifacts reduction and super-resolution at the same time. DANet is based on FRVSR which estimates the optical flow between frames and uses motion compensation to align local frames for the super-resolution.
Our DANet performs 0.28dB and 0.81 VMAF better than the FRVSR on the PRIME7 test set with CRF23 LRC video as input.
After integrating DANet to the SHVC codec, on the PRIMT7 test set, using PSNR as the metric, this VSR-integrated scalable video coding framework achieved -5.62% BD-rate reduction at the same video quality and 0.17 dB BD-PSNR quality improvement at the same bitrates compared with the original SHVC. Using VMAF as the metric, our VSR-SHVC achieved -10.01% BD-rate reduction and 0.79 BD-VMAF quality improvement
