Temporally Consistent Edge-Informed Video Super-Resolution (Edge-VSR)

Resolution enhancement of a given video sequence is known as video super-resolution. We propose an end-to-end trainable video super-resolution method as an extension of the recently developed edge-informed single image super-resolution algorithm. A two-stage adversarial-based convolutional neural network that incorporates temporal information along with the current frame's structural information will be used. The edge information in each frame along with optical flow technique for motion estimation among frames will be applied. Promising results on validation datasets will be presented

Super-Resolution Enhancement of Digital Video

SR from digital video is a relatively new field, in only its third decade of existence. There is no doubt that as imaging sensor technologies, optical fabrication techniques, and computational algorithms mature, SR will find its way into digital video products such as cameras and digital cable set-top boxes. These papers on the fundamental SR topics of image registration, regularization, photometric diversity, detector nonuniformity, compression, optical design, and performance metrics serve as pioneers in the dynamic and evolving field of SR image reconstruction research and development. We are proud to present them to the image and video processing research community. (Refers to papers appearing in the same issue of the EURASIP Journal on Advances in Signal Processing in which this editorial appeared.

Progressive Training of A Two-Stage Framework for Video Restoration

As a widely studied task, video restoration aims to enhance the quality of the videos with multiple potential degradations, such as noises, blurs and compression artifacts. Among video restorations, compressed video quality enhancement and video super-resolution are two of the main tacks with significant values in practical scenarios. Recently, recurrent neural networks and transformers attract increasing research interests in this field, due to their impressive capability in sequence-to-sequence modeling. However, the training of these models is not only costly but also relatively hard to converge, with gradient exploding and vanishing problems. To cope with these problems, we proposed a two-stage framework including a multi-frame recurrent network and a single-frame transformer. Besides, multiple training strategies, such as transfer learning and progressive training, are developed to shorten the training time and improve the model performance. Benefiting from the above technical contributions, our solution wins two champions and a runner-up in the NTIRE 2022 super-resolution and quality enhancement of compressed video challenges.Comment: Winning two championships and one runner-up in the NTIRE 2022 challenge of super-resolution and quality enhancement of compressed video; accepted to CVPRW 202

Real-Time Neural Video Recovery and Enhancement on Mobile Devices

As mobile devices become increasingly popular for video streaming, it's crucial to optimize the streaming experience for these devices. Although deep learning-based video enhancement techniques are gaining attention, most of them cannot support real-time enhancement on mobile devices. Additionally, many of these techniques are focused solely on super-resolution and cannot handle partial or complete loss or corruption of video frames, which is common on the Internet and wireless networks. To overcome these challenges, we present a novel approach in this paper. Our approach consists of (i) a novel video frame recovery scheme, (ii) a new super-resolution algorithm, and (iii) a receiver enhancement-aware video bit rate adaptation algorithm. We have implemented our approach on an iPhone 12, and it can support 30 frames per second (FPS). We have evaluated our approach in various networks such as WiFi, 3G, 4G, and 5G networks. Our evaluation shows that our approach enables real-time enhancement and results in a significant increase in video QoE (Quality of Experience) of 24\% - 82\% in our video streaming system

Super-resolution Enhancement of Video

We consider the problem of enhancing the resolution of video through the addition of perceptually plausible high frequency information

How Video Super-Resolution and Frame Interpolation Mutually Benefit

Video super-resolution (VSR) and video frame interpolation (VFI) are inter-dependent for enhancing videos of low resolution and low frame rate. However, most studies treat VSR and temporal VFI as independent tasks. In this work, we design a spatial-temporal super-resolution network based on exploring the interaction between VSR and VFI. The main idea is to improve the middle frame of VFI by the super-resolution (SR) frames and feature maps from VSR. In the meantime, VFI also provides extra information for VSR and thus, through interacting, the SR of consecutive frames of the original video can also be improved by the feedback from the generated middle frame. Drawing on this, our approach leverages a simple interaction of VSR and VFI and achieves state-of-the-art performance on various datasets. Due to such a simple strategy, our approach is universally applicable to any existing VSR or VFI networks for effectively improving their video enhancement performance

Towards Interpretable Video Super-Resolution via Alternating Optimization

In this paper, we study a practical space-time video super-resolution (STVSR) problem which aims at generating a high-framerate high-resolution sharp video from a low-framerate low-resolution blurry video. Such problem often occurs when recording a fast dynamic event with a low-framerate and low-resolution camera, and the captured video would suffer from three typical issues: i) motion blur occurs due to object/camera motions during exposure time; ii) motion aliasing is unavoidable when the event temporal frequency exceeds the Nyquist limit of temporal sampling; iii) high-frequency details are lost because of the low spatial sampling rate. These issues can be alleviated by a cascade of three separate sub-tasks, including video deblurring, frame interpolation, and super-resolution, which, however, would fail to capture the spatial and temporal correlations among video sequences. To address this, we propose an interpretable STVSR framework by leveraging both model-based and learning-based methods. Specifically, we formulate STVSR as a joint video deblurring, frame interpolation, and super-resolution problem, and solve it as two sub-problems in an alternate way. For the first sub-problem, we derive an interpretable analytical solution and use it as a Fourier data transform layer. Then, we propose a recurrent video enhancement layer for the second sub-problem to further recover high-frequency details. Extensive experiments demonstrate the superiority of our method in terms of quantitative metrics and visual quality.Comment: ECCV 202

Wavelet-based image and video super-resolution reconstruction.

Super-resolution reconstruction process offers the solution to overcome the high-cost and inherent resolution limitations of current imaging systems. The wavelet transform is a powerful tool for super-resolution reconstruction. This research provides a detailed study of the wavelet-based super-resolution reconstruction process, and wavelet-based resolution enhancement process (with which it is closely associated). It was addressed to handle an explicit need for a robust wavelet-based method that guarantees efficient utilisation of the SR reconstruction problem in the wavelet-domain, which will lead to a consistent solution of this problem and improved performance. This research proposes a novel performance assessment approach to improve the performance of the existing wavelet-based image resolution enhancement techniques. The novel approach is based on identifying the factors that effectively influence on the performance of these techniques, and designing a novel optimal factor analysis (OFA) algorithm. A new wavelet-based image resolution enhancement method, based on discrete wavelet transform and new-edge directed interpolation (DWT-NEDI), and an adaptive thresholding process, has been developed. The DWT-NEDI algorithm aims to correct the geometric errors and remove the noise for degraded satellite images. A robust wavelet-based video super-resolution technique, based on global motion is developed by combining the DWT-NEDI method, with super-resolution reconstruction methods, in order to increase the spatial-resolution and remove the noise and aliasing artefacts. A new video super-resolution framework is designed using an adaptive local motion decomposition and wavelet transform reconstruction (ALMD-WTR). This is to address the challenge of the super-resolution problem for the real-world video sequences containing complex local motions. The results show that OFA approach improves the performance of the selected wavelet-based methods. The DWT-NEDI algorithm outperforms the state-of-the art wavelet-based algorithms. The global motion-based algorithm has the best performance over the super-resolution techniques, namely Keren and structure-adaptive normalised convolution methods. ALMD-WTR framework surpass the state-of-the-art wavelet-based algorithm, namely local motion-based video super-resolution.PhD in Manufacturin

Super-resolution of 3-dimensional scenes

Super-resolution is an image enhancement method that increases the resolution of images and video. Previously this technique could only be applied to 2D scenes. The super-resolution algorithm developed in this thesis creates high-resolution views of 3-dimensional scenes, using low-resolution images captured from varying, unknown positions