Learning Moore-Penrose based residuals for robust non-blind image deconvolution

This work was supported by grants P20_00286 and B-TIC-324-UGR20 funded by Consejería de Universidad, Investigación e Innovación ( Junta de Andalucía ) and by “ ERDF A way of making Europe”. Funding for open access charge: Universidad de Granada / CBUA.This paper proposes a deep learning-based method for image restoration given an inaccurate knowledge of the degradation. We first show how the impulse response of a Wiener filter can approximate the Moore-Penrose pseudo-inverse of the blur convolution operator. The deconvolution problem is then cast as the learning of a residual in the null space of the blur kernel, which, when added to the Wiener restoration, will satisfy the image formation model. This approach is expected to make the network capable of dealing with different blurs since only residuals associated with the Wiener filter have to be learned. Artifacts caused by inaccuracies in the blur estimation and other image formation model inconsistencies are removed by a Dynamic Filter Network. The extensive experiments carried out on several synthetic and real image datasets assert the proposed method's performance and robustness and demonstrate the advantage of the proposed method over existing ones.Junta de Andalucía P20_00286, B-TIC-324-UGR20ERDF A way of making EuropeUniversidad de Granada / CBU

심층 신경망을 활용한 영상 품질 강화 기법

학위논문(박사) -- 서울대학교대학원 : 자연과학대학 협동과정 계산과학전공, 2021.8. 노형민.In this thesis, we focus on deep learning methods to enhance the quality of a single image. We first categorize the image quality enhancement problem into three tasks: denoising, deblurring, and super-resolution, then introduce deep learning techniques optimized for each problem. To solve these problems, we introduce a novel deep neural network suitable for multi-scale analysis and propose efficient model-agnostic methods that help the network extract information from high-frequency domains to reconstruct clearer images. Experiments on SIDD, Flickr2K, DIV2K, and REDS datasets show that our method achieves state-of-the-art performance on each task. Furthermore, we show that our model can overcome the over-smoothing problem commonly observed in existing PSNR-oriented methods and generate more natural high-resolution images by applying adversarial training.본 학위 논문은 단일 영상의 품질 강화를 위한 딥러닝 기법에 대한 연구를 다룬다. 영상 품질 강화를 손상된 이미지의 잡음 제거 및 디블러링과 저해상도 이미지를 고해상도로 변환하는 초해상도 문제로 세분화한 뒤, 각각의 문제 해결에 최적화된 딥러닝 기법을 단계별로 소개한다. 특히, 손상된 영상의 특성을 효과적으로 분석하고 보다 깔끔한 고해상도 영상을 생성하기 위하여 주어진 영상을 다중 스케일로 분석하는 심층 신경망 구조를 제안하였으며, 이외에도 딥러닝 모델이 영상 내 복잡한 고주파수 영역에 대한 정보를 효과적으로 추출하고 재건할 수 있도록 돕는 기법들을 소개한다. 우리는 제안된 기법들을 SIDD, Flickr2K, DIV2K, REDS 등 데이터셋에 적용하여 기존의 딥러닝 기반 기법보다 향상된 성능을 실험적으로 증명하였다. 또한 초해상도 문제 해결을 위해 학습된 심층 신경망에 추가적인 적대적 학습을 적용함으로써 기존 딥러닝 기법들의 한계로 지적되었던 부분 평균화 문제를 극복하고 보다 자연스러운 고해상도 영상을 생성할 수 있음을 보였다.1. Introduction 1 2. Preliminaries 4 2.1 Image Denoising 4 2.1.1 Problem Formulation: AWGN 4 2.1.2 Existing Methods 6 2.2 Image Deblurring 7 2.2.1 Problem Formulation: Blind Deblur 7 2.2.2 Existing Methods 7 2.3 Single Image Super-Resolution 9 2.3.1 Problem Formulation: SISR 9 2.3.2 Existing Methods 12 3. Image Denoising 15 3.1 Proposed Methods 15 3.1.1 Multi-scale Edge Filtering 15 3.1.2 Feature Attention Module 17 3.1.3 Network Architecture 19 3.2 Experiments 21 3.2.1 Training Details 21 3.2.2 Experimental Results on DIV2K+AWGN dataset 21 3.2.3 Experimental Results on SIDD dataset 26 4. Image Deblurring 28 4.1 Proposed Methods 28 4.1.1 Multi-Scale Feature Analysis 29 4.1.2 Network Architecture 29 4.2 Experiments 31 4.2.1 Training Details 31 4.2.2 Experimental Results on Flickr2K dataset 31 4.2.3 Experimental Results on REDS dataset 34 5. Single Image Super-Resolution 38 5.1 Proposed Methods 38 5.1.1 High-Pass Filtering Loss 39 5.1.2 Gradient Magnitude Similarity Map Masking 41 5.1.3 Soft Gradient Magnitude Similarity Map Masking 43 5.1.4 Network Architecture 44 5.1.5 Adversarial Training for Perceptual Generative Model 45 5.2 Experiments 47 5.2.1 Training Details 47 5.2.2 Experimental Results on DIV2K dataset 48 5.2.3 Experimental Results on Set5/Set14 dataset 55 5.2.4 Experimental Results on REDS dataset 60 6. Conclusion and Future Works 63박

Selected Topics in Bayesian Image/Video Processing

In this dissertation, three problems in image deblurring, inpainting and virtual content insertion are solved in a Bayesian framework.;Camera shake, motion or defocus during exposure leads to image blur. Single image deblurring has achieved remarkable results by solving a MAP problem, but there is no perfect solution due to inaccurate image prior and estimator. In the first part, a new non-blind deconvolution algorithm is proposed. The image prior is represented by a Gaussian Scale Mixture(GSM) model, which is estimated from non-blurry images as training data. Our experimental results on a total twelve natural images have shown that more details are restored than previous deblurring algorithms.;In augmented reality, it is a challenging problem to insert virtual content in video streams by blending it with spatial and temporal information. A generic virtual content insertion (VCI) system is introduced in the second part. To the best of my knowledge, it is the first successful system to insert content on the building facades from street view video streams. Without knowing camera positions, the geometry model of a building facade is established by using a detection and tracking combined strategy. Moreover, motion stabilization, dynamic registration and color harmonization contribute to the excellent augmented performance in this automatic VCI system.;Coding efficiency is an important objective in video coding. In recent years, video coding standards have been developing by adding new tools. However, it costs numerous modifications in the complex coding systems. Therefore, it is desirable to consider alternative standard-compliant approaches without modifying the codec structures. In the third part, an exemplar-based data pruning video compression scheme for intra frame is introduced. Data pruning is used as a pre-processing tool to remove part of video data before they are encoded. At the decoder, missing data is reconstructed by a sparse linear combination of similar patches. The novelty is to create a patch library to exploit similarity of patches. The scheme achieves an average 4% bit rate reduction on some high definition videos