Recent Progress in Image Deblurring

This paper comprehensively reviews the recent development of image deblurring, including non-blind/blind, spatially invariant/variant deblurring techniques. Indeed, these techniques share the same objective of inferring a latent sharp image from one or several corresponding blurry images, while the blind deblurring techniques are also required to derive an accurate blur kernel. Considering the critical role of image restoration in modern imaging systems to provide high-quality images under complex environments such as motion, undesirable lighting conditions, and imperfect system components, image deblurring has attracted growing attention in recent years. From the viewpoint of how to handle the ill-posedness which is a crucial issue in deblurring tasks, existing methods can be grouped into five categories: Bayesian inference framework, variational methods, sparse representation-based methods, homography-based modeling, and region-based methods. In spite of achieving a certain level of development, image deblurring, especially the blind case, is limited in its success by complex application conditions which make the blur kernel hard to obtain and be spatially variant. We provide a holistic understanding and deep insight into image deblurring in this review. An analysis of the empirical evidence for representative methods, practical issues, as well as a discussion of promising future directions are also presented.Comment: 53 pages, 17 figure

A deep learning framework for quality assessment and restoration in video endoscopy

Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Artifacts such as motion blur, bubbles, specular reflections, floating objects and pixel saturation impede the visual interpretation and the automated analysis of endoscopy videos. Given the widespread use of endoscopy in different clinical applications, we contend that the robust and reliable identification of such artifacts and the automated restoration of corrupted video frames is a fundamental medical imaging problem. Existing state-of-the-art methods only deal with the detection and restoration of selected artifacts. However, typically endoscopy videos contain numerous artifacts which motivates to establish a comprehensive solution. We propose a fully automatic framework that can: 1) detect and classify six different primary artifacts, 2) provide a quality score for each frame and 3) restore mildly corrupted frames. To detect different artifacts our framework exploits fast multi-scale, single stage convolutional neural network detector. We introduce a quality metric to assess frame quality and predict image restoration success. Generative adversarial networks with carefully chosen regularization are finally used to restore corrupted frames. Our detector yields the highest mean average precision (mAP at 5% threshold) of 49.0 and the lowest computational time of 88 ms allowing for accurate real-time processing. Our restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos we show that our approach preserves an average of 68.7% which is 25% more frames than that retained from the raw videos.Comment: 14 page

Recent Progress in Image Deblurring

This paper comprehensively reviews the recent development of image deblurring, including non-blind/blind, spatially invariant/variant deblurring techniques. Indeed, these techniques share the same objective of inferring a latent sharp image from one or several corresponding blurry images, while the blind deblurring techniques are also required to derive an accurate blur kernel. Considering the critical role of image restoration in modern imaging systems to provide high-quality images under complex environments such as motion, undesirable lighting conditions, and imperfect system components, image deblurring has attracted growing attention in recent years. From the viewpoint of how to handle the ill-posedness which is a crucial issue in deblurring tasks, existing methods can be grouped into five categories: Bayesian inference framework, variational methods, sparse representation-based methods, homography-based modeling, and region-based methods. In spite of achieving a certain level of development, image deblurring, especially the blind case, is limited in its success by complex application conditions which make the blur kernel hard to obtain and be spatially variant. We provide a holistic understanding and deep insight into image deblurring in this review. An analysis of the empirical evidence for representative methods, practical issues, as well as a discussion of promising future directions are also presented

심층 신경망을 활용한 영상 품질 강화 기법

학위논문(박사) -- 서울대학교대학원 : 자연과학대학 협동과정 계산과학전공, 2021.8. 노형민.In this thesis, we focus on deep learning methods to enhance the quality of a single image. We first categorize the image quality enhancement problem into three tasks: denoising, deblurring, and super-resolution, then introduce deep learning techniques optimized for each problem. To solve these problems, we introduce a novel deep neural network suitable for multi-scale analysis and propose efficient model-agnostic methods that help the network extract information from high-frequency domains to reconstruct clearer images. Experiments on SIDD, Flickr2K, DIV2K, and REDS datasets show that our method achieves state-of-the-art performance on each task. Furthermore, we show that our model can overcome the over-smoothing problem commonly observed in existing PSNR-oriented methods and generate more natural high-resolution images by applying adversarial training.본 학위 논문은 단일 영상의 품질 강화를 위한 딥러닝 기법에 대한 연구를 다룬다. 영상 품질 강화를 손상된 이미지의 잡음 제거 및 디블러링과 저해상도 이미지를 고해상도로 변환하는 초해상도 문제로 세분화한 뒤, 각각의 문제 해결에 최적화된 딥러닝 기법을 단계별로 소개한다. 특히, 손상된 영상의 특성을 효과적으로 분석하고 보다 깔끔한 고해상도 영상을 생성하기 위하여 주어진 영상을 다중 스케일로 분석하는 심층 신경망 구조를 제안하였으며, 이외에도 딥러닝 모델이 영상 내 복잡한 고주파수 영역에 대한 정보를 효과적으로 추출하고 재건할 수 있도록 돕는 기법들을 소개한다. 우리는 제안된 기법들을 SIDD, Flickr2K, DIV2K, REDS 등 데이터셋에 적용하여 기존의 딥러닝 기반 기법보다 향상된 성능을 실험적으로 증명하였다. 또한 초해상도 문제 해결을 위해 학습된 심층 신경망에 추가적인 적대적 학습을 적용함으로써 기존 딥러닝 기법들의 한계로 지적되었던 부분 평균화 문제를 극복하고 보다 자연스러운 고해상도 영상을 생성할 수 있음을 보였다.1. Introduction 1 2. Preliminaries 4 2.1 Image Denoising 4 2.1.1 Problem Formulation: AWGN 4 2.1.2 Existing Methods 6 2.2 Image Deblurring 7 2.2.1 Problem Formulation: Blind Deblur 7 2.2.2 Existing Methods 7 2.3 Single Image Super-Resolution 9 2.3.1 Problem Formulation: SISR 9 2.3.2 Existing Methods 12 3. Image Denoising 15 3.1 Proposed Methods 15 3.1.1 Multi-scale Edge Filtering 15 3.1.2 Feature Attention Module 17 3.1.3 Network Architecture 19 3.2 Experiments 21 3.2.1 Training Details 21 3.2.2 Experimental Results on DIV2K+AWGN dataset 21 3.2.3 Experimental Results on SIDD dataset 26 4. Image Deblurring 28 4.1 Proposed Methods 28 4.1.1 Multi-Scale Feature Analysis 29 4.1.2 Network Architecture 29 4.2 Experiments 31 4.2.1 Training Details 31 4.2.2 Experimental Results on Flickr2K dataset 31 4.2.3 Experimental Results on REDS dataset 34 5. Single Image Super-Resolution 38 5.1 Proposed Methods 38 5.1.1 High-Pass Filtering Loss 39 5.1.2 Gradient Magnitude Similarity Map Masking 41 5.1.3 Soft Gradient Magnitude Similarity Map Masking 43 5.1.4 Network Architecture 44 5.1.5 Adversarial Training for Perceptual Generative Model 45 5.2 Experiments 47 5.2.1 Training Details 47 5.2.2 Experimental Results on DIV2K dataset 48 5.2.3 Experimental Results on Set5/Set14 dataset 55 5.2.4 Experimental Results on REDS dataset 60 6. Conclusion and Future Works 63박

Image Restoration

This book represents a sample of recent contributions of researchers all around the world in the field of image restoration. The book consists of 15 chapters organized in three main sections (Theory, Applications, Interdisciplinarity). Topics cover some different aspects of the theory of image restoration, but this book is also an occasion to highlight some new topics of research related to the emergence of some original imaging devices. From this arise some real challenging problems related to image reconstruction/restoration that open the way to some new fundamental scientific questions closely related with the world we interact with

Unsupervised Image Restoration Using Partially Linear Denoisers.

Deep neural network based methods are the state of the art in various image restoration problems. Standard supervised learning frameworks require a set of noisy measurement and clean image pairs for which a distance between the output of the restoration model and the ground truth, clean images is minimized. The ground truth images, however, are often unavailable or very expensive to acquire in real-world applications. We circumvent this problem by proposing a class of structured denoisers that can be decomposed as the sum of a nonlinear image-dependent mapping, a linear noise-dependent term and a small residual term. We show that these denoisers can be trained with only noisy images under the condition that the noise has zero mean and known variance. The exact distribution of the noise, however, is not assumed to be known. We show the superiority of our approach for image denoising, and demonstrate its extension to solving other restoration problems such as image deblurring where the ground truth is not available. Our method outperforms some recent unsupervised and self-supervised deep denoising models that do not require clean images for their training. For deblurring problems, the method, using only one noisy and blurry observation per image, reaches a quality not far away from its fully supervised counterparts on a benchmark dataset

Variational Deep Image Restoration

This paper presents a new variational inference framework for image restoration and a convolutional neural network (CNN) structure that can solve the restoration problems described by the proposed framework. Earlier CNN-based image restoration methods primarily focused on network architecture design or training strategy with non-blind scenarios where the degradation models are known or assumed. For a step closer to real-world applications, CNNs are also blindly trained with the whole dataset, including diverse degradations. However, the conditional distribution of a high-quality image given a diversely degraded one is too complicated to be learned by a single CNN. Therefore, there have also been some methods that provide additional prior information to train a CNN. Unlike previous approaches, we focus more on the objective of restoration based on the Bayesian perspective and how to reformulate the objective. Specifically, our method relaxes the original posterior inference problem to better manageable sub-problems and thus behaves like a divide-and-conquer scheme. As a result, the proposed framework boosts the performance of several restoration problems compared to the previous ones. Specifically, our method delivers state-of-the-art performance on Gaussian denoising, real-world noise reduction, blind image super-resolution, and JPEG compression artifacts reduction.Comment: IEEE Transactions on Image Processing (TIP 2022