Recent Progress in Image Deblurring

This paper comprehensively reviews the recent development of image deblurring, including non-blind/blind, spatially invariant/variant deblurring techniques. Indeed, these techniques share the same objective of inferring a latent sharp image from one or several corresponding blurry images, while the blind deblurring techniques are also required to derive an accurate blur kernel. Considering the critical role of image restoration in modern imaging systems to provide high-quality images under complex environments such as motion, undesirable lighting conditions, and imperfect system components, image deblurring has attracted growing attention in recent years. From the viewpoint of how to handle the ill-posedness which is a crucial issue in deblurring tasks, existing methods can be grouped into five categories: Bayesian inference framework, variational methods, sparse representation-based methods, homography-based modeling, and region-based methods. In spite of achieving a certain level of development, image deblurring, especially the blind case, is limited in its success by complex application conditions which make the blur kernel hard to obtain and be spatially variant. We provide a holistic understanding and deep insight into image deblurring in this review. An analysis of the empirical evidence for representative methods, practical issues, as well as a discussion of promising future directions are also presented.Comment: 53 pages, 17 figure

Focusing on out-of-focus : assessing defocus estimation algorithms for the benefit of automated image masking

Acquiring photographs as input for an image-based modelling pipeline is less trivial than often assumed. Photographs should be correctly exposed, cover the subject sufficiently from all possible angles, have the required spatial resolution, be devoid of any motion blur, exhibit accurate focus and feature an adequate depth of field. The last four characteristics all determine the " sharpness " of an image and the photogrammetric, computer vision and hybrid photogrammetric computer vision communities all assume that the object to be modelled is depicted " acceptably " sharp throughout the whole image collection. Although none of these three fields has ever properly quantified " acceptably sharp " , it is more or less standard practice to mask those image portions that appear to be unsharp due to the limited depth of field around the plane of focus (whether this means blurry object parts or completely out-of-focus backgrounds). This paper will assess how well-or ill-suited defocus estimating algorithms are for automatically masking a series of photographs, since this could speed up modelling pipelines with many hundreds or thousands of photographs. To that end, the paper uses five different real-world datasets and compares the output of three state-of-the-art edge-based defocus estimators. Afterwards, critical comments and plans for the future finalise this paper

DMTNet: Dynamic Multi-scale Network for Dual-pixel Images Defocus Deblurring with Transformer

Recent works achieve excellent results in defocus deblurring task based on dual-pixel data using convolutional neural network (CNN), while the scarcity of data limits the exploration and attempt of vision transformer in this task. In addition, the existing works use fixed parameters and network architecture to deblur images with different distribution and content information, which also affects the generalization ability of the model. In this paper, we propose a dynamic multi-scale network, named DMTNet, for dual-pixel images defocus deblurring. DMTNet mainly contains two modules: feature extraction module and reconstruction module. The feature extraction module is composed of several vision transformer blocks, which uses its powerful feature extraction capability to obtain richer features and improve the robustness of the model. The reconstruction module is composed of several Dynamic Multi-scale Sub-reconstruction Module (DMSSRM). DMSSRM can restore images by adaptively assigning weights to features from different scales according to the blur distribution and content information of the input images. DMTNet combines the advantages of transformer and CNN, in which the vision transformer improves the performance ceiling of CNN, and the inductive bias of CNN enables transformer to extract more robust features without relying on a large amount of data. DMTNet might be the first attempt to use vision transformer to restore the blurring images to clarity. By combining with CNN, the vision transformer may achieve better performance on small datasets. Experimental results on the popular benchmarks demonstrate that our DMTNet significantly outperforms state-of-the-art methods

LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network

Recovering sharp images from dual-pixel (DP) pairs with disparity-dependent blur is a challenging task.~Existing blur map-based deblurring methods have demonstrated promising results. In this paper, we propose, to the best of our knowledge, the first framework to introduce the contrastive language-image pre-training framework (CLIP) to achieve accurate blur map estimation from DP pairs unsupervisedly. To this end, we first carefully design text prompts to enable CLIP to understand blur-related geometric prior knowledge from the DP pair. Then, we propose a format to input stereo DP pair to the CLIP without any fine-tuning, where the CLIP is pre-trained on monocular images. Given the estimated blur map, we introduce a blur-prior attention block, a blur-weighting loss and a blur-aware loss to recover the all-in-focus image. Our method achieves state-of-the-art performance in extensive experiments

Learnable Blur Kernel for Single-Image Defocus Deblurring in the Wild

Recent research showed that the dual-pixel sensor has made great progress in defocus map estimation and image defocus deblurring. However, extracting real-time dual-pixel views is troublesome and complex in algorithm deployment. Moreover, the deblurred image generated by the defocus deblurring network lacks high-frequency details, which is unsatisfactory in human perception. To overcome this issue, we propose a novel defocus deblurring method that uses the guidance of the defocus map to implement image deblurring. The proposed method consists of a learnable blur kernel to estimate the defocus map, which is an unsupervised method, and a single-image defocus deblurring generative adversarial network (DefocusGAN) for the first time. The proposed network can learn the deblurring of different regions and recover realistic details. We propose a defocus adversarial loss to guide this training process. Competitive experimental results confirm that with a learnable blur kernel, the generated defocus map can achieve results comparable to supervised methods. In the single-image defocus deblurring task, the proposed method achieves state-of-the-art results, especially significant improvements in perceptual quality, where PSNR reaches 25.56 dB and LPIPS reaches 0.111.Comment: 9 pages, 7 figure

듀얼 픽셀 이미지 기반 제로샷 디포커스 디블러링

학위논문(석사) -- 서울대학교대학원 : 공과대학 협동과정 인공지능전공, 2022. 8. 한보형.Defocus deblurring in dual-pixel (DP) images is a challenging problem due to diverse camera optics and scene structures. Most of the existing algorithms rely on supervised learning approaches trained on the Canon DSLR dataset but often suffer from weak generalizability to out-of-distribution images including the ones captured by smartphones. We propose a novel zero-shot defocus deblurring algorithm, which only requires a pair of DP images without any training data and a pre-calibrated ground-truth blur kernel. Specifically, our approach first initializes a sharp latent map using a parametric blur kernel with a symmetry constraint. It then uses a convolutional neural network (CNN) to estimate the defocus map that best describes the observed DP image. Finally, it employs a generative model to learn scene-specific non-uniform blur kernels to compute the final enhanced images. We demonstrate that the proposed unsupervised technique outperforms the counterparts based on supervised learning when training and testing run in different datasets. We also present that our model achieves competitive accuracy when tested on in-distribution data.듀얼 픽셀(DP) 이미지 센서를 사용하는 스마트폰에서의 Defocus Blur 현상은 다양한 카메라 광학 구조와 물체의 깊이 마다 다른 흐릿함 정도로 인해 원 영상 복원이 쉽지 않습니다. 기존 알고리즘들은 모두 Canon DSLR 데이터에서 훈련된 지도 학습 접근 방식에 의존하여 스마트폰으로 촬영된 사진에서는 잘 일반화가 되지 않습니다. 본 논문에서는 훈련 데이터와 사전 보정된 실제 Blur 커널 없이도, 한 쌍의 DP 사진만으로도 학습이 가능한 Zero-shot Defocus Deblurring 알고리즘을 제안합니다. 특히, 본 논문에서는 대칭적으로 모델링 된 Blur Kernel을 사용하여 초기 영상을 복원하며, 이후 CNN(Convolutional Neural Network)을 사용하여 관찰된 DP 이미지를 가장 잘 설명하는 Defocus Map을 추정합니다. 마지막으로 CNN을 사용하여 장면 별 Non-uniform한 Blur Kernel을 학습하여 최종 복원 영상의 성능을 개선합니다. 학습과 추론이 다른 데이터 세트에서 실행될 때, 제안된 방법은 비지도 기술 임에도 불구하고 최근에 발표된 지도 학습을 기반의 방법들보다 우수한 성능을 보여줍니다. 또한 학습 된 것과 같은 분포 내 데이터에서 추론할 때도 지도 학습 기반의 방법들과 정량적 또는 정성적으로 비슷한 성능을 보이는 것을 확인할 수 있었습니다.1. Introduction 6 1.1. Background 6 1.2. Overview 9 1.3. Contribution 11 2. Related Works 12 2.1.Defocus Deblurring 12 2.2.Defocus Map 13 2.3.Multiplane Image Representation 14 2.4.DP Blur Kernel 14 3. Proposed Methods 16 3.1. Latent Map Initialization 17 3.2. Defocus Map Estimation 20 3.3. Learning Blur Kernel s 22 3.4. Implementation Details 25 4. Experiments 28 4.1. Dataset 28 4.2. Quantitative Results 29 4.3. Qualitative Results 31 5. Conclusions 37 5.1.Summary 37 5.2. Discussion 38석