8,436 research outputs found
Recent Progress in Image Deblurring
This paper comprehensively reviews the recent development of image
deblurring, including non-blind/blind, spatially invariant/variant deblurring
techniques. Indeed, these techniques share the same objective of inferring a
latent sharp image from one or several corresponding blurry images, while the
blind deblurring techniques are also required to derive an accurate blur
kernel. Considering the critical role of image restoration in modern imaging
systems to provide high-quality images under complex environments such as
motion, undesirable lighting conditions, and imperfect system components, image
deblurring has attracted growing attention in recent years. From the viewpoint
of how to handle the ill-posedness which is a crucial issue in deblurring
tasks, existing methods can be grouped into five categories: Bayesian inference
framework, variational methods, sparse representation-based methods,
homography-based modeling, and region-based methods. In spite of achieving a
certain level of development, image deblurring, especially the blind case, is
limited in its success by complex application conditions which make the blur
kernel hard to obtain and be spatially variant. We provide a holistic
understanding and deep insight into image deblurring in this review. An
analysis of the empirical evidence for representative methods, practical
issues, as well as a discussion of promising future directions are also
presented.Comment: 53 pages, 17 figure
Convolutional Deblurring for Natural Imaging
In this paper, we propose a novel design of image deblurring in the form of
one-shot convolution filtering that can directly convolve with naturally
blurred images for restoration. The problem of optical blurring is a common
disadvantage to many imaging applications that suffer from optical
imperfections. Despite numerous deconvolution methods that blindly estimate
blurring in either inclusive or exclusive forms, they are practically
challenging due to high computational cost and low image reconstruction
quality. Both conditions of high accuracy and high speed are prerequisites for
high-throughput imaging platforms in digital archiving. In such platforms,
deblurring is required after image acquisition before being stored, previewed,
or processed for high-level interpretation. Therefore, on-the-fly correction of
such images is important to avoid possible time delays, mitigate computational
expenses, and increase image perception quality. We bridge this gap by
synthesizing a deconvolution kernel as a linear combination of Finite Impulse
Response (FIR) even-derivative filters that can be directly convolved with
blurry input images to boost the frequency fall-off of the Point Spread
Function (PSF) associated with the optical blur. We employ a Gaussian low-pass
filter to decouple the image denoising problem for image edge deblurring.
Furthermore, we propose a blind approach to estimate the PSF statistics for two
Gaussian and Laplacian models that are common in many imaging pipelines.
Thorough experiments are designed to test and validate the efficiency of the
proposed method using 2054 naturally blurred images across six imaging
applications and seven state-of-the-art deconvolution methods.Comment: 15 pages, for publication in IEEE Transaction Image Processin
GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks
In the last decade, supervised deep learning approaches have been extensively
employed in visual odometry (VO) applications, which is not feasible in
environments where labelled data is not abundant. On the other hand,
unsupervised deep learning approaches for localization and mapping in unknown
environments from unlabelled data have received comparatively less attention in
VO research. In this study, we propose a generative unsupervised learning
framework that predicts 6-DoF pose camera motion and monocular depth map of the
scene from unlabelled RGB image sequences, using deep convolutional Generative
Adversarial Networks (GANs). We create a supervisory signal by warping view
sequences and assigning the re-projection minimization to the objective loss
function that is adopted in multi-view pose estimation and single-view depth
generation network. Detailed quantitative and qualitative evaluations of the
proposed framework on the KITTI and Cityscapes datasets show that the proposed
method outperforms both existing traditional and unsupervised deep VO methods
providing better results for both pose estimation and depth recovery.Comment: ICRA 2019 - accepte
Scene classification from degraded images: comparing human and computer vision performance
People can recognize the context of a scene with just a brief glance. Visual information such as color, objects and their properties, and texture are all important in correctly determining the type of scene (e.g. indoors versus outdoors). Although these properties are all useful, it is unclear which features of an image play a more important role in the task of scene recognition. To this aim, we compare and contrast a state-of-the-art neural network and GIST model with human performance on the task of classifying images as indoors or outdoors. We analyze the impact of image manipulations, such as blurring and scrambling, on computational models of scene recognition and human perception. We then create and analyze a measure of local-global information to represent how each perceptual system relies on local and global image features. Finally, we train a variety of neural networks on degraded images to attempt to build a neural network that emulates human performance on both classificaton accuracies and this local-global measure
New Datasets, Models, and Optimization
ํ์๋
ผ๋ฌธ(๋ฐ์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ๊ณต๊ณผ๋ํ ์ ๊ธฐยท์ ๋ณด๊ณตํ๋ถ, 2021.8. ์ํํ.์ฌ์ง ์ดฌ์์ ๊ถ๊ทน์ ์ธ ๋ชฉํ๋ ๊ณ ํ์ง์ ๊นจ๋ํ ์์์ ์ป๋ ๊ฒ์ด๋ค. ํ์ค์ ์ผ๋ก, ์ผ์์ ์ฌ์ง์ ์์ฃผ ํ๋ค๋ฆฐ ์นด๋ฉ๋ผ์ ์์ง์ด๋ ๋ฌผ์ฒด๊ฐ ์๋ ๋์ ํ๊ฒฝ์์ ์ฐ๋๋ค. ๋
ธ์ถ์๊ฐ ์ค์ ์นด๋ฉ๋ผ์ ํผ์ฌ์ฒด๊ฐ์ ์๋์ ์ธ ์์ง์์ ์ฌ์ง๊ณผ ๋์์์์ ๋ชจ์
๋ธ๋ฌ๋ฅผ ์ผ์ผํค๋ฉฐ ์๊ฐ์ ์ธ ํ์ง์ ์ ํ์ํจ๋ค. ๋์ ํ๊ฒฝ์์ ๋ธ๋ฌ์ ์ธ๊ธฐ์ ์์ง์์ ๋ชจ์์ ๋งค ์ด๋ฏธ์ง๋ง๋ค, ๊ทธ๋ฆฌ๊ณ ๋งค ํฝ์
๋ง๋ค ๋ค๋ฅด๋ค. ๊ตญ์ง์ ์ผ๋ก ๋ณํํ๋ ๋ธ๋ฌ์ ์ฑ์ง์ ์ฌ์ง๊ณผ ๋์์์์์ ๋ชจ์
๋ธ๋ฌ ์ ๊ฑฐ๋ฅผ ์ฌ๊ฐํ๊ฒ ํ๊ธฐ ์ด๋ ค์ฐ๋ฉฐ ํด๋ต์ด ํ๋๋ก ์ ํด์ง์ง ์์, ์ ์ ์๋์ง ์์ ๋ฌธ์ ๋ก ๋ง๋ ๋ค.
๋ฌผ๋ฆฌ์ ์ธ ์์ง์ ๋ชจ๋ธ๋ง์ ํตํด ํด์์ ์ธ ์ ๊ทผ๋ฒ์ ์ค๊ณํ๊ธฐ๋ณด๋ค๋ ๋จธ์ ๋ฌ๋ ๊ธฐ๋ฐ์ ์ ๊ทผ๋ฒ์ ์ด๋ฌํ ์ ์ ์๋์ง ์์ ๋ฌธ์ ๋ฅผ ํธ๋ ๋ณด๋ค ํ์ค์ ์ธ ๋ต์ด ๋ ์ ์๋ค. ํนํ ๋ฅ ๋ฌ๋์ ์ต๊ทผ ์ปดํจํฐ ๋น์ ํ๊ณ์์ ํ์ค์ ์ธ ๊ธฐ๋ฒ์ด ๋์ด ๊ฐ๊ณ ์๋ค. ๋ณธ ํ์๋
ผ๋ฌธ์ ์ฌ์ง ๋ฐ ๋น๋์ค ๋๋ธ๋ฌ๋ง ๋ฌธ์ ์ ๋ํด ๋ฅ ๋ฌ๋ ๊ธฐ๋ฐ ์๋ฃจ์
์ ๋์
ํ๋ฉฐ ์ฌ๋ฌ ํ์ค์ ์ธ ๋ฌธ์ ๋ฅผ ๋ค๊ฐ์ ์ผ๋ก ๋ค๋ฃฌ๋ค.
์ฒซ ๋ฒ์งธ๋ก, ๋๋ธ๋ฌ๋ง ๋ฌธ์ ๋ฅผ ๋ค๋ฃจ๊ธฐ ์ํ ๋ฐ์ดํฐ์
์ ์ทจ๋ํ๋ ์๋ก์ด ๋ฐฉ๋ฒ์ ์ ์ํ๋ค. ๋ชจ์
๋ธ๋ฌ๊ฐ ์๋ ์ด๋ฏธ์ง์ ๊นจ๋ํ ์ด๋ฏธ์ง๋ฅผ ์๊ฐ์ ์ผ๋ก ์ ๋ ฌ๋ ์ํ๋ก ๋์์ ์ทจ๋ํ๋ ๊ฒ์ ์ฌ์ด ์ผ์ด ์๋๋ค. ๋ฐ์ดํฐ๊ฐ ๋ถ์กฑํ ๊ฒฝ์ฐ ๋๋ธ๋ฌ๋ง ์๊ณ ๋ฆฌ์ฆ๋ค์ ํ๊ฐํ๋ ๊ฒ ๋ฟ๋ง ์๋๋ผ ์ง๋ํ์ต ๊ธฐ๋ฒ์ ๊ฐ๋ฐํ๋ ๊ฒ๋ ๋ถ๊ฐ๋ฅํด์ง๋ค. ๊ทธ๋ฌ๋ ๊ณ ์ ๋น๋์ค๋ฅผ ์ฌ์ฉํ์ฌ ์นด๋ฉ๋ผ ์์ ์ทจ๋ ํ์ดํ๋ผ์ธ์ ๋ชจ๋ฐฉํ๋ฉด ์ค์ ์ ์ธ ๋ชจ์
๋ธ๋ฌ ์ด๋ฏธ์ง๋ฅผ ํฉ์ฑํ๋ ๊ฒ์ด ๊ฐ๋ฅํ๋ค. ๊ธฐ์กด์ ๋ธ๋ฌ ํฉ์ฑ ๊ธฐ๋ฒ๋ค๊ณผ ๋ฌ๋ฆฌ ์ ์ํ๋ ๋ฐฉ๋ฒ์ ์ฌ๋ฌ ์์ง์ด๋ ํผ์ฌ์ฒด๋ค๊ณผ ๋ค์ํ ์์ ๊น์ด, ์์ง์ ๊ฒฝ๊ณ์์์ ๊ฐ๋ฆฌ์์ง ๋ฑ์ผ๋ก ์ธํ ์์ฐ์ค๋ฌ์ด ๊ตญ์์ ๋ธ๋ฌ์ ๋ณต์ก๋๋ฅผ ๋ฐ์ํ ์ ์๋ค.
๋ ๋ฒ์งธ๋ก, ์ ์๋ ๋ฐ์ดํฐ์
์ ๊ธฐ๋ฐํ์ฌ ์๋ก์ด ๋จ์ผ์์ ๋๋ธ๋ฌ๋ง์ ์ํ ๋ด๋ด ๋คํธ์ํฌ ๊ตฌ์กฐ๋ฅผ ์ ์ํ๋ค. ์ต์ ํ๊ธฐ๋ฒ ๊ธฐ๋ฐ ์ด๋ฏธ์ง ๋๋ธ๋ฌ๋ง ๋ฐฉ์์์ ๋๋ฆฌ ์ฐ์ด๊ณ ์๋ ์ ์ฐจ์ ๋ฏธ์ธํ ์ ๊ทผ๋ฒ์ ๋ฐ์ํ์ฌ ๋ค์ค๊ท๋ชจ ๋ด๋ด ๋คํธ์ํฌ๋ฅผ ์ค๊ณํ๋ค. ์ ์๋ ๋ค์ค๊ท๋ชจ ๋ชจ๋ธ์ ๋น์ทํ ๋ณต์ก๋๋ฅผ ๊ฐ์ง ๋จ์ผ๊ท๋ชจ ๋ชจ๋ธ๋ค๋ณด๋ค ๋์ ๋ณต์ ์ ํ๋๋ฅผ ๋ณด์ธ๋ค.
์ธ ๋ฒ์งธ๋ก, ๋น๋์ค ๋๋ธ๋ฌ๋ง์ ์ํ ์ํ ๋ด๋ด ๋คํธ์ํฌ ๋ชจ๋ธ ๊ตฌ์กฐ๋ฅผ ์ ์ํ๋ค. ๋๋ธ๋ฌ๋ง์ ํตํด ๊ณ ํ์ง์ ๋น๋์ค๋ฅผ ์ป๊ธฐ ์ํด์๋ ๊ฐ ํ๋ ์๊ฐ์ ์๊ฐ์ ์ธ ์ ๋ณด์ ํ๋ ์ ๋ด๋ถ์ ์ธ ์ ๋ณด๋ฅผ ๋ชจ๋ ์ฌ์ฉํด์ผ ํ๋ค. ์ ์ํ๋ ๋ด๋ถํ๋ ์ ๋ฐ๋ณต์ ์ฐ์ฐ๊ตฌ์กฐ๋ ๋ ์ ๋ณด๋ฅผ ํจ๊ณผ์ ์ผ๋ก ํจ๊ป ์ฌ์ฉํจ์ผ๋ก์จ ๋ชจ๋ธ ํ๋ผ๋ฏธํฐ ์๋ฅผ ์ฆ๊ฐ์ํค์ง ์๊ณ ๋ ๋๋ธ๋ฌ ์ ํ๋๋ฅผ ํฅ์์ํจ๋ค.
๋ง์ง๋ง์ผ๋ก, ์๋ก์ด ๋๋ธ๋ฌ๋ง ๋ชจ๋ธ๋ค์ ๋ณด๋ค ์ ์ต์ ํํ๊ธฐ ์ํด ๋ก์ค ํจ์๋ฅผ ์ ์ํ๋ค. ๊นจ๋ํ๊ณ ๋๋ ทํ ์ฌ์ง ํ ์ฅ์ผ๋ก๋ถํฐ ์์ฐ์ค๋ฌ์ด ๋ชจ์
๋ธ๋ฌ๋ฅผ ๋ง๋ค์ด๋ด๋ ๊ฒ์ ๋ธ๋ฌ๋ฅผ ์ ๊ฑฐํ๋ ๊ฒ๊ณผ ๋ง์ฐฌ๊ฐ์ง๋ก ์ด๋ ค์ด ๋ฌธ์ ์ด๋ค. ๊ทธ๋ฌ๋ ํต์ ์ฌ์ฉํ๋ ๋ก์ค ํจ์๋ก ์ป์ ๋๋ธ๋ฌ๋ง ๋ฐฉ๋ฒ๋ค์ ๋ธ๋ฌ๋ฅผ ์์ ํ ์ ๊ฑฐํ์ง ๋ชปํ๋ฉฐ ๋๋ธ๋ฌ๋ ์ด๋ฏธ์ง์ ๋จ์์๋ ๋ธ๋ฌ๋ก๋ถํฐ ์๋์ ๋ธ๋ฌ๋ฅผ ์ฌ๊ฑดํ ์ ์๋ค. ์ ์ํ๋ ๋ฆฌ๋ธ๋ฌ๋ง ๋ก์ค ํจ์๋ ๋๋ธ๋ฌ๋ง ์ํ์ ๋ชจ์
๋ธ๋ฌ๋ฅผ ๋ณด๋ค ์ ์ ๊ฑฐํ๋๋ก ์ค๊ณ๋์๋ค. ์ด์ ๋์๊ฐ ์ ์ํ ์๊ธฐ์ง๋ํ์ต ๊ณผ์ ์ผ๋ก๋ถํฐ ํ
์คํธ์ ๋ชจ๋ธ์ด ์๋ก์ด ๋ฐ์ดํฐ์ ์ ์ํ๋๋ก ํ ์ ์๋ค.
์ด๋ ๊ฒ ์ ์๋ ๋ฐ์ดํฐ์
, ๋ชจ๋ธ ๊ตฌ์กฐ, ๊ทธ๋ฆฌ๊ณ ๋ก์ค ํจ์๋ฅผ ํตํด ๋ฅ ๋ฌ๋์ ๊ธฐ๋ฐํ์ฌ ๋จ์ผ ์์ ๋ฐ ๋น๋์ค ๋๋ธ๋ฌ๋ง ๊ธฐ๋ฒ๋ค์ ์ ์ํ๋ค. ๊ด๋ฒ์ํ ์คํ ๊ฒฐ๊ณผ๋ก๋ถํฐ ์ ๋์ ๋ฐ ์ ์ฑ์ ์ผ๋ก ์ต์ฒจ๋จ ๋๋ธ๋ฌ๋ง ์ฑ๊ณผ๋ฅผ ์ฆ๋ช
ํ๋ค.Obtaining a high-quality clean image is the ultimate goal of photography. In practice, daily photography is often taken in dynamic environments with moving objects as well as shaken cameras. The relative motion between the camera and the objects during the exposure causes motion blur in images and videos, degrading the visual quality. The degree of blur strength and the shape of motion trajectory varies by every image and every pixel in dynamic environments. The locally-varying property makes the removal of motion blur in images and videos severely ill-posed.
Rather than designing analytic solutions with physical modelings, using machine learning-based approaches can serve as a practical solution for such a highly ill-posed problem. Especially, deep-learning has been the recent standard in computer vision literature. This dissertation introduces deep learning-based solutions for image and video deblurring by tackling practical issues in various aspects.
First, a new way of constructing the datasets for dynamic scene deblurring task is proposed. It is nontrivial to simultaneously obtain a pair of the blurry and the sharp image that are temporally aligned. The lack of data prevents the supervised learning techniques to be developed as well as the evaluation of deblurring algorithms. By mimicking the camera image pipeline with high-speed videos, realistic blurry images could be synthesized. In contrast to the previous blur synthesis methods, the proposed approach can reflect the natural complex local blur from and multiple moving objects, varying depth, and occlusion at motion boundaries.
Second, based on the proposed datasets, a novel neural network architecture for single-image deblurring task is presented. Adopting the coarse-to-fine approach that is widely used in energy optimization-based methods for image deblurring, a multi-scale neural network architecture is derived. Compared with the single-scale model with similar complexity, the multi-scale model exhibits higher accuracy and faster speed.
Third, a light-weight recurrent neural network model architecture for video deblurring is proposed. In order to obtain a high-quality video from deblurring, it is important to exploit the intrinsic information in the target frame as well as the temporal relation between the neighboring frames. Taking benefits from both sides, the proposed intra-frame iterative scheme applied to the RNNs achieves accuracy improvements without increasing the number of model parameters.
Lastly, a novel loss function is proposed to better optimize the deblurring models.
Estimating a dynamic blur for a clean and sharp image without given motion information is another ill-posed problem. While the goal of deblurring is to completely get rid of motion blur, conventional loss functions fail to train neural networks to fulfill the goal, leaving the trace of blur in the deblurred images. The proposed reblurring loss functions are designed to better eliminate the motion blur and to produce sharper images. Furthermore, the self-supervised learning process facilitates the adaptation of the deblurring model at test-time.
With the proposed datasets, model architectures, and the loss functions, the deep learning-based single-image and video deblurring methods are presented. Extensive experimental results demonstrate the state-of-the-art performance both quantitatively and qualitatively.1 Introduction 1
2 Generating Datasets for Dynamic Scene Deblurring 7
2.1 Introduction 7
2.2 GOPRO dataset 9
2.3 REDS dataset 11
2.4 Conclusion 18
3 Deep Multi-Scale Convolutional Neural Networks for Single Image Deblurring 19
3.1 Introduction 19
3.1.1 Related Works 21
3.1.2 Kernel-Free Learning for Dynamic Scene Deblurring 23
3.2 Proposed Method 23
3.2.1 Model Architecture 23
3.2.2 Training 26
3.3 Experiments 29
3.3.1 Comparison on GOPRO Dataset 29
3.3.2 Comparison on Kohler Dataset 33
3.3.3 Comparison on Lai et al. [54] dataset 33
3.3.4 Comparison on Real Dynamic Scenes 34
3.3.5 Effect of Adversarial Loss 34
3.4 Conclusion 41
4 Intra-Frame Iterative RNNs for Video Deblurring 43
4.1 Introduction 43
4.2 Related Works 46
4.3 Proposed Method 50
4.3.1 Recurrent Video Deblurring Networks 51
4.3.2 Intra-Frame Iteration Model 52
4.3.3 Regularization by Stochastic Training 56
4.4 Experiments 58
4.4.1 Datasets 58
4.4.2 Implementation details 59
4.4.3 Comparisons on GOPRO [72] dataset 59
4.4.4 Comparisons on [97] Dataset and Real Videos 60
4.5 Conclusion 61
5 Learning Loss Functions for Image Deblurring 67
5.1 Introduction 67
5.2 Related Works 71
5.3 Proposed Method 73
5.3.1 Clean Images are Hard to Reblur 73
5.3.2 Supervision from Reblurring Loss 75
5.3.3 Test-time Adaptation by Self-Supervision 76
5.4 Experiments 78
5.4.1 Effect of Reblurring Loss 78
5.4.2 Effect of Sharpness Preservation Loss 80
5.4.3 Comparison with Other Perceptual Losses 81
5.4.4 Effect of Test-time Adaptation 81
5.4.5 Comparison with State-of-The-Art Methods 82
5.4.6 Real World Image Deblurring 85
5.4.7 Combining Reblurring Loss with Other Perceptual Losses 86
5.4.8 Perception vs. Distortion Trade-Off 87
5.4.9 Visual Comparison of Loss Function 88
5.4.10 Implementation Details 89
5.4.11 Determining Reblurring Module Size 94
5.5 Conclusion 95
6 Conclusion 97
๊ตญ๋ฌธ ์ด๋ก 115
๊ฐ์ฌ์ ๊ธ 117๋ฐ
- โฆ