Bringing Blurry Images Alive: High-Quality Image Restoration and Video Reconstruction

Consumer-level cameras are affordable for customers. While handy and easy to use, images and videos are likely to suffer from motion blur effect, especially under low-lighting conditions. Moreover, it is rather difficult to take high frame-rate videos due to the hardware limitations of conventional RGB-sensors. Therefore, our thesis mainly focuses on restoring high-quality (sharp, and high frame-rate) images and videos, from the low-quality (blur, and low frame-rate) ones for better practical applications. In this thesis, we mainly address the problem of how to restore a sharp image from a blurred stereo video sequence, a blurred RGB-D image, or a single blurred image. Then, by utilizing the faithful information about the motion provided by blurry effects in the image, we reconstruct high frame-rate and sharp videos based on an event camera, that brings blurry frame alive. Stereo camera systems can provide motion information incorporated to help to remove complex spatially-varying motion blur in dynamic scenes. Given consecutive blurred stereo video frames, we recover the latent images, estimate the 3D scene flow, and segment the multiple moving objects simultaneously. We represent the dynamic scenes with the piece-wise planar model, which exploits the local structure of the scene and expresses various dynamic scenes. These three tasks are naturally connected under our model and expressed as the parameter estimation of 3D scene structure and camera motion (structure and motion for the dynamic scenes). To tackle the challenging, minimal image deblurring case, namely, single-image deblurring, we first focus on blur caused by camera shake during the exposure time. We propose to jointly estimate the 6 DoF camera motion and remove the non-uniform blur by exploiting their underlying geometric relationships, with a single blurred RGB-D image as input. We formulate our joint deblurring and 6 DoF camera motion estimation as an energy minimization problem solved in an alternative manner. In general cases, we solve the single-image deblurring task by studying the problem in the frequency domain. We show that the auto-correlation of the absolute phase-only image (phase-only image means the image is reconstructed only from the phase information of the blurry image) can provide faithful information about the motion (e.g., the motion direction and magnitude) that caused the blur, leading to a new and efficient blur kernel estimation approach. Event cameras are gaining attention for they measure intensity changes (called `events') with microsecond accuracy. The event camera allows the simultaneous output of the intensity frames. However, the images are captured at a relatively low frame-rate and often suffer from motion blur. A blurred image can be regarded as the integral of a sequence of latent images, while the events indicate the changes between the latent images. Therefore, we model the blur-generation process by associating event data to a latent image. We propose a simple and effective approach, the EDI model, to reconstruct a high frame-rate, sharp video (>1000 fps) from a single blurry frame and its event data. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Then, we improved the EDI model by using multiple images and their events to handle flickering effects and noise in the generated video. Also, we provide a more efficient solver to minimize the proposed energy model. Last, the blurred image and events also contribute to optical flow estimation. We propose a single image and events based optical flow estimation approach to unlock their potential applications. In summary, this thesis addresses how to recover sharp images from blurred ones and reconstruct a high temporal resolution video from a single image and event. Our extensive experimental results demonstrate our proposed methods outperform the state-of-the-art

Blur perception: An evaluation of focus measures

Since the middle of the 20th century the technological development of conventional photographic cameras has taken advantage of the advances in electronics and signal processing. One speci c area that has bene ted from these developments is that of auto-focus, the ability for a cameras optical arrangement to be altered so as to ensure the subject of the scene is in focus. However, whilst the precise focus point can be known for a single point in a scene, the method for selecting a best focus for the entire scene is an unsolved problem. Many focus algorithms have been proposed and compared, though no overall comparison between all algorithms has been made, nor have the results been compared with human observers. This work describes a methodology that was developed to benchmark focus algorithms against human results. Experiments that capture quantitative metrics about human observers were developed and conducted with a large set of observers on a diverse range of equipment. From these experiments, it was found that humans were highly consensual in their experimental responses. The human results were then used as a benchmark, against which equivalent experiments were performed by each of the candidate focus algorithms. A second set of experiments, conducted in a controlled environment, captured the underlying human psychophysical blur discrimination thresholds in natural scenes. The resultant thresholds were then characterised and compared against equivalent discrimination thresholds obtained by using the candidate focus algorithms as automated observers. The results of this comparison and how this should guide the selection of an auto-focus algorithm are discussed, with comment being passed on how focus algorithms may need to change to cope with future imaging techniques

Brachytherapy Seed and Applicator Localization via Iterative Forward Projection Matching Algorithm using Digital X-ray Projections

Interstitial and intracavitary brachytherapy plays an essential role in management of several malignancies. However, the achievable accuracy of brachytherapy treatment for prostate and cervical cancer is limited due to the lack of intraoperative planning and adaptive replanning. A major problem in implementing TRUS-based intraoperative planning is an inability of TRUS to accurately localize individual seed poses (positions and orientations) relative to the prostate volume during or after the implantation. For the locally advanced cervical cancer patient, manual drawing of the source positions on orthogonal films can not localize the full 3D intracavitary brachytherapy (ICB) applicator geometry. A new iterative forward projection matching (IFPM) algorithm can explicitly localize each individual seed/applicator by iteratively matching computed projections of the post-implant patient with the measured projections. This thesis describes adaptation and implementation of a novel IFPM algorithm that addresses hitherto unsolved problems in localization of brachytherapy seeds and applicators. The prototype implementation of 3-parameter point-seed IFPM algorithm was experimentally validated using a set of a few cone-beam CT (CBCT) projections of both the phantom and post-implant patient’s datasets. Geometric uncertainty due to gantry angle inaccuracy was incorporated. After this, IFPM algorithm was extended to 5-parameter elongated line-seed model which automatically reconstructs individual seed orientation as well as position. The accuracy of this algorithm was tested using both the synthetic-measured projections of clinically-realistic Model-6711 125I seed arrangements and measured projections of an in-house precision-machined prostate implant phantom that allows the orientations and locations of up to 100 seeds to be set to known values. The seed reconstruction error for simulation was less than 0.6 mm/3o. For the physical phantom experiments, IFPM absolute accuracy for position, polar angle, and azimuthal angel were (0.78 ± 0.57) mm, (5.8 ± 4.8)o, and (6.8 ± 4.0)o, respectively. It avoids the need to match corresponding seeds in each projection and accommodates incomplete data, overlapping seed clusters, and highly-migrated seeds. IFPM was further generalized from 5-parameter to 6-parameter model which was needed to reconstruct 3D pose of arbitrary-shape applicators. The voxelized 3D model of the applicator was obtained from external complex combinatorial geometric modeling. It is then integrated into the forward projection matching method for computing the 2D projections of the 3D ICB applicators, iteratively. The applicator reconstruction error for simulation was about 0.5 mm/2o. The residual 2D registration error (positional difference) between computed and actual measured applicator images was less than 1 mm for the intrauterine tandem and about 1.5 mm for the bilateral colpostats in each detector plane. By localizing the applicator’s internal structure and the sources, the effect of intra and inter-applicator attenuation can be included in the resultant dose distribution and CBCT metal streaking artifact mitigation. The localization accuracy of better than 1 mm and 6o has the potential to support more accurate Monte Carlo-based or 2D TG-43 dose calculations in clinical practice. It is hoped the clinical implementation of IFPM approach to localize elongated line-seed/applicator for intraoperative brachytherapy planning may have a positive impact on the treatment of prostate and cervical cancers

커널에 의한 비근접 부분영상과 저차수 영상을 이용한 영상 선명화 기법

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2016. 2. 유석인.Blind image deblurring aims to restore a high-quality image from a blurry image. Blind image deblurring has gained considerable attention in recent years because it involves many challenges in problem formulation, regularization, and optimization. In optimization perspective, blind image deblurring is a severely ill-posed inverse problemtherefore, effective regularizations are required in order to obtain a high-quality latent image from a single blurred one. In this paper, we propose nonlocal regularizations to improve blind image deblurring. First, we propose to use the nonlocal patches selected by similarity weighted by the kernel for the next blur-kernel estimation. Using these kernel-guided nonlocal patches, we impose regularization that nonlocal patches would produce the similar values by convolution. Imposing this regularization improves the kernel estimation. Second, we propose to use a nonlocal low-rank image obtained from the composition of nonlocal similar patches. Using this nonlocal low-rank image, we impose regularization that the latent image is similar to this nonlocal low-rank image. A nonlocal low-rank image contains less noise by its intrinsic property. Imposing this regularization improves the estimation of the latent image with less noise. We evaluated our method quantitatively and qualitatively by comparing several conventional blind deblurring methods. For the quantitative evaluation, we computed the sum of squared error, peak signal-to-noise ratio, and structural similarity index. For blurry images without noise, our method was generally superior to the other methods. Especially, the results of ours were sharper on structures and smoother on flat regions. For blurry and noisy images, our method highly outperformed the conventional methods. Most of other methods could not successfully estimate the blur-kernel, and the image blur was not removed. On the other hand, our method successfully estimate the blur-kernel by overcoming the noise and restored a high-quality of deblurred image with less noise.Chapter 1 Introduction 1 1.1 Formulation of the Blind Image Deblurring 2 1.2 Approach 4 1.2.1 The Use of Kernel-guided Nonlocal Patches 4 1.2.2 The Use of Nonlocal Low-rank Images 5 1.3 Overview 5 Chapter 2 Related Works 6 2.1 Natural Image Prior 7 2.1.1 Scale Mixture of Gaussians 8 2.1.2 Hyper-Laplacian Distribution 8 2.2 Avoiding No-blur Solution 10 2.2.1 Marginalization over Possible Images 11 2.2.2 Normalization of l1 by l2 13 2.2.3 Alternating I and k Approach 15 2.3 Sparse Representation 17 2.4 Using Sharp Edges 19 2.5 Handling Noise 20 Chapter 3 Preliminary: Optimization 24 3.1 Iterative Reweighted Least Squares (IRLS) 25 3.1.1 Least Squared Error Approximation 26 3.1.2 Weighted Least Squared Error Approximation 26 3.1.3 The lp Norm Approximation of Overdetermined System 27 3.1.4 The lp Norm Approximation of Underdetermined System 28 3.2 Optimization using Conjugacy 29 3.2.1 The Conjugate Direction Method 30 3.2.2 The Conjugate Gradient Method 33 3.3 The Singular Value Thresholding Algorithm 36 Chapter 4 Extracting Salient Structures 39 4.1 Structure-Texture Decomposition with Uniform Edge Map 39 4.2 Structure-Texture Decomposition with Adaptive Edge Map 41 4.3 Enhancing Structures and Producing Salient Edges 43 4.4 Analysis on the Method of Extracting Salient Edges 44 Chapter 5 Blind Image Deblurring using Nonlocal Patches 46 5.1 Estimating a Blur-kernel using Kernel-guided Nonlocal Patches 47 5.1.1 Sparse Prior 48 5.1.2 Continuous Prior 48 5.1.3 Nonlocal Prior by Kernel-guided Nonlocal Patches 49 5.2 Estimating an Interim Image using Nonlocal Low-rank Images 52 5.2.1 Nonlocal Low-rank Prior 52 5.3 Multiscale Implementation 55 5.4 Latent Image Estimation 56 Chapter 6 Experimental Results 58 6.1 Images with Ground Truth 61 6.2 Images without Ground Truth 105 6.3 Analysis on Preprocessing using Denoising 111 6.4 Analysis on the Size of Nonlocal Patches 121 6.5 Time Performance 125 Chapter 7 Conclusion 126 Bibliography 129 요약 140Docto