Motion Deblurring in the Wild

The task of image deblurring is a very ill-posed problem as both the image and the blur are unknown. Moreover, when pictures are taken in the wild, this task becomes even more challenging due to the blur varying spatially and the occlusions between the object. Due to the complexity of the general image model we propose a novel convolutional network architecture which directly generates the sharp image.This network is built in three stages, and exploits the benefits of pyramid schemes often used in blind deconvolution. One of the main difficulties in training such a network is to design a suitable dataset. While useful data can be obtained by synthetically blurring a collection of images, more realistic data must be collected in the wild. To obtain such data we use a high frame rate video camera and keep one frame as the sharp image and frame average as the corresponding blurred image. We show that this realistic dataset is key in achieving state-of-the-art performance and dealing with occlusions

Real-Time Under-Display Cameras Image Restoration and HDR on Mobile Devices

The new trend of full-screen devices implies positioning the camera behind the screen to bring a larger display-to-body ratio, enhance eye contact, and provide a notch-free viewing experience on smartphones, TV or tablets. On the other hand, the images captured by under-display cameras (UDCs) are degraded by the screen in front of them. Deep learning methods for image restoration can significantly reduce the degradation of captured images, providing satisfying results for the human eyes. However, most proposed solutions are unreliable or efficient enough to be used in real-time on mobile devices. In this paper, we aim to solve this image restoration problem using efficient deep learning methods capable of processing FHD images in real-time on commercial smartphones while providing high-quality results. We propose a lightweight model for blind UDC Image Restoration and HDR, and we also provide a benchmark comparing the performance and runtime of different methods on smartphones. Our models are competitive on UDC benchmarks while using x4 less operations than others. To the best of our knowledge, we are the first work to approach and analyze this real-world single image restoration problem from the efficiency and production point of view.Comment: ECCV 2022 AIM Worksho

Fast model of space-variant blurring and its application to deconvolution in astronomy

International audienceImage deblurring is essential to high resolution imaging and is therefore widely used in astronomy, microscopy or com- putational photography. While shift-invariant blur is modeled by convolution and leads to fast FFT-based algorithms, shift- variant blurring requires models both accurate and fast. When the point spread function (PSF) varies smoothly across the field, these two opposite objectives can be reached by inter- polating from a grid of PSF samples. Several models for smoothly varying PSF co-exist in the literature. We advocate that one of them is both physically- grounded and fast. Moreover, we show that the approximation can be largely improved by tuning the PSF samples and inter- polation weights with respect to a given continuous model. This improvement comes without increasing the computa- tional cost of the blurring operator. We illustrate the developed blurring model on a deconvo- lution application in astronomy. Regularized reconstruction with our model leads to large improvements over existing re- sults

Self-correcting multi-channel Bussgang blind deconvolution using expectation maximization (EM) algorithm and feedback

A Bussgang based blind deconvolution algorithm called self-correcting multi-channel Bussgang (SCMB) blind deconvolution algorithm was proposed. Unlike the original Bussgang blind deconvolution algorithm where the probability density function (pdf) of the signal being recovered is assumed to be completely known, the proposed SCMB blind deconvolution algorithm relaxes this restriction by parameterized the pdf with a Gaussian mixture model and expectation maximization (EM) algorithm, an iterative maximum likelihood approach, is employed to estimate the parameter side by side with the estimation of the equalization filters of the original Bussgang blind deconvolution algorithm. A feedback loop is also designed to compensate the effect of the parameter estimation error on the estimation of the equalization filters. Application of the SCMB blind deconvolution framework for binary image restoration, multi-pass synthetic aperture radar (SAR) autofocus and inverse synthetic aperture radar (ISAR) autofocus are exploited with great results.Ph.D.Committee Chair: Dr. Russell Mersereau; Committee Member: Dr. Doug Willams; Committee Member: Dr. Mark Richards; Committee Member: Dr. Xiaoming Huo; Committee Member: Dr. Ye (Geoffrey) L

Learning Representations for Controllable Image Restoration

Deep Convolutional Neural Networks have sparked a renaissance in all the sub-fields of computer vision. Tremendous progress has been made in the area of image restoration. The research community has pushed the boundaries of image deblurring, super-resolution, and denoising. However, given a distorted image, most existing methods typically produce a single restored output. The tasks mentioned above are inherently ill-posed, leading to an infinite number of plausible solutions. This thesis focuses on designing image restoration techniques capable of producing multiple restored results and granting users more control over the restoration process. Towards this goal, we demonstrate how one could leverage the power of unsupervised representation learning. Image restoration is vital when applied to distorted images of human faces due to their social significance. Generative Adversarial Networks enable an unprecedented level of generated facial details combined with smooth latent space. We leverage the power of GANs towards the goal of learning controllable neural face representations. We demonstrate how to learn an inverse mapping from image space to these latent representations, tuning these representations towards a specific task, and finally manipulating latent codes in these spaces. For example, we show how GANs and their inverse mappings enable the restoration and editing of faces in the context of extreme face super-resolution and the generation of novel view sharp videos from a single motion-blurred image of a face. This thesis also addresses more general blind super-resolution, denoising, and scratch removal problems, where blur kernels and noise levels are unknown. We resort to contrastive representation learning and first learn the latent space of degradations. We demonstrate that the learned representation allows inference of ground-truth degradation parameters and can guide the restoration process. Moreover, it enables control over the amount of deblurring and denoising in the restoration via manipulation of latent degradation features