Machine Learning Approaches to Image Deconvolution

Image blur is a fundamental problem in both photography and scientific imaging. Even the most well-engineered optics are imperfect, and finite exposure times cause motion blur. To reconstruct the original sharp image, the field of image deconvolution tries to recover recorded photographs algorithmically. When the blur is known, this problem is called non-blind deconvolution. When the blur is unknown and has to be inferred from the observed image, it is called blind deconvolution. The key to reconstructing information lost due to blur and noise is to use prior knowledge. To this end, this thesis develops approaches inspired by machine learning that include more available information and advance the current state of the art for both non-blind and blind image deconvolution. Optical aberrations of a lens are encoded in an initial calibration step as a spatially-varying point spread function. With prior information about the distribution of gradients in natural images, the original image is reconstructed in a maximum a posteriori (MAP) estimation, with results comparing favorably to previous methods. By including the camera’s color filter array in the forward model, the estimation procedure can perform demosaicing and deconvolution jointly and thereby surpass the quality of the results yielded by a separate demosaicing step. The applicability of removing optical aberrations is broadened further by estimating the point spread function from the image itself. We extend an existing MAP-based blind deconvolution approach to the first algorithm that is able to remove spatially-varying lens blur blindly, including chromatic aberrations. The properties of lenses restrict the class of possible point spread functions and reduce the space of parameters to be inferred, enabling results on par with the best non-blind approaches for the lenses tested in our experiments. To capture more information about the distribution of natural images and capitalize on the abundance of training data, neural networks prove to be a useful tool. As other successful non-blind deconvolution methods, a regularized inversion of the blur is performed in the Fourier domain as an initial step. Next, a large neural network learns the mapping from the preprocessed image back to the uncorrupted original. The trained network surpasses results of state-of-the-art algorithms on both artificial and real-world examples. For the first time, a learning approach also succeeds in blind image deconvolution. A deep neural network “unrolls” the estimation procedure of existing methods for this task. After training end-to-end on artificially generated example images, the network achieves performance competitive with state-of-the-art methods in the generic case, and even goes beyond when trained for a specific image category.Unscharfe Bilder sind ein häufiges Problem, sowohl in der Fotografie als auch in der wissenschaftlichen Bildgebung. Auch die leistungsfähigsten optischen Systeme sind nicht perfekt, und endliche Belichtungszeiten verursachen Bewegungsunschärfe. Dekonvolution hat das Ziel das ursprünglich scharfe Bild aus der Aufnahme mit Hilfe von algorithmischen Verfahren wiederherzustellen. Kennt man die exakte Form der Unschärfe, so wird dieses Rekonstruktions-Problem als nicht-blinde Dekonvolution bezeichnet. Wenn die Unschärfe aus dem Bild selbst inferiert werden muss, so spricht man von blinder Dekonvolution. Der Schlüssel zum Wiederherstellen von verlorengegangener Bildinformation liegt im Verwenden von verfügbarem Vorwissen über Bilder und die Entstehung der Unschärfe. Hierzu entwickelt diese Arbeit verschiedene Ansätze um dieses Vorwissen besser verwenden zu können, basierend auf Methoden des maschinellen Lernens, und verbessert damit den Stand der Technik, sowohl für nicht-blinde als auch für blinde Dekonvolution. Optische Abbildungsfehler lassen sich in einem einmal ausgeführten Kalibrierungsschritt vermessen und als eine ortsabhängige Punktverteilungsfunktion des einfallenden Lichtes beschreiben. Mit dem Vorwissen über die Verteilung von Gradienten in Bildern kann das ursprüngliche Bild durch eine Maximum-a-posteriori (MAP) Schätzung wiederhergestellt werden, wobei die resultierenden Ergebnisse vergleichbare Methoden übertreffen. Wenn man des Weiteren im Vorwärtsmodell die Farbfilter des Sensors berücksichtigt, so kann das Schätzverfahren Demosaicking und Dekonvolution simultan ausführen, in einer Qualität die den Ergebnissen durch Demosaicking in einem separaten Schritt überlegen ist. Die Korrektur von Linsenfehlern wird breiter anwendbar indem man die Punktverteilungsfunktion vom Bild selbst inferiert. Wir erweitern einen existierenden MAP-basierenden Ansatz für blinde Dekonvolution zum ersten Algorithmus, der in der Lage ist auch ortsabhängige optische Unschärfen blind zu entfernen, einschließlich chromatischer Aberration. Die spezifischen Eigenschaften von Kamera-Objektiven schränken den Raum der zu schätzenden Punktverteilungsfunktionen weit genug ein, so dass für die in unseren Experimenten untersuchten Objektive die erreichte Bildrekonstruktion ähnlich erfolgreich ist wie bei nicht-blinden Verfahren. Es zeigt sich, dass neuronale Netze von im Überfluss vorhandenen Bilddatenbanken profitieren können um mehr über die Bildern zugrundeliegende Wahrscheinlichkeitsverteilung zu lernen. Ähnlich wie in anderen erfolgreichen nicht-blinden Dekonvolutions-Ansätzen wird die Unschärfe zuerst durch eine regularisierte Inversion im Fourier-Raum vermindert. Danach ist es einem neuronalen Netz mit großer Kapazität möglich zu lernen, wie aus einem derart vorverarbeiteten Bild das fehlerfreie Original geschätzt werden kann. Das trainierte Netz produziert anderen Methoden überlegene Ergebnisse, sowohl auf künstlich generierten Beispielen als auch auf tatsächlichen unscharfen Fotos. Zum ersten Mal ist ein lernendes Verfahren auch hinsichtlich der blinden Bild-Dekonvolution erfolgreich. Ein tiefes neuronales Netz modelliert die Herangehensweise von bisherigen Schätzverfahren und wird auf künstlich generierten Beispielen trainiert die Unschärfe vorherzusagen. Nach Abschluss des Trainings ist es in der Lage, mit anderen aktuellen Methoden vergleichbare Ergebnisse zu erzielen, und geht über deren Ergebnisse hinaus, wenn man speziell für eine bestimmten Subtyp von Bildern trainiert

Observational and Theoretical Investigation of Cylindrical Line Source Blast Theory Using Meteors

During their passage through the atmosphere meteoroids produce a hypersonic shock which may be recorded at the ground in the form of infrasound. The first objective of this project was to use global infrasound measurements to estimate the influx of large (meter/decameter) objects to Earth and investigate which parameters of their ablation and disruption can be determined using infrasound records. A second objective was to evaluate and extend existing cylindrical line source blast theory for meteoroids by combining new observations with earlier analytical models, and validate these against centimetre-sized optical meteor observations. The annual terrestrial influx of large meteoroids (kinetic energies above a threshold E) was found to be N=4.5E–0.6 where E is expressed in kilotons of TNT equivalent. This indicates that estimates of the influx derived from telescopic surveys of small asteroids near Earth are too low. Infrasound records from an event over Indonesia in 2009 were used to develop a technique to estimate the altitude of meteoroid terminal bursts and their energies. The burst altitude in this case was determined to be near 20 kilometers and the energy between 8 – 67 kilotons of TNT equivalent. Using a network of optical cameras and an Infrasound Array in southern Ontario, Canada, 71 centimetre-sized meteoroids were optically detected and associated with infrasonic signals recorded at the ground. The shock source height and its uncertainty along the meteor trail from raytracing was determined including wind effects due to gravity waves perturbations, which were found to be significant for such short range (km) infrasound propagation. Approximately 75% of signals were attributed to cylindrical line source geometry, while ray deviation angles greater than 117° were associated with spherical shocks. The ReVelle (1974) meteor infrasound model was found to be accurate when using infrasound period measurements, but systematically under-predicted blast radii when amplitude is used. The latter can be better modelled assuming the wave distortion distance is “\u3c6%, as opposed to the 10% adopted by ReVelle. Infrasonic masses found from ReVelle’s theory deviate from photometric estimates largely due to meteoroid fragmentation

Image Color Correction, Enhancement, and Editing

This thesis presents methods and approaches to image color correction, color enhancement, and color editing. To begin, we study the color correction problem from the standpoint of the camera's image signal processor (ISP). A camera's ISP is hardware that applies a series of in-camera image processing and color manipulation steps, many of which are nonlinear in nature, to render the initial sensor image to its final photo-finished representation saved in the 8-bit standard RGB (sRGB) color space. As white balance (WB) is one of the major procedures applied by the ISP for color correction, this thesis presents two different methods for ISP white balancing. Afterwards, we discuss another scenario of correcting and editing image colors, where we present a set of methods to correct and edit WB settings for images that have been improperly white-balanced by the ISP. Then, we explore another factor that has a significant impact on the quality of camera-rendered colors, in which we outline two different methods to correct exposure errors in camera-rendered images. Lastly, we discuss post-capture auto color editing and manipulation. In particular, we propose auto image recoloring methods to generate different realistic versions of the same camera-rendered image with new colors. Through extensive evaluations, we demonstrate that our methods provide superior solutions compared to existing alternatives targeting color correction, color enhancement, and color editing

Stereoscopic high dynamic range imaging

Two modern technologies show promise to dramatically increase immersion in virtual environments. Stereoscopic imaging captures two images representing the views of both eyes and allows for better depth perception. High dynamic range (HDR) imaging accurately represents real world lighting as opposed to traditional low dynamic range (LDR) imaging. HDR provides a better contrast and more natural looking scenes. The combination of the two technologies in order to gain advantages of both has been, until now, mostly unexplored due to the current limitations in the imaging pipeline. This thesis reviews both fields, proposes stereoscopic high dynamic range (SHDR) imaging pipeline outlining the challenges that need to be resolved to enable SHDR and focuses on capture and compression aspects of that pipeline. The problems of capturing SHDR images that would potentially require two HDR cameras and introduce ghosting, are mitigated by capturing an HDR and LDR pair and using it to generate SHDR images. A detailed user study compared four different methods of generating SHDR images. Results demonstrated that one of the methods may produce images perceptually indistinguishable from the ground truth. Insights obtained while developing static image operators guided the design of SHDR video techniques. Three methods for generating SHDR video from an HDR-LDR video pair are proposed and compared to the ground truth SHDR videos. Results showed little overall error and identified a method with the least error. Once captured, SHDR content needs to be efficiently compressed. Five SHDR compression methods that are backward compatible are presented. The proposed methods can encode SHDR content to little more than that of a traditional single LDR image (18% larger for one method) and the backward compatibility property encourages early adoption of the format. The work presented in this thesis has introduced and advanced capture and compression methods for the adoption of SHDR imaging. In general, this research paves the way for a novel field of SHDR imaging which should lead to improved and more realistic representation of captured scenes

UAVs for the Environmental Sciences

This book gives an overview of the usage of UAVs in environmental sciences covering technical basics, data acquisition with different sensors, data processing schemes and illustrating various examples of application

Doppler Encoded Excitation Patterning (DEEP) Microscopy

Traditional optical imaging systems rely on lenses and spatially-resolved detection to probe distinct locations on the object. We develop a novel computational approach to 2D and 3D imaging that instead measures the object\u27s spatial Fourier transform using a single-element detector and without requiring precision optics. This wide-field technique can be used to image biological and synthetic structures in fluoresced or scattered light using coherent or broadband illumination. It employs dynamic structured illumination, acousto-optics, RF electronics, and tomographic algorithms to circumvent several trade-offs in conventional imaging, such as the dependence of the optical transfer function on the imaging lenses and the coupling of resolution and depth of field. We use Fourier optics concepts to derive the dynamic optical transfer function, evaluate different Fourier sampling strategies, and investigate and compare tomographic algorithms for 2D and 3D image synthesis. We also develop conceptual and analytical models to describe imaging of fluorescent as well as amplitude and phase scattering objects, the effects of broadband and spatially-incoherent illumination, and nonlinear wide-field super-resolution imaging. We consider sources of noise, analyze and simulate SNR behavior for several types of noise and Fourier sampling strategies, and compare the sensitivity of the technique to conventional imaging. We describe several experimental proof-of-concept systems and present two-dimensional high-resolution tomographic image reconstructions in both scattered and fluoresced light demonstrating a thousandfold improvement in the depth of field compared to conventional lens-based microscopy. Finally, we explore approaches for high-speed Fourier sampling and propose several related sensing techniques, including wide-field fluorescence imaging in scattering media