{HDR} Denoising and Deblurring by Learning Spatio-temporal Distortion Model

We seek to reconstruct sharp and noise-free high-dynamic range (HDR) video from a dual-exposure sensor that records different low-dynamic range (LDR) information in different pixel columns: Odd columns provide low-exposure, sharp, but noisy information; even columns complement this with less noisy, high-exposure, but motion-blurred data. Previous LDR work learns to deblur and denoise (DISTORTED->CLEAN) supervised by pairs of CLEAN and DISTORTED images. Regrettably, capturing DISTORTED sensor readings is time-consuming; as well, there is a lack of CLEAN HDR videos. We suggest a method to overcome those two limitations. First, we learn a different function instead: CLEAN->DISTORTED, which generates samples containing correlated pixel noise, and row and column noise, as well as motion blur from a low number of CLEAN sensor readings. Second, as there is not enough CLEAN HDR video available, we devise a method to learn from LDR video in-stead. Our approach compares favorably to several strong baselines, and can boost existing methods when they are re-trained on our data. Combined with spatial and temporal super-resolution, it enables applications such as re-lighting with low noise or blur

Super resolution and dynamic range enhancement of image sequences

Camera producers try to increase the spatial resolution of a camera by reducing size of sites on sensor array. However, shot noise causes the signal to noise ratio drop as sensor sites get smaller. This fact motivates resolution enhancement to be performed through software. Super resolution (SR) image reconstruction aims to combine degraded images of a scene in order to form an image which has higher resolution than all observations. There is a demand for high resolution images in biomedical imaging, surveillance, aerial/satellite imaging and high-definition TV (HDTV) technology. Although extensive research has been conducted in SR, attention has not been given to increase the resolution of images under illumination changes. In this study, a unique framework is proposed to increase the spatial resolution and dynamic range of a video sequence using Bayesian and Projection onto Convex Sets (POCS) methods. Incorporating camera response function estimation into image reconstruction allows dynamic range enhancement along with spatial resolution improvement. Photometrically varying input images complicate process of projecting observations onto common grid by violating brightness constancy. A contrast invariant feature transform is proposed in this thesis to register input images with high illumination variation. Proposed algorithm increases the repeatability rate of detected features among frames of a video. Repeatability rate is increased by computing the autocorrelation matrix using the gradients of contrast stretched input images. Presented contrast invariant feature detection improves repeatability rate of Harris corner detector around %25 on average. Joint multi-frame demosaicking and resolution enhancement is also investigated in this thesis. Color constancy constraint set is devised and incorporated into POCS framework for increasing resolution of color-filter array sampled images. Proposed method provides fewer demosaicking artifacts compared to existing POCS method and a higher visual quality in final image

Variability and Proper Motion of X-ray Knots in the Jet of Centaurus A

Accepted to ApJ, 14 pages, 8 figures, 2 tablesWe report results from Chandra observations analyzed for evidence of variability and proper motion in the X-ray jet of Centaurus A. Using data spanning 15 yr, collective proper motion of 11.3 ± 3.3 mas yr -1 , or 0.68 ± 0.20c, is detected for the fainter X-ray knots and other substructure present within the jet. The three brightest knots (AX1A, AX1C, and BX2) are found to be stationary to an upper limit of . Brightness variations up to 27% are detected for several X-ray knots in the jet. For the fading knots, BX2 and AX1C, the changes in spectral slope expected to accompany synchrotron cooling are not found, ruling it out and placing upper limits of ≃80 μG for each of their magnetic field strengths. Adiabatic expansion can account for the observed decreases in brightness. Constraints on models for the origin of the knots are established. Jet plasma overrunning an obstacle is favored as the generator of stationary knots, while moving knots are likely produced either by internal differences in jet speed or the late stages of jet interaction with nebular or cloud material.Peer reviewe

A Dual Sensor Computational Camera for High Quality Dark Videography

Videos captured under low light conditions suffer from severe noise. A variety of efforts have been devoted to image/video noise suppression and made large progress. However, in extremely dark scenarios, extensive photon starvation would hamper precise noise modeling. Instead, developing an imaging system collecting more photons is a more effective way for high-quality video capture under low illuminations. In this paper, we propose to build a dual-sensor camera to additionally collect the photons in NIR wavelength, and make use of the correlation between RGB and near-infrared (NIR) spectrum to perform high-quality reconstruction from noisy dark video pairs. In hardware, we build a compact dual-sensor camera capturing RGB and NIR videos simultaneously. Computationally, we propose a dual-channel multi-frame attention network (DCMAN) utilizing spatial-temporal-spectral priors to reconstruct the low-light RGB and NIR videos. In addition, we build a high-quality paired RGB and NIR video dataset, based on which the approach can be applied to different sensors easily by training the DCMAN model with simulated noisy input following a physical-process-based CMOS noise model. Both experiments on synthetic and real videos validate the performance of this compact dual-sensor camera design and the corresponding reconstruction algorithm in dark videography

Multimedia Forensics

This book is open access. Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field

Machine Learning Approaches to Image Deconvolution

Image blur is a fundamental problem in both photography and scientific imaging. Even the most well-engineered optics are imperfect, and finite exposure times cause motion blur. To reconstruct the original sharp image, the field of image deconvolution tries to recover recorded photographs algorithmically. When the blur is known, this problem is called non-blind deconvolution. When the blur is unknown and has to be inferred from the observed image, it is called blind deconvolution. The key to reconstructing information lost due to blur and noise is to use prior knowledge. To this end, this thesis develops approaches inspired by machine learning that include more available information and advance the current state of the art for both non-blind and blind image deconvolution. Optical aberrations of a lens are encoded in an initial calibration step as a spatially-varying point spread function. With prior information about the distribution of gradients in natural images, the original image is reconstructed in a maximum a posteriori (MAP) estimation, with results comparing favorably to previous methods. By including the camera’s color filter array in the forward model, the estimation procedure can perform demosaicing and deconvolution jointly and thereby surpass the quality of the results yielded by a separate demosaicing step. The applicability of removing optical aberrations is broadened further by estimating the point spread function from the image itself. We extend an existing MAP-based blind deconvolution approach to the first algorithm that is able to remove spatially-varying lens blur blindly, including chromatic aberrations. The properties of lenses restrict the class of possible point spread functions and reduce the space of parameters to be inferred, enabling results on par with the best non-blind approaches for the lenses tested in our experiments. To capture more information about the distribution of natural images and capitalize on the abundance of training data, neural networks prove to be a useful tool. As other successful non-blind deconvolution methods, a regularized inversion of the blur is performed in the Fourier domain as an initial step. Next, a large neural network learns the mapping from the preprocessed image back to the uncorrupted original. The trained network surpasses results of state-of-the-art algorithms on both artificial and real-world examples. For the first time, a learning approach also succeeds in blind image deconvolution. A deep neural network “unrolls” the estimation procedure of existing methods for this task. After training end-to-end on artificially generated example images, the network achieves performance competitive with state-of-the-art methods in the generic case, and even goes beyond when trained for a specific image category.Unscharfe Bilder sind ein häufiges Problem, sowohl in der Fotografie als auch in der wissenschaftlichen Bildgebung. Auch die leistungsfähigsten optischen Systeme sind nicht perfekt, und endliche Belichtungszeiten verursachen Bewegungsunschärfe. Dekonvolution hat das Ziel das ursprünglich scharfe Bild aus der Aufnahme mit Hilfe von algorithmischen Verfahren wiederherzustellen. Kennt man die exakte Form der Unschärfe, so wird dieses Rekonstruktions-Problem als nicht-blinde Dekonvolution bezeichnet. Wenn die Unschärfe aus dem Bild selbst inferiert werden muss, so spricht man von blinder Dekonvolution. Der Schlüssel zum Wiederherstellen von verlorengegangener Bildinformation liegt im Verwenden von verfügbarem Vorwissen über Bilder und die Entstehung der Unschärfe. Hierzu entwickelt diese Arbeit verschiedene Ansätze um dieses Vorwissen besser verwenden zu können, basierend auf Methoden des maschinellen Lernens, und verbessert damit den Stand der Technik, sowohl für nicht-blinde als auch für blinde Dekonvolution. Optische Abbildungsfehler lassen sich in einem einmal ausgeführten Kalibrierungsschritt vermessen und als eine ortsabhängige Punktverteilungsfunktion des einfallenden Lichtes beschreiben. Mit dem Vorwissen über die Verteilung von Gradienten in Bildern kann das ursprüngliche Bild durch eine Maximum-a-posteriori (MAP) Schätzung wiederhergestellt werden, wobei die resultierenden Ergebnisse vergleichbare Methoden übertreffen. Wenn man des Weiteren im Vorwärtsmodell die Farbfilter des Sensors berücksichtigt, so kann das Schätzverfahren Demosaicking und Dekonvolution simultan ausführen, in einer Qualität die den Ergebnissen durch Demosaicking in einem separaten Schritt überlegen ist. Die Korrektur von Linsenfehlern wird breiter anwendbar indem man die Punktverteilungsfunktion vom Bild selbst inferiert. Wir erweitern einen existierenden MAP-basierenden Ansatz für blinde Dekonvolution zum ersten Algorithmus, der in der Lage ist auch ortsabhängige optische Unschärfen blind zu entfernen, einschließlich chromatischer Aberration. Die spezifischen Eigenschaften von Kamera-Objektiven schränken den Raum der zu schätzenden Punktverteilungsfunktionen weit genug ein, so dass für die in unseren Experimenten untersuchten Objektive die erreichte Bildrekonstruktion ähnlich erfolgreich ist wie bei nicht-blinden Verfahren. Es zeigt sich, dass neuronale Netze von im Überfluss vorhandenen Bilddatenbanken profitieren können um mehr über die Bildern zugrundeliegende Wahrscheinlichkeitsverteilung zu lernen. Ähnlich wie in anderen erfolgreichen nicht-blinden Dekonvolutions-Ansätzen wird die Unschärfe zuerst durch eine regularisierte Inversion im Fourier-Raum vermindert. Danach ist es einem neuronalen Netz mit großer Kapazität möglich zu lernen, wie aus einem derart vorverarbeiteten Bild das fehlerfreie Original geschätzt werden kann. Das trainierte Netz produziert anderen Methoden überlegene Ergebnisse, sowohl auf künstlich generierten Beispielen als auch auf tatsächlichen unscharfen Fotos. Zum ersten Mal ist ein lernendes Verfahren auch hinsichtlich der blinden Bild-Dekonvolution erfolgreich. Ein tiefes neuronales Netz modelliert die Herangehensweise von bisherigen Schätzverfahren und wird auf künstlich generierten Beispielen trainiert die Unschärfe vorherzusagen. Nach Abschluss des Trainings ist es in der Lage, mit anderen aktuellen Methoden vergleichbare Ergebnisse zu erzielen, und geht über deren Ergebnisse hinaus, wenn man speziell für eine bestimmten Subtyp von Bildern trainiert