Computational low-light flash photography

Ph.DDOCTOR OF PHILOSOPH

Reasoning about Scene and Image Structure for Computer Vision

The wide availability of cheap consumer cameras has democratized photography for novices and experts alike, with more than a trillion photographs taken each year. While many of these cameras---especially those on mobile phones---have inexpensive optics and make imperfect measurements, the use of modern computational techniques can allow the recovery of high-quality photographs as well as of scene attributes. In this dissertation, we explore algorithms to infer a wide variety of physical and visual properties of the world, including color, geometry, reflectance etc., from images taken by casual photographers in unconstrained settings. We specifically focus on neural network-based methods, while incorporating domain knowledge about scene structure and the physics of image formation. We describe novel techniques to produce high-quality images in poor lighting environments, train scene map estimators in the absence of ground-truth data and learn to output our understanding and uncertainty on the scene given observed images. The key to inferring scene properties from casual photography is to exploit the internal structure of natural scenes and the expressive capacity of neural networks. We demonstrate that neural networks can be used to identify the internal structure of scenes maps, and that our prior understanding on natural scenes can shape the design, training and the output representation of neural networks

Restoration and enhancement of historical stereo photos

Restoration of digital visual media acquired from repositories of historical photographic and cinematographic material is of key importance for the preservation, study and transmission of the legacy of past cultures to the coming generations. In this paper, a fully automatic approach to the digital restoration of historical stereo photographs is proposed, referred to as Stacked Median Restoration plus (SMR+). The approach exploits the content redundancy in stereo pairs for detecting and fixing scratches, dust, dirt spots and many other defects in the original images, as well as improving contrast and illumination. This is done by estimating the optical flow between the images, and using it to register one view onto the other both geometrically and photometrically. Restoration is then accomplished in three steps: (1) image fusion according to the stacked median operator, (2) low-resolution detail enhancement by guided supersampling, and (3) iterative visual consistency checking and refinement. Each step implements an original algorithm specifically designed for this work. The restored image is fully consistent with the original content, thus improving over the methods based on image hallucination. Comparative results on three different datasets of historical stereograms show the effectiveness of the proposed approach, and its superiority over single-image denoising and super-resolution methods. Results also show that the performance of the state-of-the-art single-image deep restoration network Bringing Old Photo Back to Life (BOPBtL) can be strongly improved when the input image is pre-processed by SMR+