Realistic Image Synthesis with Light Transport

Ph.DDOCTOR OF PHILOSOPH

Restoration of Scene Information Reflected from Non-Specular Media

A recently published experiment called dual photography exploits Helmholtz reciprocity by illuminating a scene with a pixilated light source and imaging other parts of that scene with a camera so that light transport between every pair of source-to-camera pixels is measured. The positions of the source and camera are then computationally interchanged to generate a dual image of the scene from the viewpoint of the source illuminated from the position of the camera. Although information from parts of the scene normally hidden from the camera are made available, this technique is rather contrived and therefore limited in practical applications since it requires access to the path from the source to the scene for the pixilated illumination. By radiometrically modeling the experiment described above and expanding it to the concept of indirect photography, it has been shown theoretically, by simulation and through experimentation that information in parts of the scene not directly visible to either the camera or the controlling light source can be recovered. To that end, the camera and light source (now a laser) have been collocated. The laser is reflected from a visible surface in the scene onto hidden surfaces in the scene and the camera images collect how the light is reflected from the hidden surfaces back to the visible surface. The camera images are then used to reconstruct information from the hidden surfaces in the scene. This document discusses the theory of indirect photography, describes the simulation and experiment and used to verify the theory and describes techniques used to improve the image quality, as measured by modified modulation transfer function

Compressive light field photography using overcomplete dictionaries and optimized projections

Light field photography has gained a significant research interest in the last two decades; today, commercial light field cameras are widely available. Nevertheless, most existing acquisition approaches either multiplex a low-resolution light field into a single 2D sensor image or require multiple photographs to be taken for acquiring a high-resolution light field. We propose a compressive light field camera architecture that allows for higher-resolution light fields to be recovered than previously possible from a single image. The proposed architecture comprises three key components: light field atoms as a sparse representation of natural light fields, an optical design that allows for capturing optimized 2D light field projections, and robust sparse reconstruction methods to recover a 4D light field from a single coded 2D projection. In addition, we demonstrate a variety of other applications for light field atoms and sparse coding, including 4D light field compression and denoising.Natural Sciences and Engineering Research Council of Canada (NSERC postdoctoral fellowship)United States. Defense Advanced Research Projects Agency (DARPA SCENICC program)Alfred P. Sloan Foundation (Sloan Research Fellowship)United States. Defense Advanced Research Projects Agency (DARPA Young Faculty Award

Primal-dual coding to probe light transport

We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the same time, a second sequence of patterns, derived from the first and applied in lockstep, modulates the light received at individual sensor pixels. We show that photography in this regime is equivalent to a matrix probing operation in which the elements of the scene's transport matrix are individually re-scaled and then mapped to the photo. This makes it possible to directly acquire photos in which specific light transport paths have been blocked, attenuated or enhanced. We show captured photos for several scenes with challenging light transport effects, including specular inter-reflections, caustics, diffuse inter-reflections and volumetric scattering. A key feature of primal-dual coding is that it operates almost exclusively in the optical domain: our results consist of directly-acquired, unprocessed RAW photos or differences between them.Alfred P. Sloan Foundation (Research Fellowship)United States. Defense Advanced Research Projects Agency (DARPA Young Faculty Award)Massachusetts Institute of Technology. Media Laboratory (Consortium Members

Convolutional sparse coding for high dynamic range imaging

Current HDR acquisition techniques are based on either (i) fusing multibracketed, low dynamic range (LDR) images, (ii) modifying existing hardware and capturing different exposures simultaneously with multiple sensors, or (iii) reconstructing a single image with spatially-varying pixel exposures. In this paper, we propose a novel algorithm to recover high-quality HDRI images from a single, coded exposure. The proposed reconstruction method builds on recently-introduced ideas of convolutional sparse coding (CSC); this paper demonstrates how to make CSC practical for HDR imaging. We demonstrate that the proposed algorithm achieves higher-quality reconstructions than alternative methods, we evaluate optical coding schemes, analyze algorithmic parameters, and build a prototype coded HDR camera that demonstrates the utility of convolutional sparse HDRI coding with a custom hardware platform