Learning to Synthesize a 4D RGBD Light Field from a Single Image

We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction). For training, we introduce the largest public light field dataset, consisting of over 3300 plenoptic camera light fields of scenes containing flowers and plants. Our synthesis pipeline consists of a convolutional neural network (CNN) that estimates scene geometry, a stage that renders a Lambertian light field using that geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Our algorithm builds on recent view synthesis methods, but is unique in predicting RGBD for each light field ray and improving unsupervised single image depth estimation by enforcing consistency of ray depths that should intersect the same scene point. Please see our supplementary video at https://youtu.be/yLCvWoQLnmsComment: International Conference on Computer Vision (ICCV) 201

Dynamic Focusing of Large Arrays for Wireless Power Transfer and Beyond

We present architectures, circuits, and algorithms for dynamic 3-D lensing and focusing of electromagnetic power in radiative near- and far-field regions by arrays that can be arbitrary and nonuniform. They can benefit applications such as wireless power transfer at a distance (WPT-AD), volumetric sensing and imaging, high-throughput communications, and optical phased arrays. Theoretical limits on system performance are calculated. An adaptive algorithm focuses the power at the receiver(s) without prior knowledge of its location(s). It uses orthogonal bases to change the phases of multiple elements simultaneously to enhance the dynamic range. One class of such 2-D orthogonal and pseudo-orthogonal masks is constructed using the Hadamard and pseudo-Hadamard matrices. Generation and recovery units (GU and RU) work collaboratively to focus energy quickly and reliably with no need for factory calibration. Orthogonality enables batch processing in high-latency and low-rate communication settings. Secondary vector-based calculations allow instantaneous refocusing at different locations using element-wise calculations. An emulator enables further evaluation of the system. We demonstrate modular WPT-AD GUs of up to 400 elements utilizing arrays of 65-nm CMOS ICs to focus power on RUs that convert the RF power to dc. Each RFIC synthesizes 16 independently phase-controlled RF outputs around 10 GHz from a common single low-frequency reference. Detailed measurements demonstrate the feasibility and effectiveness of RF lensing techniques presented in this article. More than 2 W of dc power can be recovered through a wireless transfer at distances greater than 1 m. The system can dynamically project power at various angles and at distances greater than 10 m. These developments are another step toward unified wireless power, sensing, and communication solutions in the future

Coded aperture and coded exposure photography : an investigation into applications and methods

This dissertation presents an introduction to the field of computational photography, and provides a survey of recent research. Specific attention is given to coded aperture and coded exposure theory and methods, as these form the basis for the experiments performed

Multiplexed photography : single-exposure capture of multiple camera settings

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 115-124).The space of camera settings is large and individual settings can vary dramatically from scene to scene. This thesis explores methods for capturing and manipulating multiple camera settings in a single exposure. Multiplexing multiple camera settings in a single exposure can allow post-exposure control and improve the quality of photographs taken in challenging lighting environments (e.g. low light or high motion). We first describe the design and implementation of a prototype optical system and associated algorithms to capture four images of a scene in a single exposure, each taken with a different aperture setting. Our system can be used with commercially available DSLR cameras and photographic lenses without modification to either. We demonstrate several applications of our multi-aperture camera, such as post-exposure depth of field control, synthetic refocusing, and depth-guided deconvolution. Next we describe multiplexed flash illumination to recover both flash and ambient light information as well as extract depth information in a single exposure. Traditional photographic flashes illuminate the scene with a spatially-constant light beam. By adding a mask and optics to a flash, we can project a spatially varying illumination onto the scene which allows us to spatially multiplex the flash and ambient illuminations onto the imager. We apply flash multiplexing to enable single exposure flash/no-flash image fusion, in particular, performing flash/no-flash relighting on dynamic scenes with moving objects. Finally, we propose spatio-temporal multiplexing, a novel image sensor feature that enables simultaneous capture of flash and ambient illumination.(cont.) We describe two possible applications of spatio-temporal multiplexing: single-image flash/no-flash relighting and white balancing scenes containing two distinct illuminants (e.g. flash and fluorescent lighting).by Paul Elijah Green.Ph.D

Large-Scale Light Field Capture and Reconstruction

This thesis discusses approaches and techniques to convert Sparsely-Sampled Light Fields (SSLFs) into Densely-Sampled Light Fields (DSLFs), which can be used for visualization on 3DTV and Virtual Reality (VR) devices. Exemplarily, a movable 1D large-scale light field acquisition system for capturing SSLFs in real-world environments is evaluated. This system consists of 24 sparsely placed RGB cameras and two Kinect V2 sensors. The real-world SSLF data captured with this setup can be leveraged to reconstruct real-world DSLFs. To this end, three challenging problems require to be solved for this system: (i) how to estimate the rigid transformation from the coordinate system of a Kinect V2 to the coordinate system of an RGB camera; (ii) how to register the two Kinect V2 sensors with a large displacement; (iii) how to reconstruct a DSLF from a SSLF with moderate and large disparity ranges. To overcome these three challenges, we propose: (i) a novel self-calibration method, which takes advantage of the geometric constraints from the scene and the cameras, for estimating the rigid transformations from the camera coordinate frame of one Kinect V2 to the camera coordinate frames of 12-nearest RGB cameras; (ii) a novel coarse-to-fine approach for recovering the rigid transformation from the coordinate system of one Kinect to the coordinate system of the other by means of local color and geometry information; (iii) several novel algorithms that can be categorized into two groups for reconstructing a DSLF from an input SSLF, including novel view synthesis methods, which are inspired by the state-of-the-art video frame interpolation algorithms, and Epipolar-Plane Image (EPI) inpainting methods, which are inspired by the Shearlet Transform (ST)-based DSLF reconstruction approaches

Improved methods for functional neuronal imaging with genetically encoded voltage indicators

Voltage imaging has the potential to revolutionise neuronal physiology, enabling high temporal and spatial resolution monitoring of sub- and supra-threshold activity in genetically defined cell classes. Before this goal is reached a number of challenges must be overcome: novel optical, genetic, and experimental techniques must be combined to deal with voltage imaging’s unique difficulties. In this thesis three techniques are applied to genetically encoded voltage indicator (GEVI) imaging. First, I describe a multifocal two-photon microscope and present a novel source localisation control and reconstruction algorithm to increase scattering resistance in functional imaging. I apply this microscope to image population and single-cell voltage signals from voltage sensitive fluorescent proteins in the first demonstration of multifocal GEVI imaging. Second, I show that a recently described genetic technique that sparsely labels cortical pyramidal cells enables single-cell resolution imaging in a one-photon widefield imaging configuration. This genetic technique allows simple, high signal-to-noise optical access to the primary excitatory cells in the cerebral cortex. Third, I present the first application of lightfield microscopy to single cell resolution neuronal voltage imaging. This technique enables single-shot capture of dendritic arbours and resolves 3D localised somatic and dendritic voltage signals. These approaches are finally evaluated for their contribution to the improvement of voltage imaging for physiology.Open Acces

Efficient and Accurate Disparity Estimation from MLA-Based Plenoptic Cameras

This manuscript focuses on the processing images from microlens-array based plenoptic cameras. These cameras enable the capturing of the light field in a single shot, recording a greater amount of information with respect to conventional cameras, allowing to develop a whole new set of applications. However, the enhanced information introduces additional challenges and results in higher computational effort. For one, the image is composed of thousand of micro-lens images, making it an unusual case for standard image processing algorithms. Secondly, the disparity information has to be estimated from those micro-images to create a conventional image and a three-dimensional representation. Therefore, the work in thesis is devoted to analyse and propose methodologies to deal with plenoptic images. A full framework for plenoptic cameras has been built, including the contributions described in this thesis. A blur-aware calibration method to model a plenoptic camera, an optimization method to accurately select the best microlenses combination, an overview of the different types of plenoptic cameras and their representation. Datasets consisting of both real and synthetic images have been used to create a benchmark for different disparity estimation algorithm and to inspect the behaviour of disparity under different compression rates. A robust depth estimation approach has been developed for light field microscopy and image of biological samples

Using Shadows to Detect Targets in Synthetic Aperture Radar Imagery

Synthetic Aperture Radar (SAR) can generate high resolution imagery of re- mote scenes by combining the phase information of multiple radar pulses along a given path. SAR based Intelligence, Surveillance, and Reconnaissance (ISR) has the advantage over optical ISR that it can provide usable imagery in adverse weather or nighttime conditions. Certain radar frequencies can even result in foliage or limited soil penetration, enabling imagery to be created of objects of interest that would otherwise be hidden from optical surveillance systems. This thesis demonstrates the capability of locating stationary targets of interest based on the locations of their shadows and the characteristics of pixel intensity distributions within the SAR imagery. Shadows, in SAR imagery, represent the absence of a detectable signal reflection due to the physical obstruction of the transmitted radar energy. An object\u27s shadow indicates its true geospatial location. This thesis demonstrates target detection based on shadow location using three types of target vehicles, each located in urban and rural clutter scenes, from the publicly available Moving and Stationary Target Acquisition and Recognition (MSTAR) data set. The proposed distribution characterization method for detecting shadows demonstrates the capability of isolating distinct regions within SAR imagery and using the junctions between shadow and non-shadow regions to locate individual shadow-casting objects. Targets of interest are then located within that collection of objects with an average detection accuracy rate of 93%. The shadow-based target detection algorithm results in a lower false alarm rate compared to previous research conducted with the same data set, with 71% fewer false alarms for the same clutter region. Utilizing the absence of signal, in conjunction with surrounding signal reflections, provides accurate stationary target detection. This capability could greatly assist in track initialization or the location of otherwise obscured targets of interest

Development of a light-sheet fluorescence microscope employing an ALPAO deformable mirror to achieve video-rate remote refocusing and volumetric imaging.

There are numerous situations in microscopy where it is desirable to remotely refocus a microscope employing a high numerical aperture (NA) objective lens. This thesis describes the characterisation, development and implementation of an Alpao membrane deformable mirror-based system to achieve this goal for a light-sheet fluorescence microscope (LSFM). The Alpao deformable mirror (DM) DM97-15 used is this work has 97 actuators and was sufficiently fast to perform refocus sweeps at 25 Hz and faster. However, a known issue with using Alpao deformable mirrors in open-loop mode is that they exhibit viscoelastic creep and temperature- dependent variations in the mirror response. The effect of visco-elastic creep was reduced by ensuring that the mirror profile was on average constant on timescales shorter than the characteristic time of the visco-elastic creep. The thermal effect was managed by ensuring that the electrical power delivered to the actuators was constant prior to optimisation and use. This was achieved by ensuring that the frequency and amplitude of oscillation of the mirror was constant prior to optimisation, so that it reached a thermal steady state, was approximately constant during optimisation and constant during use. The image-based optimisation procedure employed used an estimate of the Strehl ratio of the optical system calculated from an image of an array of 1 μm diameter holes. The optimisation procedure included optimising the amount of high-NA defocus and the Zernike modes from Noll indices 4 to 24. The system was tested at 26.3 refocus sweeps per second over a refocus range of -50 to 50 μm with a 40x/0.85 air objective and a 40x/0.80 water immersion objective. The air objective enabled a mean Strehl metric of more than 0.6 over a lateral field of view of 200x200 microns and for a refocus range of 45 microns. The water objective achieved a mean Strehl metric of more than 0.6 over a lateral field of view of 200x200 microns over a larger refocus range of 77 microns. The DM-based refocusing system was then incorporated into a LSFM setup. The spatial resolution of the system was characterised using fluorescent beads imaged volumetrically at 26.3 volumes per second. The performance of the system was also demonstrated for imaging fluorescence pollen grain samples.Open Acces