Robust and Efficient Inference of Scene and Object Motion in Multi-Camera Systems

Multi-camera systems have the ability to overcome some of the fundamental limitations of single camera based systems. Having multiple view points of a scene goes a long way in limiting the influence of field of view, occlusion, blur and poor resolution of an individual camera. This dissertation addresses robust and efficient inference of object motion and scene in multi-camera and multi-sensor systems. The first part of the dissertation discusses the role of constraints introduced by projective imaging towards robust inference of multi-camera/sensor based object motion. We discuss the role of the homography and epipolar constraints for fusing object motion perceived by individual cameras. For planar scenes, the homography constraints provide a natural mechanism for data association. For scenes that are not planar, the epipolar constraint provides a weaker multi-view relationship. We use the epipolar constraint for tracking in multi-camera and multi-sensor networks. In particular, we show that the epipolar constraint reduces the dimensionality of the state space of the problem by introducing a ``shared'' state space for the joint tracking problem. This allows for robust tracking even when one of the sensors fail due to poor SNR or occlusion. The second part of the dissertation deals with challenges in the computational aspects of tracking algorithms that are common to such systems. Much of the inference in the multi-camera and multi-sensor networks deal with complex non-linear models corrupted with non-Gaussian noise. Particle filters provide approximate Bayesian inference in such settings. We analyze the computational drawbacks of traditional particle filtering algorithms, and present a method for implementing the particle filter using the Independent Metropolis Hastings sampler, that is highly amenable to pipelined implementations and parallelization. We analyze the implementations of the proposed algorithm, and in particular concentrate on implementations that have minimum processing times. The last part of the dissertation deals with the efficient sensing paradigm of compressing sensing (CS) applied to signals in imaging, such as natural images and reflectance fields. We propose a hybrid signal model on the assumption that most real-world signals exhibit subspace compressibility as well as sparse representations. We show that several real-world visual signals such as images, reflectance fields, videos etc., are better approximated by this hybrid of two models. We derive optimal hybrid linear projections of the signal and show that theoretical guarantees and algorithms designed for CS can be easily extended to hybrid subspace-compressive sensing. Such methods reduce the amount of information sensed by a camera, and help in reducing the so called data deluge problem in large multi-camera systems

Light field image processing: an overview

Light field imaging has emerged as a technology allowing to capture richer visual information from our world. As opposed to traditional photography, which captures a 2D projection of the light in the scene integrating the angular domain, light fields collect radiance from rays in all directions, demultiplexing the angular information lost in conventional photography. On the one hand, this higher dimensional representation of visual data offers powerful capabilities for scene understanding, and substantially improves the performance of traditional computer vision problems such as depth sensing, post-capture refocusing, segmentation, video stabilization, material classification, etc. On the other hand, the high-dimensionality of light fields also brings up new challenges in terms of data capture, data compression, content editing, and display. Taking these two elements together, research in light field image processing has become increasingly popular in the computer vision, computer graphics, and signal processing communities. In this paper, we present a comprehensive overview and discussion of research in this field over the past 20 years. We focus on all aspects of light field image processing, including basic light field representation and theory, acquisition, super-resolution, depth estimation, compression, editing, processing algorithms for light field display, and computer vision applications of light field data

Factored axis-aligned filtering for rendering multiple distribution effects

Monte Carlo (MC) ray-tracing for photo-realistic rendering often requires hours to render a single image due to the large sampling rates needed for convergence. Previous methods have attempted to filter sparsely sampled MC renders but these methods have high reconstruction overheads. Recent work has shown fast performance for individual effects, like soft shadows and indirect illumination, using axis-aligned filtering. While some components of light transport such as indirect or area illumination are smooth, they are often multiplied by high-frequency components such as texture, which prevents their sparse sampling and reconstruction. We propose an approach to adaptively sample and filter for simultaneously rendering primary (defocus blur) and secondary (soft shadows and indirect illumination) distribution effects, based on a multi-dimensional frequency analysis of the direct and indirect illumination light fields. We describe a novel approach of factoring texture and irradiance in the presence of defocus blur, which allows for pre-filtering noisy irradiance when the texture is not noisy. Our approach naturally allows for different sampling rates for primary and secondary effects, further reducing the overall ray count. While the theory considers only Lambertian surfaces, we obtain promising results for moderately glossy surfaces. We demonstrate 30x sampling rate reduction compared to equal quality noise-free MC. Combined with a GPU implementation and low filtering over-head, we can render scenes with complex geometry and diffuse and glossy BRDFs in a few seconds.National Science Foundation (U.S.) (Grant CGV 1115242)National Science Foundation (U.S.) (Grant CGV 1116303)Intel Corporation (Science and Technology Center for Visual Computing

Image restoration from noisy and limited measurements with applications in 3D imaging

The recovery of image data from noisy and limited measurements is an important problem in image processing with many practical applications. Despite great improvements in imaging devices over the past few years, the need for a fast and robust recovery method is still essential, especially in fields such as medical imaging or remote sensing. These methods are also important for new imaging modalities where the quality of data is still limited due to current state of technology. This thesis investigates novel methods to recover signals and images from noisy or sparse measurements, in new imaging modalities, for practical 3D imaging applications. In particular, the following problems are considered. First, the Tree-based Orthogonal Matching Pursuit (TOMP) algorithm is proposed to recover sparse signals with tree structure. This is an improvement over the Orthogonal Matching Pursuit method with the incorporation of the sparse-tree prior on the data. A theoretical condition on the recovery performance as well as a detailed complexity analysis is derived. Extensive experiments are carried out to compare the proposed method with other state-of-the-art algorithms. Second, a new point clouds registration method is investigated and applied for 3D model reconstruction with a depth camera, which is a recently introduced device with many potential applications in 3D imaging and human-machine interaction. Currently, the depth camera is limited in resolution and suffers from complex types of noise. In the proposed method, the Implicit Moving Least Squares (IMLS) method is employed to derive a more robust registration method which can deal with noisy point clouds. Given a good registration, information from multiple depth images can be integrated together to help reduce the effects of noise and possibly increase the resolution. This method is essential to bring commodity depth cameras to new applications that demand accurate depth information. Third, a hybrid system which consists of a light-field camera and a depth camera rigidly attached together is proposed. The system can be applied for digital refocusing on an arbitrary surface and for recovering complex reflectance information of a surface. The light-field camera is a device that can sample the 4D spatio-angular light field and allows one to refocus the captured image digitally. Given light-field information, it is possible to rearrange the light rays appropriately to render novel views or to generate refocused photographs. In theory, it is possible to estimate the depth map from a light field. However, there is a trade-off between angular and spatial resolution in current designs of light-field cameras, which leads to low quality and resolution of the estimated depth map. Moreover, for advanced 3D imaging applications, it is important to have good quality geometric and radiometric information. Thus, a depth camera is attached to the light-field camera to achieve this goal. The calibration of the system is presented in detail. The proposed system is demonstrated to create a refocused image on an arbitrary surface. However, we believe that the proposed system has great potential in more advanced imaging applications