Depth Superresolution using Motion Adaptive Regularization

Spatial resolution of depth sensors is often significantly lower compared to that of conventional optical cameras. Recent work has explored the idea of improving the resolution of depth using higher resolution intensity as a side information. In this paper, we demonstrate that further incorporating temporal information in videos can significantly improve the results. In particular, we propose a novel approach that improves depth resolution, exploiting the space-time redundancy in the depth and intensity using motion-adaptive low-rank regularization. Experiments confirm that the proposed approach substantially improves the quality of the estimated high-resolution depth. Our approach can be a first component in systems using vision techniques that rely on high resolution depth information

Optimization for Image Segmentation

Image segmentation, i.e., assigning each pixel a discrete label, is an essential task in computer vision with lots of applications. Major techniques for segmentation include for example Markov Random Field (MRF), Kernel Clustering (KC), and nowadays popular Convolutional Neural Networks (CNN). In this work, we focus on optimization for image segmentation. Techniques like MRF, KC, and CNN optimize MRF energies, KC criteria, or CNN losses respectively, and their corresponding optimization is very different. We are interested in the synergy and the complementary benefits of MRF, KC, and CNN for interactive segmentation and semantic segmentation. Our first contribution is pseudo-bound optimization for binary MRF energies that are high-order or non-submodular. Secondly, we propose Kernel Cut, a novel formulation for segmentation, which combines MRF regularization with Kernel Clustering. We show why to combine KC with MRF and how to optimize the joint objective. In the third part, we discuss how deep CNN segmentation can benefit from non-deep (i.e., shallow) methods like MRF and KC. In particular, we propose regularized losses for weakly-supervised CNN segmentation, in which we can integrate MRF energy or KC criteria as part of the losses. Minimization of regularized losses is a principled approach to semi-supervised learning, in general. Our regularized loss method is very simple and allows different kinds of regularization losses for CNN segmentation. We also study the optimization of regularized losses beyond gradient descent. Our regularized losses approach achieves state-of-the-art accuracy in semantic segmentation with near full supervision quality

Image restoration from noisy and limited measurements with applications in 3D imaging

The recovery of image data from noisy and limited measurements is an important problem in image processing with many practical applications. Despite great improvements in imaging devices over the past few years, the need for a fast and robust recovery method is still essential, especially in fields such as medical imaging or remote sensing. These methods are also important for new imaging modalities where the quality of data is still limited due to current state of technology. This thesis investigates novel methods to recover signals and images from noisy or sparse measurements, in new imaging modalities, for practical 3D imaging applications. In particular, the following problems are considered. First, the Tree-based Orthogonal Matching Pursuit (TOMP) algorithm is proposed to recover sparse signals with tree structure. This is an improvement over the Orthogonal Matching Pursuit method with the incorporation of the sparse-tree prior on the data. A theoretical condition on the recovery performance as well as a detailed complexity analysis is derived. Extensive experiments are carried out to compare the proposed method with other state-of-the-art algorithms. Second, a new point clouds registration method is investigated and applied for 3D model reconstruction with a depth camera, which is a recently introduced device with many potential applications in 3D imaging and human-machine interaction. Currently, the depth camera is limited in resolution and suffers from complex types of noise. In the proposed method, the Implicit Moving Least Squares (IMLS) method is employed to derive a more robust registration method which can deal with noisy point clouds. Given a good registration, information from multiple depth images can be integrated together to help reduce the effects of noise and possibly increase the resolution. This method is essential to bring commodity depth cameras to new applications that demand accurate depth information. Third, a hybrid system which consists of a light-field camera and a depth camera rigidly attached together is proposed. The system can be applied for digital refocusing on an arbitrary surface and for recovering complex reflectance information of a surface. The light-field camera is a device that can sample the 4D spatio-angular light field and allows one to refocus the captured image digitally. Given light-field information, it is possible to rearrange the light rays appropriately to render novel views or to generate refocused photographs. In theory, it is possible to estimate the depth map from a light field. However, there is a trade-off between angular and spatial resolution in current designs of light-field cameras, which leads to low quality and resolution of the estimated depth map. Moreover, for advanced 3D imaging applications, it is important to have good quality geometric and radiometric information. Thus, a depth camera is attached to the light-field camera to achieve this goal. The calibration of the system is presented in detail. The proposed system is demonstrated to create a refocused image on an arbitrary surface. However, we believe that the proposed system has great potential in more advanced imaging applications

High-quality computed tomography using advanced model-based iterative reconstruction

Computed Tomography (CT) is an essential technology for the treatment, diagnosis, and study of disease, providing detailed three-dimensional images of patient anatomy. While CT image quality and resolution has improved in recent years, many clinical tasks require visualization and study of structures beyond current system capabilities. Model-Based Iterative Reconstruction (MBIR) techniques offer improved image quality over traditional methods by incorporating more accurate models of the imaging physics. In this work, we seek to improve image quality by including high-fidelity models of CT physics in a MBIR framework. Specifically, we measure and model spectral effects, scintillator blur, focal-spot blur, and gantry motion blur, paying particular attention to shift-variant blur properties and noise correlations. We derive a novel MBIR framework that is capable of modeling a wide range of physical effects, and use this framework with the physical models to reconstruct data from various systems. Physical models of varying degrees of accuracy are compared with each other and more traditional techniques. Image quality is assessed with a variety of metrics, including bias, noise, and edge-response, as well as task specific metrics such as segmentation quality and material density accuracy. These results show that improving the model accuracy generally improves image quality, as the measured data is used more efficiently. For example, modeling focal-spot blur, scintillator blur, and noise iicorrelations enables more accurate trabecular bone visualization and trabecular thickness calculation as compared to methods that ignore blur or model blur but ignore noise correlations. Additionally, MBIR with advanced modeling typically outperforms traditional methods, either with more accurate reconstructions or by including physical effects that cannot otherwise be modeled, such as shift-variant focal-spot blur. This work provides a means to produce high-quality and high-resolution CT reconstructions for a wide variety of systems with different hardware and geometries, providing new tradeoffs in system design, enabling new applications in CT, and ultimately improving patient care