Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations

We propose a novel approach for enhancing depth videos containing non-rigidly deforming objects. Depth sensors are capable of capturing depth maps in real-time but suffer from high noise levels and low spatial resolutions. While solutions for reconstructing 3D details in static scenes, or scenes with rigid global motions have been recently proposed, handling unconstrained non-rigid deformations in relative complex scenes remains a challenge. Our solution consists in a recursive dynamic multi-frame superresolution algorithm where the relative local 3D motions between consecutive frames are directly accounted for. We rely on the assumption that these 3D motions can be decoupled into lateral motions and radial displacements. This allows to perform a simple local per-pixel tracking where both depth measurements and deformations are dynamically optimized. The geometric smoothness is subsequently added using a multi-level L1 minimization with a bilateral total variation regularization. The performance of this method is thoroughly evaluated on both real and synthetic data. As compared to alternative approaches, the results show a clear improvement in reconstruction accuracy and in robustness to noise, to relative large non-rigid deformations, and to topological changes. Moreover, the proposed approach, implemented on a CPU, is shown to be computationally efficient and working in real-time

Bio-oriented Micro- and Nano- Structures Based on Stimuli-responsive Polymers

Nowadays, the ability to pattern surfaces on the micro- and nano- scale is the basis for a wide range of research fields. Over last few decades, a lot of processing technologies offer the possibility to fabricate complex 2D and 3D polymeric designs which are mostly static in nature since they cannot be physically and chemically modified once fabricated. The aim of the present thesis is to overcome such a limitation, exploiting stimuli-responsive materials (Chapter I). We allow to engineer polymeric architectures adding interesting functionalities, by providing an active manipulation of pre-structured systems, which could be helpfully in a wide variety of applications, such as biosensing and cell conditioning. In the first part of the present dissertation (Chapter II), a thermos-sensitive material is employed. We investigate the thermo-responsive behavior of Poly(N-isopropylacrylamide) (pNIPAAm)-based crosslinkable hydrogel as active binding matrix in optical biosensors. In this study, we propose an extension of surface plasmon resonance (SPR) and optical waveguide mode (OWS) spectroscopy, for in situ observation of nano-patterned hydrogel film that are allowed to swell and collapse by varying the external temperature of the aqueous environment. Weak refractive index contrast of hydrogel structures arranged in periodic pattern, is generally associated with intrinsically low diffraction efficiency. In order to enhance the intensity of diffracted light, the surface is probed by resonantly excited optical waveguide modes, taking advantage of the fact that the hydrogel can serve as optical waveguide (HOW) enabling the excitation of additional modes besides surface plasmons. Thus, we provide a hydrogel optical waveguide-enhanced diffraction measurements, taking advantage of strong electromagnetic field intensity enhancements that amplifies the weak diffracted light intensity. The main part of the thesis is focused in the study of azopolymer-containing materials, a specific class of light-responsive materials. Upon photon absorption, azobenzene undergo reversible trans-cis photoisomerization, which induces a substantial geometrical change of its molecular structure, that can be translated into larger-scale movements of the material below the glass transition temperature (Tg) of the polymer. In Chapter III, by exploiting the light-induced mass migration phenomenon, we demonstrate that an azopolymeric film patterned by soft imprinting technique, can be anisotropically deformed and consequent restored in its initial shape via single irradiation just by controlling the polarization state of the incident laser beam. We also propose that the light-driven morphological manipulation can induce anisotropic wettability changes. Lastly, a polarization driven birefringence effect on flat and structured surfaces is discussed. Chapter IV focuses in the design of novel azopolymeric systems, where the optical response is provided by azobenzene molecules, which doped two different host materials. The photo-responsive behavior and potential applications of azo compounds incorporated into either a soft elastomeric and in rigid matrix is discussed. Azo-embedded poly(dimethylsiloxane) (PDMS) is studied as tunable optical lens and an azo-doped photocurable commercial polymeric resin is developed to study the photo-mechanical transduction of a 3D suspended membrane fabricated by two photon lithography technique. In Chapter V, we propose a light-deformable azopolymeric micro-pillars patterned substrate as a biocompatible and “smart” platform for dynamic material-cell observation in 2D environment, modified by a holographic optical conditioning. The aim is to observe by time-lapse acquisitions, how an in situ deformation of a pre-patterned structure can influence cell functions and fate. Finally, in Chapter VI, general remarks of the present work are discussed, and directions for future perspective are summarized

Super-Resolution Approaches for Depth Video Enhancement

Sensing using 3D technologies has seen a revolution in the past years where cost-effective depth sensors are today part of accessible consumer electronics. Their ability in directly capturing depth videos in real-time has opened tremendous possibilities for multiple applications in computer vision. These sensors, however, have major shortcomings due to their high noise contamination, including missing and jagged measurements, and their low spatial resolutions. In order to extract detailed 3D features from this type of data, a dedicated data enhancement is required. We propose a generic depth multi–frame super–resolution framework that addresses the limitations of state-of-theart depth enhancement approaches. The proposed framework doesnot need any additional hardware or coupling with different modalities. It is based on a new data model that uses densely upsampled low resolution observations. This results in a robust median initial estimation, further refined by a deblurring operation using a bilateraltotal variation as the regularization term. The upsampling operation ensures a systematic improvement in the registration accuracy. This is explored in different scenarios based on the motions involved in the depth video. For the general and most challenging case of objects deforming non-rigidly in full 3D, we propose a recursive dynamic multi–frame super-resolution algorithm where the relative local 3D motions between consecutive frames are directly accounted for. We rely on the assumption that these 3D motions can be decoupled into lateral motions and radial displacements. This allows to perform a simple local per–pixel tracking where both depth measurements and deformations are optimized. As compared to alternative approaches, the results show a clear improvement in reconstruction accuracy and in robustness to noise, to relative large non-rigid deformations, and to topological changes. Moreover, the proposed approach, implemented on a CPU, is shown to be computationally efficient and working in real-time

Studying Nuclear Deformation During Migration of Cells through Confining Environments

Fibroblasts, immune cells, and cancer cells migrating through pores and fibers in tissue must deform their cell nucleus in order to move. The role of the nuclear protein lamin A/C in limiting nuclear deformation and passage has been studied in pores and microchannels with a smooth contiguous surface. However, cells such as fibroblasts and cancer cells also encounter slender extracellular matrix fibers as they migrate through interstitial tissue. To study this process, we micro-fabricated closely spaced, flexible obstacles with bending rigidity similar to collagen fibers and imaged cell migration around these obstacles. In contrast to its limiting role in nuclear passage through confining channels, lamin A/C facilitated nuclear deformation and passage through fibrous environments because nuclei in lamin-null (Lmna -/-) cells lost their overall morphology and became entangled around the obstacles. We propose that, analogous to surface tension-mediated deformation of a liquid drop, lamin A/C imparts a surface tension on the nucleus preventing nuclear entanglement and allowing nuclear passage through fibrous environments. We investigated a related long-standing question of how the cell shapes the nucleus in migrating cells. We cultured cells on surfaces with micro-patterned fibronectin such that narrow fibronectin lines (5 microns in width) culminated into wider rectangles. Nuclei in cells on 1-D lines had folds in the nuclear lamina, which disappeared as the cells migrated onto the 2-D rectangles and the nucleus reached a steady-state shape. These data confirmed the prediction of our previously proposed model that the resistance to nuclear shape changes in low so long as there are folds in the nuclear lamina. We also validated another model prediction- that the nuclear surface should move in response to local cell protrusions – through dynamic vertical imaging and tracking of protrusions and nuclear shapes in breast cancer cells. To address the errors in vertical imaging in these studies due to the well-known phenomenon of refractive index mismatch, we developed a new approach for improved imaging of the x-z cross-section of fluorescent samples. Finally, we reported the discovery that Barrier-to-autointegration Factor (BAF) mediates the repair of the nuclear envelope after nuclear envelope rupture during confined cell migration

Doctor of Philosophy

dissertationThe statistical study of anatomy is one of the primary focuses of medical image analysis. It is well-established that the appropriate mathematical settings for such analyses are Riemannian manifolds and Lie group actions. Statistically defined atlases, in which a mean anatomical image is computed from a collection of static three-dimensional (3D) scans, have become commonplace. Within the past few decades, these efforts, which constitute the field of computational anatomy, have seen great success in enabling quantitative analysis. However, most of the analysis within computational anatomy has focused on collections of static images in population studies. The recent emergence of large-scale longitudinal imaging studies and four-dimensional (4D) imaging technology presents new opportunities for studying dynamic anatomical processes such as motion, growth, and degeneration. In order to make use of this new data, it is imperative that computational anatomy be extended with methods for the statistical analysis of longitudinal and dynamic medical imaging. In this dissertation, the deformable template framework is used for the development of 4D statistical shape analysis, with applications in motion analysis for individualized medicine and the study of growth and disease progression. A new method for estimating organ motion directly from raw imaging data is introduced and tested extensively. Polynomial regression, the staple of curve regression in Euclidean spaces, is extended to the setting of Riemannian manifolds. This polynomial regression framework enables rigorous statistical analysis of longitudinal imaging data. Finally, a new diffeomorphic model of irrotational shape change is presented. This new model presents striking practical advantages over standard diffeomorphic methods, while the study of this new space promises to illuminate aspects of the structure of the diffeomorphism group

Flow pattern analysis for magnetic resonance velocity imaging

Blood flow in the heart is highly complex. Although blood flow patterns have been investigated by both computational modelling and invasive/non-invasive imaging techniques, their evolution and intrinsic connection with cardiovascular disease has yet to be explored. Magnetic resonance (MR) velocity imaging provides a comprehensive distribution of multi-directional in vivo flow distribution so that detailed quantitative analysis of flow patterns is now possible. However, direct visualisation or quantification of vector fields is of little clinical use, especially for inter-subject or serial comparison of changes in flow patterns due to the progression of the disease or in response to therapeutic measures. In order to achieve a comprehensive and integrated description of flow in health and disease, it is necessary to characterise and model both normal and abnormal flows and their effects. To accommodate the diversity of flow patterns in relation to morphological and functional changes, we have described in this thesis an approach of detecting salient topological features prior to analytical assessment of dynamical indices of the flow patterns. To improve the accuracy of quantitative analysis of the evolution of topological flow features, it is essential to restore the original flow fields so that critical points associated with salient flow features can be more reliably detected. We propose a novel framework for the restoration, abstraction, extraction and tracking of flow features such that their dynamic indices can be accurately tracked and quantified. The restoration method is formulated as a constrained optimisation problem to remove the effects of noise and to improve the consistency of the MR velocity data. A computational scheme is derived from the First Order Lagrangian Method for solving the optimisation problem. After restoration, flow abstraction is applied to partition the entire flow field into clusters, each of which is represented by a local linear expansion of its velocity components. This process not only greatly reduces the amount of data required to encode the velocity distribution but also permits an analytical representation of the flow field from which critical points associated with salient flow features can be accurately extracted. After the critical points are extracted, phase portrait theory can be applied to separate them into attracting/repelling focuses, attracting/repelling nodes, planar vortex, or saddle. In this thesis, we have focused on vortical flow features formed in diastole. To track the movement of the vortices within a cardiac cycle, a tracking algorithm based on relaxation labelling is employed. The constraints and parameters used in the tracking algorithm are designed using the characteristics of the vortices. The proposed framework is validated with both simulated and in vivo data acquired from patients with sequential MR examination following myocardial infarction. The main contribution of the thesis is in the new vector field restoration and flow feature abstraction method proposed. They allow the accurate tracking and quantification of dynamic indices associated with salient features so that inter- and intra-subject comparisons can be more easily made. This provides further insight into the evolution of blood flow patterns and permits the establishment of links between blood flow patterns and localised genesis and progression of cardiovascular disease.Open acces