Real-time 3D reconstruction of non-rigid shapes with a single moving camera

© . This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/This paper describes a real-time sequential method to simultaneously recover the camera motion and the 3D shape of deformable objects from a calibrated monocular video. For this purpose, we consider the Navier-Cauchy equations used in 3D linear elasticity and solved by finite elements, to model the time-varying shape per frame. These equations are embedded in an extended Kalman filter, resulting in sequential Bayesian estimation approach. We represent the shape, with unknown material properties, as a combination of elastic elements whose nodal points correspond to salient points in the image. The global rigidity of the shape is encoded by a stiffness matrix, computed after assembling each of these elements. With this piecewise model, we can linearly relate the 3D displacements with the 3D acting forces that cause the object deformation, assumed to be normally distributed. While standard finite-element-method techniques require imposing boundary conditions to solve the resulting linear system, in this work we eliminate this requirement by modeling the compliance matrix with a generalized pseudoinverse that enforces a pre-fixed rank. Our framework also ensures surface continuity without the need for a post-processing step to stitch all the piecewise reconstructions into a global smooth shape. We present experimental results using both synthetic and real videos for different scenarios ranging from isometric to elastic deformations. We also show the consistency of the estimation with respect to 3D ground truth data, include several experiments assessing robustness against artifacts and finally, provide an experimental validation of our performance in real time at frame rate for small mapsPeer ReviewedPostprint (author's final draft

A non-invasive image based system for early diagnosis of prostate cancer.

Prostate cancer is the second most fatal cancer experienced by American males. The average American male has a 16.15% chance of developing prostate cancer, which is 8.38% higher than lung cancer, the second most likely cancer. The current in-vitro techniques that are based on analyzing a patients blood and urine have several limitations concerning their accuracy. In addition, the prostate Specific Antigen (PSA) blood-based test, has a high chance of false positive diagnosis, ranging from 28%-58%. Yet, biopsy remains the gold standard for the assessment of prostate cancer, but only as the last resort because of its invasive nature, high cost, and potential morbidity rates. The major limitation of the relatively small needle biopsy samples is the higher possibility of producing false positive diagnosis. Moreover, the visual inspection system (e.g., Gleason grading system) is not quantitative technique and different observers may classify a sample differently, leading to discrepancies in the diagnosis. As reported in the literature that the early detection of prostate cancer is a crucial step for decreasing prostate cancer related deaths. Thus, there is an urgent need for developing objective, non-invasive image based technology for early detection of prostate cancer. The objective of this dissertation is to develop a computer vision methodology, later translated into a clinically usable software tool, which can improve sensitivity and specificity of early prostate cancer diagnosis based on the well-known hypothesis that malignant tumors are will connected with the blood vessels than the benign tumors. Therefore, using either Diffusion Weighted Magnetic Resonance imaging (DW-MRI) or Dynamic Contrast Enhanced Magnetic Resonance Imaging (DCE-MRI), we will be able to interrelate the amount of blood in the detected prostate tumors by estimating either the Apparent Diffusion Coefficient (ADC) in the prostate with the malignancy of the prostate tumor or perfusion parameters. We intend to validate this hypothesis by demonstrating that automatic segmentation of the prostate from either DW-MRI or DCE-MRI after handling its local motion, provides discriminatory features for early prostate cancer diagnosis. The proposed CAD system consists of three majors components, the first two of which constitute new research contributions to a challenging computer vision problem. The three main components are: (1) A novel Shape-based segmentation approach to segment the prostate from either low contrast DW-MRI or DCE-MRI data; (2) A novel iso-contours-based non-rigid registration approach to ensure that we have voxel-on-voxel matches of all data which may be more difficult due to gross patient motion, transmitted respiratory effects, and intrinsic and transmitted pulsatile effects; and (3) Probabilistic models for the estimated diffusion and perfusion features for both malignant and benign tumors. Our results showed a 98% classification accuracy using Leave-One-Subject-Out (LOSO) approach based on the estimated ADC for 30 patients (12 patients diagnosed as malignant; 18 diagnosed as benign). These results show the promise of the proposed image-based diagnostic technique as a supplement to current technologies for diagnosing prostate cancer

Current and Future Trends in Magnetic Resonance Imaging Assessments of the Response of Breast Tumors to Neoadjuvant Chemotherapy

The current state-of-the-art assessment of treatment response in breast cancer is based on the response evaluation criteria in solid tumors (RECIST). RECIST reports on changes in gross morphology and divides response into one of four categories. In this paper we highlight how dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) and diffusion-weighted MRI (DW-MRI) may be able to offer earlier, and more precise, information on treatment response in the neoadjuvant setting than RECIST. We then describe how longitudinal registration of breast images and the incorporation of intelligent bioinformatics approaches with imaging data have the potential to increase the sensitivity of assessing treatment response. We conclude with a discussion of the potential benefits of breast MRI at the higher field strength of 3T. For each of these areas, we provide a review, illustrative examples from clinical trials, and offer insights into future research directions

Cardiac magnetic resonance T1 and extracellular volume mapping with motion correction and co-registration based on fast elastic image registration

OBJECTIVE: Our aim was to investigate the technical feasibility of a novel motion compensation method for cardiac magntic resonance (MR) T1 and extracellular volume fraction (ECV) mapping. MATERIALS AND METHODS: Native and post-contrast T1 maps were obtained using modified look-locker inversion recovery (MOLLI) pulse sequences with acquisition scheme defined in seconds. A nonrigid, nonparametric, fast elastic registration method was applied to generate motion-corrected T1 maps and subsequently ECV maps. Qualitative rating was performed based on T1 fitting-error maps and overlay images. Local deformation vector fields were produced for quantitative assessment. Intra- and inter-observer reproducibility were compared with and without motion compensation. RESULTS: Eighty-two T1 and 39 ECV maps were obtained in 21 patients with diverse myocardial diseases. Approximately 60% demonstrated clear quality improvement after motion correction for T1 mapping, particularly for the poor-rating cases (23% before vs 2% after). Approximately 67% showed further improvement with co-registration in ECV mapping. Although T1 and ECV values were not clinically significantly different before and after motion compensation, there was improved intra- and inter-observer reproducibility after motion compensation. CONCLUSIONS: Automated motion correction and co-registration improved the qualitative assessment and reproducibility of cardiac MR T1 and ECV measurements, allowing for more reliable ECV mapping

Phenomenological model of diffuse global and regional atrophy using finite-element methods

The main goal of this work is the generation of ground-truth data for the validation of atrophy measurement techniques, commonly used in the study of neurodegenerative diseases such as dementia. Several techniques have been used to measure atrophy in cross-sectional and longitudinal studies, but it is extremely difficult to compare their performance since they have been applied to different patient populations. Furthermore, assessment of performance based on phantom measurements or simple scaled images overestimates these techniques' ability to capture the complexity of neurodegeneration of the human brain. We propose a method for atrophy simulation in structural magnetic resonance (MR) images based on finite-element methods. The method produces cohorts of brain images with known change that is physically and clinically plausible, providing data for objective evaluation of atrophy measurement techniques. Atrophy is simulated in different tissue compartments or in different neuroanatomical structures with a phenomenological model. This model of diffuse global and regional atrophy is based on volumetric measurements such as the brain or the hippocampus, from patients with known disease and guided by clinical knowledge of the relative pathological involvement of regions and tissues. The consequent biomechanical readjustment of structures is modelled using conventional physics-based techniques based on biomechanical tissue properties and simulating plausible tissue deformations with finite-element methods. A thermoelastic model of tissue deformation is employed, controlling the rate of progression of atrophy by means of a set of thermal coefficients, each one corresponding to a different type of tissue. Tissue characterization is performed by means of the meshing of a labelled brain atlas, creating a reference volumetric mesh that will be introduced to a finite-element solver to create the simulated deformations. Preliminary work on the simulation of acquisition artefa- - cts is also presented. Cross-sectional and

Effective Volumetric Feature Modeling and Coarse Correspondence via Improved 3DSIFT and Spectral Matching

This paper presents a nonrigid coarse correspondence computation algorithm for volumetric images. Our matching algorithm first extracts then correlates image features based on a revised and improved 3DSIFT (I3DSIFT) algorithm. With a scale-related keypoint reorientation and descriptor construction, this feature correlation is less sensitive to image rotation and scaling. Then, we present an improved spectral matching (ISM) algorithm on correlated features to obtain a one-to-one mapping between corresponded features. One can effectively extend this feature correspondence to dense correspondence between volume images. Our algorithm can benefit nonrigid volumetric image registration in many tasks such as motion modeling in medical image analysis and processing

Analysis of contrast-enhanced medical images.

Early detection of human organ diseases is of great importance for the accurate diagnosis and institution of appropriate therapies. This can potentially prevent progression to end-stage disease by detecting precursors that evaluate organ functionality. In addition, it also assists the clinicians for therapy evaluation, tracking diseases progression, and surgery operations. Advances in functional and contrast-enhanced (CE) medical images enabled accurate noninvasive evaluation of organ functionality due to their ability to provide superior anatomical and functional information about the tissue-of-interest. The main objective of this dissertation is to develop a computer-aided diagnostic (CAD) system for analyzing complex data from CE magnetic resonance imaging (MRI). The developed CAD system has been tested in three case studies: (i) early detection of acute renal transplant rejection, (ii) evaluation of myocardial perfusion in patients with ischemic heart disease after heart attack; and (iii), early detection of prostate cancer. However, developing a noninvasive CAD system for the analysis of CE medical images is subject to multiple challenges, including, but are not limited to, image noise and inhomogeneity, nonlinear signal intensity changes of the images over the time course of data acquisition, appearances and shape changes (deformations) of the organ-of-interest during data acquisition, determination of the best features (indexes) that describe the perfusion of a contrast agent (CA) into the tissue. To address these challenges, this dissertation focuses on building new mathematical models and learning techniques that facilitate accurate analysis of CAs perfusion in living organs and include: (i) accurate mathematical models for the segmentation of the object-of-interest, which integrate object shape and appearance features in terms of pixel/voxel-wise image intensities and their spatial interactions; (ii) motion correction techniques that combine both global and local models, which exploit geometric features, rather than image intensities to avoid problems associated with nonlinear intensity variations of the CE images; (iii) fusion of multiple features using the genetic algorithm. The proposed techniques have been integrated into CAD systems that have been tested in, but not limited to, three clinical studies. First, a noninvasive CAD system is proposed for the early and accurate diagnosis of acute renal transplant rejection using dynamic contrast-enhanced MRI (DCE-MRI). Acute rejection–the immunological response of the human immune system to a foreign kidney–is the most sever cause of renal dysfunction among other diagnostic possibilities, including acute tubular necrosis and immune drug toxicity. In the U.S., approximately 17,736 renal transplants are performed annually, and given the limited number of donors, transplanted kidney salvage is an important medical concern. Thus far, biopsy remains the gold standard for the assessment of renal transplant dysfunction, but only as the last resort because of its invasive nature, high cost, and potential morbidity rates. The diagnostic results of the proposed CAD system, based on the analysis of 50 independent in-vivo cases were 96% with a 95% confidence interval. These results clearly demonstrate the promise of the proposed image-based diagnostic CAD system as a supplement to the current technologies, such as nuclear imaging and ultrasonography, to determine the type of kidney dysfunction. Second, a comprehensive CAD system is developed for the characterization of myocardial perfusion and clinical status in heart failure and novel myoregeneration therapy using cardiac first-pass MRI (FP-MRI). Heart failure is considered the most important cause of morbidity and mortality in cardiovascular disease, which affects approximately 6 million U.S. patients annually. Ischemic heart disease is considered the most common underlying cause of heart failure. Therefore, the detection of the heart failure in its earliest forms is essential to prevent its relentless progression to premature death. While current medical studies focus on detecting pathological tissue and assessing contractile function of the diseased heart, this dissertation address the key issue of the effects of the myoregeneration therapy on the associated blood nutrient supply. Quantitative and qualitative assessment in a cohort of 24 perfusion data sets demonstrated the ability of the proposed framework to reveal regional perfusion improvements with therapy, and transmural perfusion differences across the myocardial wall; thus, it can aid in follow-up on treatment for patients undergoing the myoregeneration therapy. Finally, an image-based CAD system for early detection of prostate cancer using DCE-MRI is introduced. Prostate cancer is the most frequently diagnosed malignancy among men and remains the second leading cause of cancer-related death in the USA with more than 238,000 new cases and a mortality rate of about 30,000 in 2013. Therefore, early diagnosis of prostate cancer can improve the effectiveness of treatment and increase the patient’s chance of survival. Currently, needle biopsy is the gold standard for the diagnosis of prostate cancer. However, it is an invasive procedure with high costs and potential morbidity rates. Additionally, it has a higher possibility of producing false positive diagnosis due to relatively small needle biopsy samples. Application of the proposed CAD yield promising results in a cohort of 30 patients that would, in the near future, represent a supplement of the current technologies to determine prostate cancer type. The developed techniques have been compared to the state-of-the-art methods and demonstrated higher accuracy as shown in this dissertation. The proposed models (higher-order spatial interaction models, shape models, motion correction models, and perfusion analysis models) can be used in many of today’s CAD applications for early detection of a variety of diseases and medical conditions, and are expected to notably amplify the accuracy of CAD decisions based on the automated analysis of CE images

Organ-focused mutual information for nonrigid multimodal registration of liver CT and Gd–EOB–DTPA-enhanced MRI

Accurate detection of liver lesions is of great importance in hepatic surgery planning. Recent studies have shown that the detection rate of liver lesions is significantly higher in gadoxetic acid-enhanced magnetic resonance imaging (Gd–EOB–DTPA-enhanced MRI) than in contrast-enhanced portal-phase computed tomography (CT); however, the latter remains essential because of its high specificity, good performance in estimating liver volumes and better vessel visibility. To characterize liver lesions using both the above image modalities, we propose a multimodal nonrigid registration framework using organ-focused mutual information (OF-MI). This proposal tries to improve mutual information (MI) based registration by adding spatial information, benefiting from the availability of expert liver segmentation in clinical protocols. The incorporation of an additional information channel containing liver segmentation information was studied. A dataset of real clinical images and simulated images was used in the validation process. A Gd–EOB–DTPA-enhanced MRI simulation framework is presented. To evaluate results, warping index errors were calculated for the simulated data, and landmark-based and surface-based errors were calculated for the real data. An improvement of the registration accuracy for OF-MI as compared with MI was found for both simulated and real datasets. Statistical significance of the difference was tested and confirmed in the simulated dataset (p < 0.01)