Real-time 3D reconstruction of non-rigid shapes with a single moving camera

© . This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/This paper describes a real-time sequential method to simultaneously recover the camera motion and the 3D shape of deformable objects from a calibrated monocular video. For this purpose, we consider the Navier-Cauchy equations used in 3D linear elasticity and solved by finite elements, to model the time-varying shape per frame. These equations are embedded in an extended Kalman filter, resulting in sequential Bayesian estimation approach. We represent the shape, with unknown material properties, as a combination of elastic elements whose nodal points correspond to salient points in the image. The global rigidity of the shape is encoded by a stiffness matrix, computed after assembling each of these elements. With this piecewise model, we can linearly relate the 3D displacements with the 3D acting forces that cause the object deformation, assumed to be normally distributed. While standard finite-element-method techniques require imposing boundary conditions to solve the resulting linear system, in this work we eliminate this requirement by modeling the compliance matrix with a generalized pseudoinverse that enforces a pre-fixed rank. Our framework also ensures surface continuity without the need for a post-processing step to stitch all the piecewise reconstructions into a global smooth shape. We present experimental results using both synthetic and real videos for different scenarios ranging from isometric to elastic deformations. We also show the consistency of the estimation with respect to 3D ground truth data, include several experiments assessing robustness against artifacts and finally, provide an experimental validation of our performance in real time at frame rate for small mapsPeer ReviewedPostprint (author's final draft

Biomechanical Modeling for Lung Tumor Motion Prediction during Brachytherapy and Radiotherapy

A novel technique is proposed to develop a biomechanical model for estimating lung’s tumor position as a function of respiration cycle time. Continuous tumor motion is a major challenge in lung cancer treatment techniques where the tumor needs to be targeted; e.g. in external beam radiotherapy and brachytherapy. If not accounted for, this motion leads to areas of radiation over and/or under dosage for normal tissue and tumors. In this thesis, biomechanical models were developed for lung tumor motion predication in two distinct cases of lung brachytherapy and lung external beam radiotherapy. The lung and other relevant surrounding organs geometries, loading, boundary conditions and mechanical properties were considered and incorporated properly for each case. While using material model with constant incompressibility is sufficient to model the lung tissue in the brachytherapy case, in external beam radiation therapy the tissue incompressibility varies significantly due to normal breathing. One of the main issues tackled in this research is characterizing lung tissue incompressibility variations and measuring its corresponding parameters as a function of respiration cycle time. Results obtained from an ex-vivo porcine deflated lung indicated feasibility and reliability of using the developed biomechanical model to predict tumor motion during brachytherapy. For external beam radiotherapy, in-silico studies indicated very significant impact of considering the lung tissue incompressibility on the accuracy of predicting tumor motion. Furthermore, ex-vivo porcine lung experiments demonstrated the capability and reliability of the proposed approach for predicting tumor motion as a function of cyclic time. As such, the proposed models have a good potential to be incorporated effectively in computer assisted lung radiotherapy treatment systems

Designing electronic properties of two-dimensional crystals through optimization of deformations

One of the enticing features common to most of the two-dimensional electronic systems that are currently at the forefront of materials science research is the ability to easily introduce a combination of planar deformations and bending in the system. Since the electronic properties are ultimately determined by the details of atomic orbital overlap, such mechanical manipulations translate into modified electronic properties. Here, we present a general-purpose optimization framework for tailoring physical properties of two-dimensional electronic systems by manipulating the state of local strain, allowing a one-step route from their design to experimental implementation. A definite example, chosen for its relevance in light of current experiments in graphene nanostructures, is the optimization of the experimental parameters that generate a prescribed spatial profile of pseudomagnetic fields in graphene. But the method is general enough to accommodate a multitude of possible experimental parameters and conditions whereby deformations can be imparted to the graphene lattice, and complies, by design, with graphene's elastic equilibrium and elastic compatibility constraints. As a result, it efficiently answers the inverse problem of determining the optimal values of a set of external or control parameters that result in a graphene deformation whose associated pseudomagnetic field profile best matches a prescribed target. The ability to address this inverse problem in an expedited way is one key step for practical implementations of the concept of two-dimensional systems with electronic properties strain-engineered to order. The general-purpose nature of this calculation strategy means that it can be easily applied to the optimization of other relevant physical quantities which directly depend on the local strain field, not just in graphene but in other two-dimensional electronic membranes.Comment: 37 pages, 9 figures. This submission contains low-resolution bitmap images; high-resolution images can be found in version 1, which is ~13.5 M

Quantitative three-dimensional elasticity imaging from quasi-static deformation: a phantom study

We present a methodology to image and quantify the shear elastic modulus of three-dimensional (3D) breast tissue volumes held in compression under conditions similar to those of a clinical mammography system. Tissue phantoms are made to mimic the ultrasonic and mechanical properties of breast tissue. Stiff lesions are created in these phantoms with size and modulus contrast values, relative to the background, that are within the range of values of clinical interest. A two-dimensional ultrasound system, scanned elevationally, is used to acquire 3D images of these phantoms as they are held in compression. From two 3D ultrasound images, acquired at different compressed states, a three-dimensional displacement vector field is measured. The measured displacement field is then used to solve an inverse problem, assuming the phantom material to be an incompressible, linear elastic solid, to recover the shear modulus distribution within the imaged volume. The reconstructed values are then compared to values measured independently by direct mechanical testing.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/65094/2/pmb9_3_019.pd

Characterization of biaxial mechanical properties of rubber and skin

textBreast cancer is one of the most frequently diagnosed cancers affecting women in the United States. An ongoing objective of many research groups is to develop a biomechanical breast model for different applications, ranging from surgical outcome predictions for patients undergoing breast reconstruction surgery, to image registration for planning plastic surgery. Achieving the goal of developing a physics based biomechanical model of the human breast requires the determination of material properties of the various tissues constituting the breast. The objective of this thesis is to develop an appropriate hybrid experimental-numerical technique to enable the calibration of material parameters of skin for different constitutive models (commonly used for skin). The quantification of the material parameters thus obtained validates the bulge test method to be used in testing soft tissue specimens like skin. A bulge test device was custom-built for this work; it consists of a pressure chamber, two digital cameras, and a syringe pump as its main components. The syringe pump provides a constant flow rate of water into the pressure chamber and results in the bulging of specimens with a diameter between 45 mm and 80 mm. Three-dimensional Digital Image Correlation technique is used to obtain full field displacement measurements of the three dimensional shape of the bulge. Tests were performed on commercial rubber sheets of different thickness and on porcine skin specimens; in these tests, the bulge shape was measured at monotonically increasing and decreasing pressure levels, as well as during cyclic loading allowing determination of the deformation and strain fields over the specimen surface. In order to extract the material properties, a hybrid experimental-numerical method was used: the experiment was modeled numerically using the finite element analysis software Abaqus, imposing the commonly used Mooney-Rivlin model for isotropic materials and the Gasser-Ogden-Holzapfel model for anisotropic materials. A comparison between the experimentally measured and numerically simulated bulge shapes was used to determine the optimized material parameters under biaxial loading conditions over a large range of stretch levels.Biomedical Engineerin

Variational Methods in Shape Space

This dissertation deals with the application of variational methods in spaces of geometric shapes. In particular, the treated topics include shape averaging, principal component analysis in shape space, computation of geodesic paths in shape space, as well as shape optimisation. Chapter 1 provides a brief overview over the employed models of shape space. Geometric shapes are identified with two- or three-dimensional, deformable objects. Deformations will be described via physical models; in particular, the objects will be interpreted as consisting of either a hyperelastic solid or a viscous liquid material. Furthermore, the description of shapes via phase fields or level sets is briefly introduced. Chapter 2 reviews different and related approaches to shape space modelling. References to related topics in image segmentation and registration are also provided. Finally, the relevant shape optimisation literature is introduced. Chapter 3 recapitulates the employed concepts from continuum mechanics and phase field modelling and states basic theoretical results needed for the later analysis. Chapter 4 addresses the computation of shape averages, based on a hyperelastic notion of shape dissimilarity: The dissimilarity between two shapes is measured as the minimum deformation energy required to deform the first into the second shape. A corresponding phase-field model is introduced, analysed, and finally implemented numerically via finite elements. A principal component analysis of shapes, which is consistent with the previously introduced average, is considered in Chapter 5. Elastic boundary stresses on the average shape are used as representatives of the input shapes in a linear vector space. On these linear representatives, a standard principal component analysis can be performed, where the employed covariance metric should be properly chosen to depend on the input shapes. Chapter 6 interprets shapes as belonging to objects made of a viscous liquid and correspondingly defines geodesic paths between shapes. The energy of a path is given as the total physical dissipation during the deformation of an object along the path. A rigid body motion invariant time discretisation is achieved by approximating the dissipation along a path segment by the deformation energy of a small solid deformation. The numerical implementation is based on level sets. Chapter 7 is concerned with the optimisation of the geometry and topology of solid structures that are subject to a mechanical load. Given the load configuration, the structure rigidity, its volume, and its surface area shall be optimally balanced. A phase field model is devised and analysed for this purpose. In this context, the use of nonlinear elasticity allows to detect buckling phenomena which would be ignored in linearised elasticity

A biomechanical approach for real-time tracking of lung tumors during External Beam Radiation Therapy (EBRT)

Lung cancer is the most common cause of cancer related death in both men and women. Radiation therapy is widely used for lung cancer treatment. However, this method can be challenging due to respiratory motion. Motion modeling is a popular method for respiratory motion compensation, while biomechanics-based motion models are believed to be more robust and accurate as they are based on the physics of motion. In this study, we aim to develop a biomechanics-based lung tumor tracking algorithm which can be used during External Beam Radiation Therapy (EBRT). An accelerated lung biomechanical model can be used during EBRT only if its boundary conditions (BCs) are defined in a way that they can be updated in real-time. As such, we have developed a lung finite element (FE) model in conjunction with a Neural Networks (NNs) based method for predicting the BCs of the lung model from chest surface motion data. To develop the lung FE model for tumor motion prediction, thoracic 4D CT images of lung cancer patients were processed to capture the lung and diaphragm geometry, trans-pulmonary pressure, and diaphragm motion. Next, the chest surface motion was obtained through tracking the motion of the ribcage in 4D CT images. This was performed to simulate surface motion data that can be acquired using optical tracking systems. Finally, two feedforward NNs were developed, one for estimating the trans-pulmonary pressure and another for estimating the diaphragm motion from chest surface motion data. The algorithm development consists of four steps of: 1) Automatic segmentation of the lungs and diaphragm, 2) diaphragm motion modelling using Principal Component Analysis (PCA), 3) Developing the lung FE model, and 4) Using two NNs to estimate the trans-pulmonary pressure values and diaphragm motion from chest surface motion data. The results indicate that the Dice similarity coefficient between actual and simulated tumor volumes ranges from 0.76±0.04 to 0.91±0.01, which is favorable. As such, real-time lung tumor tracking during EBRT using the proposed algorithm is feasible. Hence, further clinical studies involving lung cancer patients to assess the algorithm performance are justified

Comparing Regularized Kelvinlet Functions and the Finite Element Method for Registration of Medical Images to Sparse Organ Data

Image-guided surgery collocates patient-specific data with the physical environment to facilitate surgical decision making in real-time. Unfortunately, these guidance systems commonly become compromised by intraoperative soft-tissue deformations. Nonrigid image-to-physical registration methods have been proposed to compensate for these deformations, but intraoperative clinical utility requires compatibility of these techniques with data sparsity and temporal constraints in the operating room. While linear elastic finite element models are effective in sparse data scenarios, the computation time for finite element simulation remains a limitation to widespread deployment. This paper proposes a registration algorithm that uses regularized Kelvinlets, which are analytical solutions to linear elasticity in an infinite domain, to overcome these barriers. This algorithm is demonstrated and compared to finite element-based registration on two datasets: a phantom dataset representing liver deformations and an in vivo dataset representing breast deformations. The regularized Kelvinlets algorithm resulted in a significant reduction in computation time compared to the finite element method. Accuracy as evaluated by target registration error was comparable between both methods. Average target registration errors were 4.6 +/- 1.0 and 3.2 +/- 0.8 mm on the liver dataset and 5.4 +/- 1.4 and 6.4 +/- 1.5 mm on the breast dataset for the regularized Kelvinlets and finite element method models, respectively. This work demonstrates the generalizability of using a regularized Kelvinlets registration algorithm on multiple soft tissue elastic organs. This method may improve and accelerate registration for image-guided surgery applications, and it shows the potential of using regularized Kelvinlets solutions on medical imaging data.Comment: 17 pages, 9 figure

Biomechanical Modeling and Inverse Problem Based Elasticity Imaging for Prostate Cancer Diagnosis

Early detection of prostate cancer plays an important role in successful prostate cancer treatment. This requires screening the prostate periodically after the age of 50. If screening tests lead to prostate cancer suspicion, prostate needle biopsy is administered which is still considered as the clinical gold standard for prostate cancer diagnosis. Given that needle biopsy is invasive and is associated with issues including discomfort and infection, it is desirable to develop a prostate cancer diagnosis system that has high sensitivity and specificity for early detection with a potential to improve needle biopsy outcome. Given the complexity and variability of prostate cancer pathologies, many research groups have been pursuing multi-parametric imaging approach as no single modality imaging technique has proven to be adequate. While imaging additional tissue properties increases the chance of reliable prostate cancer detection and diagnosis, selecting an additional property needs to be done carefully by considering clinical acceptability and cost. Clinical acceptability entails ease with respect to both operating by the radiologist and patient comfort. In this work, effective tissue biomechanics based diagnostic techniques are proposed for prostate cancer assessment with the aim of early detection and minimizing the numbers of prostate biopsies. The techniques take advantage of the low cost, widely available and well established TRUS imaging method. The proposed techniques include novel elastography methods which were formulated based on an inverse finite element frame work. Conventional finite element analysis is known to have high computational complexity, hence computation time demanding. This renders the proposed elastography methods not suitable for real-time applications. To address this issue, an accelerated finite element method was proposed which proved to be suitable for prostate elasticity reconstruction. In this method, accurate finite element analysis of a large number of prostates undergoing TRUS probe loadings was performed. Geometry input and displacement and stress fields output obtained from the analysis were used to train a neural network mapping function to be used for elastopgraphy imaging of prostate cancer patients. The last part of the research presented in this thesis tackles an issue with the current 3D TRUS prostate needle biopsy. Current 3D TRUS prostate needle biopsy systems require registering preoperative 3D TRUS to intra-operative 2D TRUS images. Such image registration is time-consuming while its real-time implementation is yet to be developed. To bypass this registration step, concept of a robotic system was proposed which can reliably determine the preoperative TRUS probe position relative to the prostate to place at the same position relative to the prostate intra-operatively. For this purpose, a contact pressure feedback system is proposed to ensure similar prostate deformation during 3D and 2D image acquisition in order to bypass the registration step