A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning

Progress in biomechanical modelling of human soft tissue is the basis for the development of new clinical applications capable of improving the diagnosis and treatment of some diseases (e.g. cancer), as well as the surgical planning and guidance of some interventions. The finite element method (FEM) is one of the most popular techniques used to predict the deformation of the human soft tissue due to its high accuracy. However, FEM has an associated high computational cost, which makes it difficult its integration in real-time computer-aided surgery systems. An alternative for simulating the mechanical behaviour of human organs in real time comes from the use of machine learning (ML) techniques, which are much faster than FEM. This paper assesses the feasibility of ML methods for modelling the biomechanical behaviour of the human liver during the breathing process, which is crucial for guiding surgeons during interventions where it is critical to track this deformation (e.g. some specific kind of biopsies) or for the accurate application of radiotherapy dose to liver tumours. For this purpose, different ML regression models were investigated, including three tree-based methods (decision trees, random forests and extremely randomised trees) and other two simpler regression techniques (dummy model and linear regression). In order to build and validate the ML models, a labelled data set was constructed from modelling the deformation of eight ex-vivo human livers using FEM. The best prediction performance was obtained using extremely randomised trees, with a mean error of 0.07 mm and all the samples with an error under 1 mm. The achieved results lay the foundation for the future development of some real-time software capable of simulating the human liver deformation during the breathing process during clinical interventions.This work has been funded by the Spanish Ministry of Economy and Competitiveness (MINECO) through research projects TIN2014-52033-R and DPI2013-40859-R, both also supported by European FEDER funds. The authors acknowledge the kind collaboration of the personnel from the hospital involved in the research.Lorente, D.; Martínez-Martínez, F.; Rupérez Moreno, MJ.; Lago, MA.; Martínez-Sober, M.; Escandell-Montero, P.; Martínez-Martínez, JM.... (2017). A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning. Expert Systems with Applications. 71:342-357. doi:10.1016/j.eswa.2016.11.037S3423577

Real-time biomechanical modeling of the liver using Machine Learning models trained on Finite Element Method simulations

[EN] The development of accurate real-time models of the biomechanical behavior of different organs and tissues still poses a challenge in the field of biomechanical engineering. In the case of the liver, specifically, such a model would constitute a great leap forward in the implementation of complex applications such as surgical simulators, computed-assisted surgery or guided tumor irradiation. In this work, a relatively novel approach for developing such a model is presented. It consists in the use of a machine learning algorithm, which provides real-time inference, trained on tens of thousands of simulations of the biomechanical behavior of the liver carried out by the finite element method on more than 100 different liver geometries. Considering a target accuracy threshold of 3 mm for the Euclidean Error, four different scenarios were modeled and assessed: a single liver with an arbitrary force applied (99.96% of samples within the accepted error range), a single liver with two simultaneous forces applied (99.84% samples in range), a single liver with different material properties and an arbitrary force applied (98.46% samples in range), and a much more general model capable of modeling the behavior of any liver with an arbitrary force applied (99.01% samples in range for the median liver). The results show that the Machine Learning models perform extremely well on all the scenarios, managing to keep the Mean Euclidean Error under 1 mm in all cases. Furthermore, the proposed model achieves working frequencies above 100Hz on modest hardware (with frequencies above 1000Hz being easily achievable on more powerful GPUs) thus fulfilling the real-time requirements. These results constitute a remarkable improvement in this field and may involve a prompt implementation in clinical practice.This work has been funded by the Spanish Ministry of Economy and Competitiveness (MINECO) through research projects TIN2014-52033-R, also supported by European FEDER funds.Pellicer-Valero, OJ.; Rupérez Moreno, MJ.; Martinez-Sanchis, S.; Martín-Guerrero, JD. (2020). Real-time biomechanical modeling of the liver using Machine Learning models trained on Finite Element Method simulations. Expert Systems with Applications. 143:1-12. https://doi.org/10.1016/j.eswa.2019.113083S112143Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., & Zheng, X. (2016). TensorFlow: A system for large-scale machine learning. arXiv:1605.08695.Brunon, A., Bruyère-Garnier, K., & Coret, M. (2010). Mechanical characterization of liver capsule through uniaxial quasi-static tensile tests until failure. Journal of Biomechanics, 43(11), 2221-2227. doi:10.1016/j.jbiomech.2010.03.038Chinesta, F., Leygue, A., Bordeu, F., Aguado, J. V., Cueto, E., Gonzalez, D., … Huerta, A. (2013). PGD-Based Computational Vademecum for Efficient Design, Optimization and Control. Archives of Computational Methods in Engineering, 20(1), 31-59. doi:10.1007/s11831-013-9080-xClifford, M. A., Banovac, F., Levy, E., & Cleary, K. (2002). Assessment of Hepatic Motion Secondary to Respiration for Computer Assisted Interventions. Computer Aided Surgery, 7(5), 291-299. doi:10.3109/10929080209146038Cotin, S., Delingette, H., & Ayache, N. (2000). A hybrid elastic model for real-time cutting, deformations, and force feedback for surgery training and simulation. The Visual Computer, 16(8), 437-452. doi:10.1007/pl00007215Duysak, A., Zhang, J. J., & Ilankovan, V. (2003). Efficient modelling and simulation of soft tissue deformation using mass-spring systems. International Congress Series, 1256, 337-342. doi:10.1016/s0531-5131(03)00423-0Fung, Y. C., & Skalak, R. (1981). Biomechanics: Mechanical Properties of Living Tissues. Journal of Biomechanical Engineering, 103(4), 231-298. doi:10.1115/1.3138285González, D., Aguado, J. V., Cueto, E., Abisset-Chavanne, E., & Chinesta, F. (2016). kPCA-Based Parametric Solutions Within the PGD Framework. Archives of Computational Methods in Engineering, 25(1), 69-86. doi:10.1007/s11831-016-9173-4González, D., Cueto, E., & Chinesta, F. (2015). Computational Patient Avatars for Surgery Planning. Annals of Biomedical Engineering, 44(1), 35-45. doi:10.1007/s10439-015-1362-zJahya, A., Herink, M., & Misra, S. (2013). A framework for predicting three-dimensional prostate deformation in real time. The International Journal of Medical Robotics and Computer Assisted Surgery, 9(4), e52-e60. doi:10.1002/rcs.1493Lister, K., Gao, Z., & Desai, J. P. (2010). Development of In Vivo Constitutive Models for Liver: Application to Surgical Simulation. Annals of Biomedical Engineering, 39(3), 1060-1073. doi:10.1007/s10439-010-0227-8Lorente, D., Martínez-Martínez, F., Rupérez, M. J., Lago, M. A., Martínez-Sober, M., Escandell-Montero, P., … Martín-Guerrero, J. D. (2017). A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning. Expert Systems with Applications, 71, 342-357. doi:10.1016/j.eswa.2016.11.037Maas, S. A., Ellis, B. J., Ateshian, G. A., & Weiss, J. A. (2012). FEBio: Finite Elements for Biomechanics. Journal of Biomechanical Engineering, 134(1). doi:10.1115/1.4005694Myronenko, A., & Xubo Song. (2010). Point Set Registration: Coherent Point Drift. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(12), 2262-2275. doi:10.1109/tpami.2010.46Niroomandi, S., Alfaro, I., Cueto, E., & Chinesta, F. (2012). Accounting for large deformations in real-time simulations of soft tissues based on reduced-order models. Computer Methods and Programs in Biomedicine, 105(1), 1-12. doi:10.1016/j.cmpb.2010.06.012Plantefève, R., Peterlik, I., Haouchine, N., & Cotin, S. (2015). Patient-Specific Biomechanical Modeling for Guidance During Minimally-Invasive Hepatic Surgery. Annals of Biomedical Engineering, 44(1), 139-153. doi:10.1007/s10439-015-1419-zLarge elastic deformations of isotropic materials. I. Fundamental concepts. (1948). Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 240(822), 459-490. doi:10.1098/rsta.1948.0002Large elastic deformations of isotropic materials IV. further developments of the general theory. (1948). Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 241(835), 379-397. doi:10.1098/rsta.1948.0024Ruder, S. (2016). An overview of gradient descent optimization algorithms. (pp. 1–14). arXiv: 1609.04747.Untaroiu, C. D., & Lu, Y.-C. (2013). Material characterization of liver parenchyma using specimen-specific finite element models. Journal of the Mechanical Behavior of Biomedical Materials, 26, 11-22. doi:10.1016/j.jmbbm.2013.05.013Valanis, K. C., & Landel, R. F. (1967). The Strain‐Energy Function of a Hyperelastic Material in Terms of the Extension Ratios. Journal of Applied Physics, 38(7), 2997-3002. doi:10.1063/1.171003

Simulation of hyperelastic materials in real-time using Deep Learning

The finite element method (FEM) is among the most commonly used numerical methods for solving engineering problems. Due to its computational cost, various ideas have been introduced to reduce computation times, such as domain decomposition, parallel computing, adaptive meshing, and model order reduction. In this paper we present U-Mesh: a data-driven method based on a U-Net architecture that approximates the non-linear relation between a contact force and the displacement field computed by a FEM algorithm. We show that deep learning, one of the latest machine learning methods based on artificial neural networks, can enhance computational mechanics through its ability to encode highly non-linear models in a compact form. Our method is applied to two benchmark examples: a cantilever beam and an L-shape subject to moving punctual loads. A comparison between our method and proper orthogonal decomposition (POD) is done through the paper. The results show that U-Mesh can perform very fast simulations on various geometries, mesh resolutions and number of input forces with very small errors

Determining the Biomechanical Behavior of the Liver Using Medical Image Analysis and Evolutionary Computation

Modeling the liver deformation forms the basis for the development of new clinical applications that improve the diagnosis, planning and guidance in liver surgery. However, the patient-specific modeling of this organ and its validation are still a challenge in Biomechanics. The reason is the difficulty to measure the mechanical response of the in vivo liver tissue. The current approach consist of performing minimally invasive or open surgery aimed at estimating the elastic constant of the proposed biomechanical models. This dissertation presents how the use of medical image analysis and evolutionary computation allows the characterization of the biomechanical behavior of the liver, avoiding the use of these minimally invasive techniques. In particular, the use of similarity coefficients commonly used in medical image analysis has permitted, on one hand, to estimate the patient-specific biomechanical model of the liver avoiding the invasive measurement of its mechanical response. On the other hand, these coefficients have also permitted to validate the proposed biomechanical models. Jaccard coefficient and Hausdorff distance have been used to validate the models proposed to simulate the behavior of ex vivo lamb livers, calculating the error between the volume of the experimentally deformed samples of the livers and the volume from biomechanical simulations of these deformations. These coefficients has provided information, such as the shape of the samples and the error distribution along their volume. For this reason, both coefficients have also been used to formulate a novel function, the Geometric Similarity Function (GSF). This function has permitted to establish a methodology to estimate the elastic constants of the models proposed for the human liver using evolutionary computation. Several optimization strategies, using GSF as cost function, have been developed aimed at estimating the patient-specific elastic constants of the biomechanical models proposed for the human liver. Finally, this methodology has been used to define and validate a biomechanical model proposed for an in vitro human liver.Martínez Martínez, F. (2014). Determining the Biomechanical Behavior of the Liver Using Medical Image Analysis and Evolutionary Computation [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/39337TESI

Patient-specific simulation environment for surgical planning and preoperative rehearsal

Surgical simulation is common practice in the fields of surgical education and training. Numerous surgical simulators are available from commercial and academic organisations for the generic modelling of surgical tasks. However, a simulation platform is still yet to be found that fulfils the key requirements expected for patient-specific surgical simulation of soft tissue, with an effective translation into clinical practice. Patient-specific modelling is possible, but to date has been time-consuming, and consequently costly, because data preparation can be technically demanding. This motivated the research developed herein, which addresses the main challenges of biomechanical modelling for patient-specific surgical simulation. A novel implementation of soft tissue deformation and estimation of the patient-specific intraoperative environment is achieved using a position-based dynamics approach. This modelling approach overcomes the limitations derived from traditional physically-based approaches, by providing a simulation for patient-specific models with visual and physical accuracy, stability and real-time interaction. As a geometrically- based method, a calibration of the simulation parameters is performed and the simulation framework is successfully validated through experimental studies. The capabilities of the simulation platform are demonstrated by the integration of different surgical planning applications that are found relevant in the context of kidney cancer surgery. The simulation of pneumoperitoneum facilitates trocar placement planning and intraoperative surgical navigation. The implementation of deformable ultrasound simulation can assist surgeons in improving their scanning technique and definition of an optimal procedural strategy. Furthermore, the simulation framework has the potential to support the development and assessment of hypotheses that cannot be tested in vivo. Specifically, the evaluation of feedback modalities, as a response to user-model interaction, demonstrates improved performance and justifies the need to integrate a feedback framework in the robot-assisted surgical setting.Open Acces

A biomechanical approach for real-time tracking of lung tumors during External Beam Radiation Therapy (EBRT)

Lung cancer is the most common cause of cancer related death in both men and women. Radiation therapy is widely used for lung cancer treatment. However, this method can be challenging due to respiratory motion. Motion modeling is a popular method for respiratory motion compensation, while biomechanics-based motion models are believed to be more robust and accurate as they are based on the physics of motion. In this study, we aim to develop a biomechanics-based lung tumor tracking algorithm which can be used during External Beam Radiation Therapy (EBRT). An accelerated lung biomechanical model can be used during EBRT only if its boundary conditions (BCs) are defined in a way that they can be updated in real-time. As such, we have developed a lung finite element (FE) model in conjunction with a Neural Networks (NNs) based method for predicting the BCs of the lung model from chest surface motion data. To develop the lung FE model for tumor motion prediction, thoracic 4D CT images of lung cancer patients were processed to capture the lung and diaphragm geometry, trans-pulmonary pressure, and diaphragm motion. Next, the chest surface motion was obtained through tracking the motion of the ribcage in 4D CT images. This was performed to simulate surface motion data that can be acquired using optical tracking systems. Finally, two feedforward NNs were developed, one for estimating the trans-pulmonary pressure and another for estimating the diaphragm motion from chest surface motion data. The algorithm development consists of four steps of: 1) Automatic segmentation of the lungs and diaphragm, 2) diaphragm motion modelling using Principal Component Analysis (PCA), 3) Developing the lung FE model, and 4) Using two NNs to estimate the trans-pulmonary pressure values and diaphragm motion from chest surface motion data. The results indicate that the Dice similarity coefficient between actual and simulated tumor volumes ranges from 0.76±0.04 to 0.91±0.01, which is favorable. As such, real-time lung tumor tracking during EBRT using the proposed algorithm is feasible. Hence, further clinical studies involving lung cancer patients to assess the algorithm performance are justified

Predicting the Cancer Tumor Position in Liver Using Finite Element Analysis (FEA) and Artificial Intelligence (AI)

The computational power and advantages of the Finite Element Method (FEM) are noticeable. When dealing with high nonlinearity of the materials and geometrical complexity, FEM is a powerful solution, depending on the correct definition of the problem. The availability of this method has benefited many engineering areas. In the field of biomechanics and, more specifically, in Computer-Assisted Surgery, FEM is even more appreciated. This approach, however, comes at a high computational cost. Thus, a significant delay in the response impedes its implementation for real-time applications in clinical practices, even by using parallelization or utilizing Graphics Processing Unit (GPU). This is where an alternative approach is needed to accelerate FEM-based simulations to provide the desired outputs and minimizing the time lag, preventing using FEM during intra-operative applications. A novel technique that may help to overcome the obstacles mentioned above and improve the response time is the field of Machine Learning (ML). In particular, the Artificial Neural Network (ANN), as a subset of ML, has demonstrated high potentials in computer vision and pattern recognition, whose implementation can be extended to replace a FEM model once it has been trained with sufficient inputs. In this work, a FEM-ML framework is established to drastically increase the response time for predicting tumor and internal structures’ locations in the human liver for surgical applications by using ANN. This technique takes advantage of the FEM results to train a model capable of capturing large deformations of liver tissue during the surgical intervention while reporting back the nodal locations of the components with high accuracy and efficiency. For doing so, a biomechanical model of the liver, accounting for the effect of the stiffness of blood vessels, is developed, and multiple simulations with random nodal loads on the surface of the liver are conducted in the commercial software Abaqus to produce the input required for the ANN. The ANN then predicts the nodes’ coordinates resulting from the applied forces that can be used to reconstruct the deformed model of the organ

A surrogate model based on a finite element model of abdomen for real-time visualisation of tissue stress during physical examination training

Robotic patients show great potential to improve medical palpation training as they can provide feedback that cannot be obtained in a real patient. Providing information about internal organs deformation can significantly enhance palpation training by giving medical trainees visual insight based on their finger behaviours. This can be achieved by using computational models of abdomen mechanics. However, such models are computationally expensive, thus able to provide real-time predictions. In this work, we proposed an innovative surrogate model of abdomen mechanics using machine learning (ML) and finite element (FE) modelling to virtually render internal tissue deformation in real-time. We first developed a new high-fidelity FE model of the abdomen mechanics from computerized tomography (CT) images. We performed palpation simulations to produce a large database of stress distribution on the liver edge, an area of interest in most examinations. We then used artificial neural networks (ANN) to develop the surrogate model and demonstrated its application in an experimental palpation platform. Our FE simulations took 1.5 hrs to predict stress distribution for each palpation while this only took a fraction of a second for the surrogate model. Our results show that the ANN has a 92.6% accuracy. We also show that the surrogate model is able to use the experimental input of palpation location and force to provide real-time projections onto the robotics platform. This enhanced robotics platform has potential to be used as a training simulator for trainees to hone their palpation skills

A Composite Material-based Computational Model for Diaphragm Muscle Biomechanical Simulation

Lung cancer is the most common cause of cancer related death among both men and women. Radiation therapy is the most widely used treatment for this disease. Motion compensation for tumor movement is often clinically important and biomechanics-based motion models may provide the most robust method as they are based on the physics of motion. In this study, we aim to develop a patient specific biomechanical model that predicts the deformation field of the diaphragm muscle during respiration. The first part of the project involved developing an accurate and adaptable micro-to-macro mechanical approach for skeletal muscle tissue modelling for application in a FE solver. The next objective was to develop the FE-based mechanical model of the diaphragm muscle based on patient specific 4D-CT data. The model shows adaptability to pathologies and may have the potential to be incorporated into respiratory models for the aid in treatment and diagnosis of diseases