Real-time Error Control for Surgical Simulation

Objective: To present the first real-time a posteriori error-driven adaptive finite element approach for real-time simulation and to demonstrate the method on a needle insertion problem. Methods: We use corotational elasticity and a frictional needle/tissue interaction model. The problem is solved using finite elements within SOFA. The refinement strategy relies upon a hexahedron-based finite element method, combined with a posteriori error estimation driven local $h$-refinement, for simulating soft tissue deformation. Results: We control the local and global error level in the mechanical fields (e.g. displacement or stresses) during the simulation. We show the convergence of the algorithm on academic examples, and demonstrate its practical usability on a percutaneous procedure involving needle insertion in a liver. For the latter case, we compare the force displacement curves obtained from the proposed adaptive algorithm with that obtained from a uniform refinement approach. Conclusions: Error control guarantees that a tolerable error level is not exceeded during the simulations. Local mesh refinement accelerates simulations. Significance: Our work provides a first step to discriminate between discretization error and modeling error by providing a robust quantification of discretization error during simulations.Comment: 12 pages, 16 figures, change of the title, submitted to IEEE TBM

Virtual reality surgery simulation: A survey on patient specific solution

For surgeons, the precise anatomy structure and its dynamics are important in the surgery interaction, which is critical for generating the immersive experience in VR based surgical training applications. Presently, a normal therapeutic scheme might not be able to be straightforwardly applied to a specific patient, because the diagnostic results are based on averages, which result in a rough solution. Patient Specific Modeling (PSM), using patient-specific medical image data (e.g. CT, MRI, or Ultrasound), could deliver a computational anatomical model. It provides the potential for surgeons to practice the operation procedures for a particular patient, which will improve the accuracy of diagnosis and treatment, thus enhance the prophetic ability of VR simulation framework and raise the patient care. This paper presents a general review based on existing literature of patient specific surgical simulation on data acquisition, medical image segmentation, computational mesh generation, and soft tissue real time simulation

Development of Interactive Simulator for Telepresence Robot in Surgical Applications

The purpose of this Diploma thesis is to develop and implement an interactive simulator for a surgical robot in a Telepresence application. The focus is on an incision procedure of a scalpel during an operation. The geometric deformation of the simulation is based on a Finite Element Method (FEM) model which has to cope with discontinuity due to the incision through the body. The FEM modelling can be done using e.g. FEM with remeshing method or XFEM which treats the cut in the body as a type of material discontinuity. A detailed analyse of the eXtended Finite Element Method (XFEM) and a comparison to the FEM with remeshing is made. For the implementation of simulation, a FEM remeshing method is used. A scalpel mounted to the end effector of a robot is controlled by a Haptic device and cuts through a silicone block. The deformation of the real test object are measured by a 2D scanning device and compared to the results of the simulation. The deviation of the reality and simulation were less than 1% based on the dimension of the body

GPU Implementation of extended total Lagrangian explicit (gpuXTLED) for Surgical Incision Application

An extended total Lagrangian explicit dynamic (XTLED) is presented as a potential numerical method for simulating interactive or physics-based surgical incisions of soft tissues. The simulation of surgical incision is vital to the integrity of virtual reality simulators that are used for immersive surgical training. However, most existing numerical methods either compromise on computational speed for accuracy or vice versa. This is due to the challenge of modelling nonlinear behaviour of soft tissues, incorporating incision and subsequently updating topology to account for the incision. To tackle these challenges, XTLED method which combines the extended finite element method (XFEM) using total Lagrangian formulation with explicit time integration method was developed. The algorithm was developed and deformations of 3D geometries under tension, were simulated. An attempt was made to validate the XTLED method using silicon samples with different incision configuration and a comparison was made between XTLED and FEM. Results show that XTLED could potentially be used to simulate interactive soft tissue incision. However, further quantitative verification and validation are required. In addition, numerical analyses conducted show that solutions may not be obtainable due to simulation errors. However, it is unclear whether these errors are inherent in the XTLED method or the algorithm created for the XTLED method in this thesis

Data-driven finite elements for geometry and material design

Crafting the behavior of a deformable object is difficult---whether it is a biomechanically accurate character model or a new multimaterial 3D printable design. Getting it right requires constant iteration, performed either manually or driven by an automated system. Unfortunately, Previous algorithms for accelerating three-dimensional finite element analysis of elastic objects suffer from expensive precomputation stages that rely on a priori knowledge of the object's geometry and material composition. In this paper we introduce Data-Driven Finite Elements as a solution to this problem. Given a material palette, our method constructs a metamaterial library which is reusable for subsequent simulations, regardless of object geometry and/or material composition. At runtime, we perform fast coarsening of a simulation mesh using a simple table lookup to select the appropriate metamaterial model for the coarsened elements. When the object's material distribution or geometry changes, we do not need to update the metamaterial library---we simply need to update the metamaterial assignments to the coarsened elements. An important advantage of our approach is that it is applicable to non-linear material models. This is important for designing objects that undergo finite deformation (such as those produced by multimaterial 3D printing). Our method yields speed gains of up to two orders of magnitude while maintaining good accuracy. We demonstrate the effectiveness of the method on both virtual and 3D printed examples in order to show its utility as a tool for deformable object design.National Science Foundation (U.S.) (Grant CCF-1138967)United States. Defense Advanced Research Projects Agency (N66001-12-1-4242

Real-time Error Control for Surgical Simulation

International audienceObjective: To present the first real-time a poste-riori error-driven adaptive finite element approach for real-time simulation and to demonstrate the method on a needle insertion problem. Methods: We use corotational elasticity and a frictional needle/tissue interaction model. The problem is solved using finite elements within SOFA 1. The refinement strategy relies upon a hexahedron-based finite element method, combined with a posteriori error estimation driven local h-refinement, for simulating soft tissue deformation. Results: We control the local and global error level in the mechanical fields (e.g. displacement or stresses) during the simulation. We show the convergence of the algorithm on academic examples, and demonstrate its practical usability on a percutaneous procedure involving needle insertion in a liver. For the latter case, we compare the force displacement curves obtained from the proposed adaptive algorithm with that obtained from a uniform refinement approach. Conclusions: Error control guarantees that a tolerable error level is not exceeded during the simulations. Local mesh refinement accelerates simulations. Significance: Our work provides a first step to discriminate between discretization error and modeling error by providing a robust quantification of discretization error during simulations. Index Terms—Finite element method, real-time error estimate, adaptive refinement, constraint-based interaction

Subject-specific Finite Element Models of the Human Knee for Transtibial Amputees to Analyze Tibial Cartilage Pressure for Gait, Cycling, and Elliptical Training

It is estimated that approximately 10-12% of the adult population suffers from osteoarthritis (OA), with long reaching burdens personally and socioeconomically. OA also causes mild discomfort to severe pain in those suffering from the disease. The incidence rate of OA for individuals with transtibial amputations is much than average in the tibiofemoral joint (TF). It is well understood that abnormal articular cartilage stress, whether that be magnitude or location, increases the risk of developing OA. Finite element (FE) simulations can predict stress in the TF joint, many studies throughout the years have validated the technology used for this purpose. This thesis is the first to successfully validate a procedure for creating subject-specific FE models for transtibial amputees to simulate the TF joint in gait, cycling and elliptical exercises. Maximum tibial cartilage pressure was extracted post-simulation and compared to historical data. The body weight normalized contact pressure on the tibial articular cartilage for the two amputee participants was larger in magnitude than the control participant in all but the medial compartment in cycling. Additionally, cycling exercise produced the smallest values of contact pressure with elliptical and gait producing similar max values but different areas of effect. The results from this thesis align with the body of work preceding it and further the goal of a FE model that predicts in-vivo articular cartilage stress in the TF joint. Future studies can further refine this methodology and create additional subject-specific models to allow for a statistical analysis of the observed differences to find if the results are significantly different. Refining the methodology could include investigating the full effect of the damping factor on contact pressure and exploring alternative methods of mesh generation

Study of medical image data transformation techniques and compatibility analysis for 3D printing

Various applications exist for additive manufacturing (AM) and reverse engineering (RE) within the medical sector. One of the significant challenges identified in the literature is the accuracy of 3D printed medical models compared to their original CAD models. Some studies have reported that 3D printed models are accurate, while others claim the opposite. This thesis aims to highlight the medical applications of AM and RE, study medical image reconstruction techniques into a 3D printable file format, and the deviations of a 3D printed model using RE. A case study on a human femur bone was conducted through medical imaging, 3D printing, and RE for comparative deviation analysis. In addition, another medical application of RE has been presented, which is for solid modelling. Segmentation was done using opensource software for trial and training purposes, while the experiment was done using commercial software. The femur model was 3D printed using an industrial FDM printer. Three different non-contact 3D scanners were investigated for the RE process. Post-processing of the point cloud was done in the VX Elements software environment, while mesh analysis was conducted in MeshLab. The scanning performance was measured using the VX Inspect environment and MeshLab. Both relative and absolute metrics were used to determine the deviation of the scanned models from the reference mesh. The scanners' range of deviations was approximately from -0.375 mm to 0.388 mm (range of about 0.763mm) with an average RMS of about 0.22 mm. The results showed that the mean deviation of the 3D printed model (based on 3D scanning) has an average range of about 0.46mm, with an average mean value of about 0.16 mm

Non-Rigid Liver Registration for Laparoscopy using Data-Driven Biomechanical Models

During laparoscopic liver resection, the limited access to the organ, the small field of view and lack of palpation can obstruct a surgeon’s workflow. Automatic navigation systems could use the images from preoperative volumetric organ scans to help the surgeons find their target (tumors) and risk-structures (vessels) more efficiently. This requires the preoperative data to be fused (or registered) with the intraoperative scene in order to display information at the correct intraoperative position. One key challenge in this setting is the automatic estimation of the organ’s current intra-operative deformation, which is required in order to predict the position of internal structures. Parameterizing the many patient-specific unknowns (tissue properties, boundary conditions, interactions with other tissues, direction of gravity) is very difficult. Instead, this work explores how to employ deep neural networks to solve the registration problem in a data-driven manner. To this end, convolutional neural networks are trained on synthetic data to estimate an organ’s intraoperative displacement field and thus its current deformation. To drive this estimation, visible surface cues from the intraoperative camera view must be supplied to the networks. Since reliable surface features are very difficult to find, the networks are adapted to also find correspondences between the pre- and intraoperative liver geometry automatically. This combines the search for correspondences with the biomechanical behavior estimation and allows the networks to tackle the full non-rigid registration problem in one single step. The result is a model which can quickly predict the volume deformation of a liver, given only sparse surface information. The model combines the advantages of a physically accurate biomechanical simulation with the speed and powerful feature extraction capabilities of deep neural networks. To test the method intraoperatively, a registration pipeline is developed which constructs a map of the liver and its surroundings from the laparoscopic video and then uses the neural networks to fuse the preoperative volume data into this map. The deformed organ volume can then be rendered as an overlay directly onto the laparoscopic video stream. The focus of this pipeline is to be applicable to real surgery, where everything should be quick and non-intrusive. To meet these requirements, a SLAM system is used to localize the laparoscopic camera (avoiding setup of an external tracking system), various neural networks are used to quickly interpret the scene and semi-automatic tools let the surgeons guide the system. Beyond the concrete advantages of the data-driven approach for intraoperative registration, this work also demonstrates general benefits of training a registration system preoperatively on synthetic data. The method lets the engineer decide which values need to be known explicitly and which should be estimated implicitly by the networks, which opens the door to many new possibilities.:1 Introduction 1.1 Motivation 1.1.1 Navigated Liver Surgery 1.1.2 Laparoscopic Liver Registration 1.2 Challenges in Laparoscopic Liver Registration 1.2.1 Preoperative Model 1.2.2 Intraoperative Data 1.2.3 Fusion/Registration 1.2.4 Data 1.3 Scope and Goals of this Work 1.3.1 Data-Driven, Biomechanical Model 1.3.2 Data-Driven Non-Rigid Registration 1.3.3 Building a Working Prototype 2 State of the Art 2.1 Rigid Registration 2.2 Non-Rigid Liver Registration 2.3 Neural Networks for Simulation and Registration 3 Theoretical Background 3.1 Liver 3.2 Laparoscopic Liver Resection 3.2.1 Staging Procedure 3.3 Biomechanical Simulation 3.3.1 Physical Balance Principles 3.3.2 Material Models 3.3.3 Numerical Solver: The Finite Element Method (FEM) 3.3.4 The Lagrangian Specification 3.4 Variables and Data in Liver Registration 3.4.1 Observable 3.4.2 Unknowns 4 Generating Simulations of Deforming Organs 4.1 Organ Volume 4.2 Forces and Boundary Conditions 4.2.1 Surface Forces 4.2.2 Zero-Displacement Boundary Conditions 4.2.3 Surrounding Tissues and Ligaments 4.2.4 Gravity 4.2.5 Pressure 4.3 Simulation 4.3.1 Static Simulation 4.3.2 Dynamic Simulation 4.4 Surface Extraction 4.4.1 Partial Surface Extraction 4.4.2 Surface Noise 4.4.3 Partial Surface Displacement 4.5 Voxelization 4.5.1 Voxelizing the Liver Geometry 4.5.2 Voxelizing the Displacement Field 4.5.3 Voxelizing Boundary Conditions 4.6 Pruning Dataset - Removing Unwanted Results 4.7 Data Augmentation 5 Deep Neural Networks for Biomechanical Simulation 5.1 Training Data 5.2 Network Architecture 5.3 Loss Functions and Training 6 Deep Neural Networks for Non-Rigid Registration 6.1 Training Data 6.2 Architecture 6.3 Loss 6.4 Training 6.5 Mesh Deformation 6.6 Example Application 7 Intraoperative Prototype 7.1 Image Acquisition 7.2 Stereo Calibration 7.3 Image Rectification, Disparity- and Depth- estimation 7.4 Liver Segmentation 7.4.1 Synthetic Image Generation 7.4.2 Automatic Segmentation 7.4.3 Manual Segmentation Modifier 7.5 SLAM 7.6 Dense Reconstruction 7.7 Rigid Registration 7.8 Non-Rigid Registration 7.9 Rendering 7.10 Robotic Operating System 8 Evaluation 8.1 Evaluation Datasets 8.1.1 In-Silico 8.1.2 Phantom Torso and Liver 8.1.3 In-Vivo, Human, Breathing Motion 8.1.4 In-Vivo, Human, Laparoscopy 8.2 Metrics 8.2.1 Mean Displacement Error 8.2.2 Target Registration Error (TRE) 8.2.3 Champfer Distance 8.2.4 Volumetric Change 8.3 Evaluation of the Synthetic Training Data 8.4 Data-Driven Biomechanical Model (DDBM) 8.4.1 Amount of Intraoperative Surface 8.4.2 Dynamic Simulation 8.5 Volume to Surface Registration Network (V2S-Net) 8.5.1 Amount of Intraoperative Surface 8.5.2 Dependency on Initial Rigid Alignment 8.5.3 Registration Accuracy in Comparison to Surface Noise 8.5.4 Registration Accuracy in Comparison to Material Stiffness 8.5.5 Champfer-Distance vs. Mean Displacement Error 8.5.6 In-vivo, Human Breathing Motion 8.6 Full Intraoperative Pipeline 8.6.1 Intraoperative Reconstruction: SLAM and Intraoperative Map 8.6.2 Full Pipeline on Laparoscopic Human Data 8.7 Timing 9 Discussion 9.1 Intraoperative Model 9.2 Physical Accuracy 9.3 Limitations in Training Data 9.4 Limitations Caused by Difference in Pre- and Intraoperative Modalities 9.5 Ambiguity 9.6 Intraoperative Prototype 10 Conclusion 11 List of Publications List of Figures Bibliograph