Real-time hybrid cutting with dynamic fluid visualization for virtual surgery

It is widely accepted that a reform in medical teaching must be made to meet today's high volume training requirements. Virtual simulation offers a potential method of providing such trainings and some current medical training simulations integrate haptic and visual feedback to enhance procedure learning. The purpose of this project is to explore the capability of Virtual Reality (VR) technology to develop a training simulator for surgical cutting and bleeding in a general surgery

Real-time simulation of soft tissue deformation for surgical simulation

Surgical simulation plays an important role in the training, planning and evaluation of many surgical procedures. It requires realistic and real-time simulation of soft tissue deformation under interaction with surgical tools. However, it is challenging to satisfy both of these conflicting requirements. On one hand, biological soft tissues are complex in terms of material compositions, structural formations, and mechanical behaviours, resulting in nonlinear deformation characteristics under an external load. Due to the involvement of both material and geometric nonlinearities, the use of nonlinear elasticity causes a highly expensive computational load, leading to the difficulty to achieve the real-time computational performance required by surgical simulation. On the other hand, in order to satisfy the real-time computational requirement, most of the existing methods are mainly based on linear elasticity under the assumptions of small deformation and homogeneity to describe deformation of soft tissues. Such simplifications allow reduced runtime computation; however, they are inadequate for modelling nonlinear material properties such as anisotropy, heterogeneity and large deformation of soft tissues. In general, the two conflicting requirements of surgical simulation raise immense complexity in modelling of soft tissue deformation. This thesis focuses on establishment of new methodologies for modelling of soft tissue deformation for surgical simulation. Due to geometric and material nonlinearities in soft tissue deformation, the existing methods have only limited capabilities in achieving nonlinear soft tissue deformation in real-time. In this thesis, the main focus is devoted to the real-time and realistic modelling of nonlinear soft tissue deformation for surgical simulation. New methodologies, namely new ChainMail algorithms, energy propagation method, and energy balance method, are proposed to address soft tissue deformation. Results demonstrate that the proposed methods can simulate the typical soft tissue mechanical properties, accommodate isotropic and homogeneous, anisotropic and heterogeneous materials, handle incompressibility and viscoelastic behaviours, conserve system energy, and achieve realistic, real-time and stable deformation. In the future, it is projected to extend the proposed methodologies to handle surgical operations, such as cutting, joining and suturing, for topology changes occurred in surgical simulation

Real-time measurement corrected prediction of soft tissue response for medical simulations

Medical simulators, such as in palpation and disease diagnosis, require an efficient model of the biological soft tissue deformation. Hence, a computationally fast and accurate algorithm is required to support and enhance user interactions in near real-time simulations. The visual accuracy of such simulators is dependent on the user¿s reaction time. Static visual images that update at a rate of 25 Hz are perceived as real-time moving images. Hence, visualizing software requires fast algorithms to compute the deformation of soft tissue to facilitate a meaningful simulation. Furthermore, soft tissue behaviour should be modelled accurately while compatible with real-time computation. This work proposes a fast solver for the linearized finite element method (FEM) and validates the proposed algorithm with experimental results. The novelty of the method lies in the utilization of real-time force/displacement measurements that are embedded in the solution via the Kalman filter. A novel computational algorithm that utilizes the strength of the FEM in terms of accuracy and employs direct measurements from the manipulated tissue to overcome the slow computational process of the FEM is proposed in the first part of the thesis. As the behaviour of the mechanically loaded tissue can be regarded as linearly responding at each time step, a constant acceleration temporal discretization method, i.e., the Newmark-ß is employed. In real-time applications, the accuracy of the target variable highly depends on the accuracy of the inputs while differentiating noise from the signal is hardly ever possible. To address this problem, a Kalman filter-based method is developed. The proposed algorithm not only filters the noise from the measurements but also adapts the filter gain to the estimates of the target variable, i.e., the resulting tissue deformation. For a simulated tension test of a cubic model, the proposed algorithm achieves the update frequency of 63.3 Hz. This rate is a significant improvement in computational speed compared to the 5.8 Hz update rate by the classic FEM. Besides, this novel combination of the KF and the FEM makes it possible to expand the displacement estimates in the spatial domain when the measurements are only partially available at certain points. The performance of the above method is validated experimentally through a comparison with indentation tests on artificial human tissue-like material and with the FEM result under identical simulation conditions. The test is repeated on several samples, and the displacement variation from the FEM outcome is considered as the model error. Simulation results show that the proposed method achieves the deformation update frequency of 145.7 Hz compared to the 2.7 Hz from the reference FEM. The proposed method shows the same predictive ability, only 0.47% difference from FEM on average. Experimental validation of the proposed KF-FEM confirms that by consideration of both the measurement noise and the model error, the proposed method is capable of achieving high-frequency response without sacrificing the accuracy. Further to this, the experiments confirmed the linearized model response is reliable within the applied displacement range and therefore proving that KF can be employed. The developed KF-FEM was modified in the next study to address the problem resulting from inaccurate external loads measurements by the force sensors. In the modified version, both the external force, i.e., driving variable, and the displacement, i.e., driven variable, are taken as system states. It is considered that the uncertainty of the model input influences the accuracy of the system estimates. The modified model is calibrated to differentiate the system noise from the input noise. Numerical simulations were conducted on a liver shape geometrical model, and the simulation results demonstrate that more than 90% of the measurement noise is removed. The computational speed is also increased, delivering up to 89 Hz update rate. While the uncertainty of the external load is replicated in the displacements in an FEM solution, the developed algorithm can differentiate the measurement noise, including the displacement and external forces, from the system error, i.e., the FE model error. In the last study, the proposed model was developed to reflect the nonlinear behaviour of the manipulated tissue. The Central Difference time discretization method was used to model large deformations. A novel feature is that the Equation of motion is formulated within the element level rather than in the global spatial domain. This approach helped to improve the computational speed. Indentation with strains of slightly over 10% was simulated to assess the performance of the proposed model. The developed algorithm achieved the 33.85 Hz update frequency on a standard-issue PC and confirmed its suitability for real-time applications. Also, the proposed model achieved estimates with a maximum 5.75% mean absolute error (MAE) concerning the measurements while the classic FEM showed 6.20% MAE under identical simulation condition. Results confirm that deformation estimates for noisy boundary loads of the FEM can be improved with the help of direct measurements and yet be realistic in terms of real-time visual update. This study proposed a novel computational algorithm that achieved update frequencies of higher than 25 Hz to be perceived as real-time in human eyes. The developed KF-FEM model has also shown the potential of improving the FEM accuracy with the help of direct measurements. The proposed algorithm used partially available measurements and expanded its estimates in the spatial domain. The method was experimentally validated, and the model input uncertainty, as well as the nonlinear behaviour of the soft tissue, were assessed and verified

Haptics-based Modeling and Simulation of Micro-Implants Surgery

Ph.DDOCTOR OF PHILOSOPH

Patient-specific simulation for autonomous surgery

An Autonomous Robotic Surgical System (ARSS) has to interact with the complex anatomical environment, which is deforming and whose properties are often uncertain. Within this context, an ARSS can benefit from the availability of patient-specific simulation of the anatomy. For example, simulation can provide a safe and controlled environment for the design, test and validation of the autonomous capabilities. Moreover, it can be used to generate large amounts of patient-specific data that can be exploited to learn models and/or tasks. The aim of this Thesis is to investigate the different ways in which simulation can support an ARSS and to propose solutions to favor its employability in robotic surgery. We first address all the phases needed to create such a simulation, from model choice in the pre-operative phase based on the available knowledge to its intra-operative update to compensate for inaccurate parametrization. We propose to rely on deep neural networks trained with synthetic data both to generate a patient-specific model and to design a strategy to update model parametrization starting directly from intra-operative sensor data. Afterwards, we test how simulation can assist the ARSS, both for task learning and during task execution. We show that simulation can be used to efficiently train approaches that require multiple interactions with the environment, compensating for the riskiness to acquire data from real surgical robotic systems. Finally, we propose a modular framework for autonomous surgery that includes deliberative functions to handle real anatomical environments with uncertain parameters. The integration of a personalized simulation proves fundamental both for optimal task planning and to enhance and monitor real execution. The contributions presented in this Thesis have the potential to introduce significant step changes in the development and actual performance of autonomous robotic surgical systems, making them closer to applicability to real clinical conditions

Using High-Level Processing of Low-Level Signals to Actively Assist Surgeons with Intelligent Surgical Robots

Robotic surgical systems are increasingly used for minimally-invasive surgeries. As such, there is opportunity for these systems to fundamentally change the way surgeries are performed by becoming intelligent assistants rather than simply acting as the extension of surgeons' arms. As a step towards intelligent assistance, this thesis looks at ways to represent different aspects of robot-assisted surgery (RAS). We identify three main components: the robot, the surgeon actions, and the patient scene dynamics. Traditional learning algorithms in these domains are predominantly supervised methods. This has several drawbacks. First many of these domains are non-categorical, like how soft-tissue deforms. This makes labeling difficult. Second, surgeries vary greatly. Estimation of the robot state may be affected by how the robot is docked and cable tensions in the instruments. Estimation of the patient anatomy and its dynamics are often inaccurate, and in any case, may change throughout a surgery. To obtain the most accurate information, these aspects must be learned during the procedure. This limits the amount of labeling that could be done. On the surgeon side, different surgeons may perform the same procedure differently and the algorithm should provide personalized estimations for surgeons. All of these considerations motivated the use of self-supervised learning throughout this thesis. We first build a representation of the robot system. In particular, we looked at learning the dynamics model of the robot. We evaluate the model by using it to estimate forces. Once we can estimate forces in free space, we extend the algorithm to take into account patient-specific interactions, namely with the trocar and the cannula seal. Accounting for surgery-specific interactions is possible because our method does not require additional sensors and can be trained in less than five minutes, including time for data collection. Next, we use cross-modal training to understand surgeon actions by looking at the bottleneck layer when mapping video to kinematics. This should contain information about the latent space of surgeon-actions, while discarding some medium-specific information about either the video or the kinematics. Lastly, to understand the patient scene, we start with modeling interactions between a robot instrument and a soft-tissue phantom. Models are often inaccurate due to imprecise material parameters and boundary conditions, particularly in clinical scenarios. Therefore, we add a depth camera to observe deformations to correct the results of simulations. We also introduce a network that learns to simulate soft-tissue deformation from physics simulators in order to speed up the estimation. We demonstrate that self-supervised learning can be used for understanding each part of RAS. The representations it learns contain information about signals that are not directly measurable. The self-supervised nature of the methods presented in this thesis lends itself well to learning throughout a surgery. With such frameworks, we can overcome some of the main barriers to adopting learning methods in the operating room: the variety in surgery and the difficulty in labeling enough training data for each case

Research on real-time physics-based deformation for haptic-enabled medical simulation

This study developed a multiple effective visuo-haptic surgical engine to handle a variety of surgical manipulations in real-time. Soft tissue models are based on biomechanical experiment and continuum mechanics for greater accuracy. Such models will increase the realism of future training systems and the VR/AR/MR implementations for the operating room

Cirugía virtual fase i: simulación de tejidos blandos

El uso de simuladores quirúrgicos utilizando realidad virtual permite a los cirujanos practicar diversos tipos de procedimientos y con ello reducir riesgos en los pacientes. Estos simuladores deben estar en capacidad de leer un conjunto de imágenes de entrada (p.e. tomografías computarizadas, resonancias magnéticas), incorporar herramientas para segmentación y reconstrucción tridimensional de las estructuras anatómicas de interés. Estas herramientas también deben permitir la configuración de un modelo de tejidos y colisiones que represente sus propiedades biomecánicas y permitir la interacción usando diferentes canales sensoriales (p.e. vista utilizando dispositivos para visión estereoscópica y tacto con dispositivos hápticos) a través de un entorno virtual.Pregrad