Realistic tool-tissue interaction models for surgical simulation and planning

Surgical simulators present a safe and potentially effective method for surgical training, and can also be used in pre- and intra-operative surgical planning. Realistic modeling of medical interventions involving tool-tissue interactions has been considered to be a key requirement in the development of high-fidelity simulators and planners. The soft-tissue constitutive laws, organ geometry and boundary conditions imposed by the connective tissues surrounding the organ, and the shape of the surgical tool interacting with the organ are some of the factors that govern the accuracy of medical intervention planning.\ud \ud This thesis is divided into three parts. First, we compare the accuracy of linear and nonlinear constitutive laws for tissue. An important consequence of nonlinear models is the Poynting effect, in which shearing of tissue results in normal force; this effect is not seen in a linear elastic model. The magnitude of the normal force for myocardial tissue is shown to be larger than the human contact force discrimination threshold. Further, in order to investigate and quantify the role of the Poynting effect on material discrimination, we perform a multidimensional scaling study. Second, we consider the effects of organ geometry and boundary constraints in needle path planning. Using medical images and tissue mechanical properties, we develop a model of the prostate and surrounding organs. We show that, for needle procedures such as biopsy or brachytherapy, organ geometry and boundary constraints have more impact on target motion than tissue material parameters. Finally, we investigate the effects surgical tool shape on the accuracy of medical intervention planning. We consider the specific case of robotic needle steering, in which asymmetry of a bevel-tip needle results in the needle naturally bending when it is inserted into soft tissue. We present an analytical and finite element (FE) model for the loads developed at the bevel tip during needle-tissue interaction. The analytical model explains trends observed in the experiments. We incorporated physical parameters (rupture toughness and nonlinear material elasticity) into the FE model that included both contact and cohesive zone models to simulate tissue cleavage. The model shows that the tip forces are sensitive to the rupture toughness. In order to model the mechanics of deflection of the needle, we use an energy-based formulation that incorporates tissue-specific parameters such as rupture toughness, nonlinear material elasticity, and interaction stiffness, and needle geometric and material properties. Simulation results follow similar trends (deflection and radius of curvature) to those observed in macroscopic experimental studies of a robot-driven needle interacting with gels

SMART IMAGE-GUIDED NEEDLE INSERTION FOR TISSUE BIOPSY

M.S

InterNAV3D: A Navigation Tool for Robot-Assisted Needle-Based Intervention for the Lung

Lung cancer is one of the leading causes of cancer deaths in North America. There are recent advances in cancer treatment techniques that can treat cancerous tumors, but require a real-time imaging modality to provide intraoperative assistive feedback. Ultrasound (US) imaging is one such modality. However, while its application to the lungs has been limited because of the deterioration of US image quality (due to the presence of air in the lungs); recent work has shown that appropriate lung deflation can help to improve the quality sufficiently to enable intraoperative, US-guided robotics-assisted techniques to be used. The work described in this thesis focuses on this approach. The thesis describes a project undertaken at Canadian Surgical Technologies and Advanced Robotics (CSTAR) that utilizes the image processing techniques to further enhance US images and implements an advanced 3D virtual visualization software approach. The application considered is that for minimally invasive lung cancer treatment using procedures such as brachytherapy and microwave ablation while taking advantage of the accuracy and teleoperation capabilities of surgical robots, to gain higher dexterity and precise control over the therapy tools (needles and probes). A number of modules and widgets are developed and explained which improve the visibility of the physical features of interest in the treatment and help the clinician to have more reliable and accurate control of the treatment. Finally the developed tools are validated with extensive experimental evaluations and future developments are suggested to enhance the scope of the applications

3D Multimodal Interaction with Physically-based Virtual Environments

The virtual has become a huge field of exploration for researchers: it could assist the surgeon, help the prototyping of industrial objects, simulate natural phenomena, be a fantastic time machine or entertain users through games or movies. Far beyond the only visual rendering of the virtual environment, the Virtual Reality aims at -literally- immersing the user in the virtual world. VR technologies simulate digital environments with which users can interact and, as a result, perceive through different modalities the effects of their actions in real time. The challenges are huge: the user's motions need to be perceived and to have an immediate impact on the virtual world by modifying the objects in real-time. In addition, the targeted immersion of the user is not only visual: auditory or haptic feedback needs to be taken into account, merging all the sensory modalities of the user into a multimodal answer. The global objective of my research activities is to improve 3D interaction with complex virtual environments by proposing novel approaches for physically-based and multimodal interaction. I have laid the foundations of my work on designing the interactions with complex virtual worlds, referring to a higher demand in the characteristics of the virtual environments. My research could be described within three main research axes inherent to the 3D interaction loop: (1) the physically-based modeling of the virtual world to take into account the complexity of the virtual object behavior, their topology modifications as well as their interactions, (2) the multimodal feedback for combining the sensory modalities into a global answer from the virtual world to the user and (3) the design of body-based 3D interaction techniques and devices for establishing the interfaces between the user and the virtual world. All these contributions could be gathered in a general framework within the 3D interaction loop. By improving all the components of this framework, I aim at proposing approaches that could be used in future virtual reality applications but also more generally in other areas such as medical simulation, gesture training, robotics, virtual prototyping for the industry or web contents.Le virtuel est devenu un vaste champ d'exploration pour la recherche et offre de nos jours de nombreuses possibilités : assister le chirurgien, réaliser des prototypes de pièces industrielles, simuler des phénomènes naturels, remonter dans le temps ou proposer des applications ludiques aux utilisateurs au travers de jeux ou de films. Bien plus que le rendu purement visuel d'environnement virtuel, la réalité virtuelle aspire à -littéralement- immerger l'utilisateur dans le monde virtuel. L'utilisateur peut ainsi interagir avec le contenu numérique et percevoir les effets de ses actions au travers de différents retours sensoriels. Permettre une véritable immersion de l'utilisateur dans des environnements virtuels de plus en plus complexes confronte la recherche en réalité virtuelle à des défis importants: les gestes de l'utilisateur doivent être capturés puis directement transmis au monde virtuel afin de le modifier en temps-réel. Les retours sensoriels ne sont pas uniquement visuels mais doivent être combinés avec les retours auditifs ou haptiques dans une réponse globale multimodale. L'objectif principal de mes activités de recherche consiste à améliorer l'interaction 3D avec des environnements virtuels complexes en proposant de nouvelles approches utilisant la simulation physique et exploitant au mieux les différentes modalités sensorielles. Dans mes travaux, je m'intéresse tout particulièrement à concevoir des interactions avec des mondes virtuels complexes. Mon approche peut être décrite au travers de trois axes principaux de recherche: (1) la modélisation dans les mondes virtuels d'environnements physiques plausibles où les objets réagissent de manière naturelle, même lorsque leur topologie est modifiée ou lorsqu'ils sont en interaction avec d'autres objets, (2) la mise en place de retours sensoriels multimodaux vers l'utilisateur intégrant des composantes visuelles, haptiques et/ou sonores, (3) la prise en compte de l'interaction physique de l'utilisateur avec le monde virtuel dans toute sa richesse : mouvements de la tête, des deux mains, des doigts, des jambes, voire de tout le corps, en concevant de nouveaux dispositifs ou de nouvelles techniques d'interactions 3D. Les différentes contributions que j'ai proposées dans chacun de ces trois axes peuvent être regroupées au sein d'un cadre plus général englobant toute la boucle d'interaction 3D avec les environnements virtuels. Elles ouvrent des perspectives pour de futures applications en réalité virtuelle mais également plus généralement dans d'autres domaines tels que la simulation médicale, l'apprentissage de gestes, la robotique, le prototypage virtuel pour l'industrie ou bien les contenus web

3D Multimodal Interaction with Physically-based Virtual Environments

The virtual has become a huge field of exploration for researchers: it could assist the surgeon, help the prototyping of industrial objects, simulate natural phenomena, be a fantastic time machine or entertain users through games or movies. Far beyond the only visual rendering of the virtual environment, the Virtual Reality aims at -literally- immersing the user in the virtual world. VR technologies simulate digital environments with which users can interact and, as a result, perceive through different modalities the effects of their actions in real time. The challenges are huge: the user's motions need to be perceived and to have an immediate impact on the virtual world by modifying the objects in real-time. In addition, the targeted immersion of the user is not only visual: auditory or haptic feedback needs to be taken into account, merging all the sensory modalities of the user into a multimodal answer. The global objective of my research activities is to improve 3D interaction with complex virtual environments by proposing novel approaches for physically-based and multimodal interaction. I have laid the foundations of my work on designing the interactions with complex virtual worlds, referring to a higher demand in the characteristics of the virtual environments. My research could be described within three main research axes inherent to the 3D interaction loop: (1) the physically-based modeling of the virtual world to take into account the complexity of the virtual object behavior, their topology modifications as well as their interactions, (2) the multimodal feedback for combining the sensory modalities into a global answer from the virtual world to the user and (3) the design of body-based 3D interaction techniques and devices for establishing the interfaces between the user and the virtual world. All these contributions could be gathered in a general framework within the 3D interaction loop. By improving all the components of this framework, I aim at proposing approaches that could be used in future virtual reality applications but also more generally in other areas such as medical simulation, gesture training, robotics, virtual prototyping for the industry or web contents.Le virtuel est devenu un vaste champ d'exploration pour la recherche et offre de nos jours de nombreuses possibilités : assister le chirurgien, réaliser des prototypes de pièces industrielles, simuler des phénomènes naturels, remonter dans le temps ou proposer des applications ludiques aux utilisateurs au travers de jeux ou de films. Bien plus que le rendu purement visuel d'environnement virtuel, la réalité virtuelle aspire à -littéralement- immerger l'utilisateur dans le monde virtuel. L'utilisateur peut ainsi interagir avec le contenu numérique et percevoir les effets de ses actions au travers de différents retours sensoriels. Permettre une véritable immersion de l'utilisateur dans des environnements virtuels de plus en plus complexes confronte la recherche en réalité virtuelle à des défis importants: les gestes de l'utilisateur doivent être capturés puis directement transmis au monde virtuel afin de le modifier en temps-réel. Les retours sensoriels ne sont pas uniquement visuels mais doivent être combinés avec les retours auditifs ou haptiques dans une réponse globale multimodale. L'objectif principal de mes activités de recherche consiste à améliorer l'interaction 3D avec des environnements virtuels complexes en proposant de nouvelles approches utilisant la simulation physique et exploitant au mieux les différentes modalités sensorielles. Dans mes travaux, je m'intéresse tout particulièrement à concevoir des interactions avec des mondes virtuels complexes. Mon approche peut être décrite au travers de trois axes principaux de recherche: (1) la modélisation dans les mondes virtuels d'environnements physiques plausibles où les objets réagissent de manière naturelle, même lorsque leur topologie est modifiée ou lorsqu'ils sont en interaction avec d'autres objets, (2) la mise en place de retours sensoriels multimodaux vers l'utilisateur intégrant des composantes visuelles, haptiques et/ou sonores, (3) la prise en compte de l'interaction physique de l'utilisateur avec le monde virtuel dans toute sa richesse : mouvements de la tête, des deux mains, des doigts, des jambes, voire de tout le corps, en concevant de nouveaux dispositifs ou de nouvelles techniques d'interactions 3D. Les différentes contributions que j'ai proposées dans chacun de ces trois axes peuvent être regroupées au sein d'un cadre plus général englobant toute la boucle d'interaction 3D avec les environnements virtuels. Elles ouvrent des perspectives pour de futures applications en réalité virtuelle mais également plus généralement dans d'autres domaines tels que la simulation médicale, l'apprentissage de gestes, la robotique, le prototypage virtuel pour l'industrie ou bien les contenus web

Modeling and simulation of an active robotic device for flexible needle insertion

Master'sMASTER OF ENGINEERIN

Tools for improving high-dose-rate prostate cancer brachytherapy using three-dimensional ultrasound and magnetic resonance imaging

High-dose-rate brachytherapy (HDR-BT) is an interstitial technique for the treatment of intermediate and high-risk localized prostate cancer that involves placement of a radiation source directly inside the prostate using needles. Dose-escalated whole-gland treatments have led to improvements in survival, and tumour-targeted treatments may offer future improvements in therapeutic ratio. The efficacy of tumour-targeted HDR-BT depends on imaging tools to enable accurate dose delivery to prostate sub-volumes. This thesis is focused on implementing ultrasound tools to improve HDR-BT needle localization accuracy and efficiency, and evaluating dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) for tumour localization. First, we implemented a device enabling sagittally-reconstructed 3D (SR3D) ultrasound, which provides sub-millimeter resolution in the needle insertion direction. We acquired SR3D and routine clinical images in a cohort of 12 consecutive eligible HDR-BT patients, with a total of 194 needles. The SR3D technique provided needle insertion depth errors within 5 mm for 93\% of needles versus 76\% for the clinical imaging technique, leading to increased precision in dose delivered to the prostate. Second, we implemented an algorithm to automatically segment multiple HDR-BT needles in a SR3D image. The algorithm was applied to the SR3D images from the first patient cohort, demonstrating mean execution times of 11.0 s per patient and successfully segmenting 82\% of needles within 3 mm. Third, we augmented SR3D imaging with live-2D sagittal ultrasound for needle tip localization. This combined technique was applied to another cohort of 10 HDR-BT patients, reducing insertion depth errors compared to routine imaging from a range of [-8.1 mm, 7.7 mm] to [-6.2 mm, 5.9 mm]. Finally, we acquired DCE-MRI in 16 patients scheduled to undergo prostatectomy, using either high spatial resolution or high temporal resolution imaging, and compared the images to whole-mount histology. The high spatial resolution images demonstrated improved high-grade cancer classification compared to the high temporal resolution images, with areas under the receiver operating characteristic curve of 0.79 and 0.70, respectively. In conclusion, we have translated and evaluated specialized imaging tools for HDR-BT which are ready to be tested in a clinical trial investigating tumour-targeted treatment

Enabling technologies for MRI guided interventional procedures

This dissertation addresses topics related to developing interventional assistant devices for Magnetic Resonance Imaging (MRI). MRI can provide high-quality 3D visualization of target anatomy and surrounding tissue, but the benefits can not be readily harnessed for interventional procedures due to difficulties associated with the use of high-field (1.5T or greater) MRI. Discussed are potential solutions to the inability to use conventional mecha- tronics and the confined physical space in the scanner bore. This work describes the development of two apparently dissimilar systems that repre- sent different approaches to the same surgical problem - coupling information and action to perform percutaneous (through the skin) needle placement with MR imaging. The first system addressed takes MR images and projects them along with a surgical plan directly on the interventional site, thus providing in-situ imaging. With anatomical images and a corresponding plan visible in the appropriate pose, the clinician can use this information to perform the surgical action. My primary research effort has focused on a robotic assistant system that overcomes the difficulties inherent to MR-guided procedures, and promises safe and reliable intra-prostatic needle placement inside closed high-field MRI scanners. The robot is a servo pneumatically operated automatic needle guide, and effectively guides needles under real- time MR imaging. This thesis describes development of the robotic system including requirements, workspace analysis, mechanism design and optimization, and evaluation of MR compatibility. Further, a generally applicable MR-compatible robot controller is de- veloped, the pneumatic control system is implemented and evaluated, and the system is deployed in pre-clinical trials. The dissertation concludes with future work and lessons learned from this endeavor

Preoperative trajectory planning for percutaneous procedures in deformable environments

International audienceIn image-guided percutaneous interventions, a precise planning of the needle path is a key factor to a successful intervention. In this paper we propose a novel method for computing a patient-specific optimal path for such interventions, accounting for both the deformation of the needle and soft tissues due to the insertion of the needle in the body. To achieve this objective, we propose an optimization method for estimating preoperatively a curved trajectory allowing to reach a target even in the case of tissue motion and needle bending. Needle insertions are simulated and regarded as evaluations of the objective function by the iterative planning process. In order to test the planning algorithm, it is coupled with a fast needle insertion simulation involving a flexible needle model and soft tissue finite element modeling, and experimented on the use-case of thermal ablation of liver tumors. Our algorithm has been successfully tested on twelve datasets of patient-specific geometries. Fast convergence to the actual optimal solution has been shown. This method is designed to be adapted to a wide range of percutaneous interventions

Real-time hybrid cutting with dynamic fluid visualization for virtual surgery

It is widely accepted that a reform in medical teaching must be made to meet today's high volume training requirements. Virtual simulation offers a potential method of providing such trainings and some current medical training simulations integrate haptic and visual feedback to enhance procedure learning. The purpose of this project is to explore the capability of Virtual Reality (VR) technology to develop a training simulator for surgical cutting and bleeding in a general surgery