Multicamera 3D Viewpoint Adjustment for Robotic Surgery via Deep Reinforcement Learning

While robot-assisted minimally invasive surgery (RMIS) procedures afford a variety of benefits over open surgery and manual laparoscopic operations (including increased tool dexterity, reduced patient pain, incision size, trauma and recovery time, and lower infection rates [1], lack of spatial awareness remains an issue. Typical laparoscopic imaging can lack sufficient depth cues and haptic feedback, if provided, rarely reflects realistic tissue-tool interactions. This work is part of a larger ongoing research effort to reconstruct 3D surfaces using multiple viewpoints in RMIS to increase visual perception. The manual placement and adjustment of multicamera systems in RMIS are nonideal and prone to error [2], and other autonomous approaches focus on tool tracking and do not consider reconstruction of the surgical scene [3,4,5]. The group\u27s previous work investigated a novel, context-aware autonomous camera positioning method [6], which incorporated both tool location and scene coverage for multiple camera viewpoint adjustments. In this paper, the authors expand upon this prior work by implementing a streamlined deep reinforcement learning approach between optimal viewpoints calculated using the prior method [6] which encourages discovery of otherwise unobserved and additional camera viewpoints. Combining the framework and robustness of the previous work with the efficiency and additional viewpoints of the augmentations presented here results in improved performance and scene coverage promising towards real-time implementation

Technologies for Biomechanically-Informed Image Guidance of Laparoscopic Liver Surgery

Laparoscopic surgery for liver resection has a number medical advantages over open surgery, but also comes with inherent technical challenges. The surgeon only has a very limited field of view through the imaging modalities routinely employed intra-operatively, laparoscopic video and ultrasound, and the pneumoperitoneum required to create the operating space and gaining access to the organ can significantly deform and displace the liver from its pre-operative configuration. This can make relating what is visible intra-operatively to the pre-operative plan and inferring the location of sub-surface anatomy a very challenging task. Image guidance systems can help overcome these challenges by updating the pre-operative plan to the situation in theatre and visualising it in relation to the position of surgical instruments. In this thesis, I present a series of contributions to a biomechanically-informed image-guidance system made during my PhD. The most recent one is work on a pipeline for the estimation of the post-insufflation configuration of the liver by means of an algorithm that uses a database of segmented training images of patient abdomens where the post-insufflation configuration of the liver is known. The pipeline comprises an algorithm for inter and intra-subject registration of liver meshes by means of non-rigid spectral point-correspondence finding. My other contributions are more fundamental and less application specific, and are all contained and made available to the public in the NiftySim open-source finite element modelling package. Two of my contributions to NiftySim are of particular interest with regards to image guidance of laparoscopic liver surgery: 1) a novel general purpose contact modelling algorithm that can be used to simulate contact interactions between, e.g., the liver and surrounding anatomy; 2) membrane and shell elements that can be used to, e.g., simulate the Glisson capsule that has been shown to significantly influence the organ’s measured stiffness

A biomechanics-based articulation model for medical applications

Computer Graphics came into the medical world especially after the arrival of 3D medical imaging. Computer Graphics techniques are already integrated in the diagnosis procedure by means of the visual tridimensional analysis of computer tomography, magnetic resonance and even ultrasound data. The representations they provide, nevertheless, are static pictures of the patients' body, lacking in functional information. We believe that the next step in computer assisted diagnosis and surgery planning depends on the development of functional 3D models of human body. It is in this context that we propose a model of articulations based on biomechanics. Such model is able to simulate the joint functionality in order to allow for a number of medical applications. It was developed focusing on the following requirements: it must be at the same time simple enough to be implemented on computer, and realistic enough to allow for medical applications; it must be visual in order for applications to be able to explore the joint in a 3D simulation environment. Then, we propose to combine kinematical motion for the parts that can be considered as rigid, such as bones, and physical simulation of the soft tissues. We also deal with the interaction between the different elements of the joint, and for that we propose a specific contact management model. Our kinematical skeleton is based on anatomy. Special considerations have been taken to include anatomical features like axis displacements, range of motion control, and joints coupling. Once a 3D model of the skeleton is built, it can be simulated by data coming from motion capture or can be specified by a specialist, a clinician for instance. Our deformation model is an extension of the classical mass-spring systems. A spherical volume is considered around mass points, and mechanical properties of real materials can be used to parameterize the model. Viscoelasticity, anisotropy and non-linearity of the tissues are simulated. We particularly proposed a method to configure the mass-spring matrix such that the objects behave according to a predefined Young's modulus. A contact management model is also proposed to deal with the geometric interactions between the elements inside the joint. After having tested several approaches, we proposed a new method for collision detection which measures in constant time the signed distance to the closest point for each point of two meshes subject to collide. We also proposed a method for collision response which acts directly on the surfaces geometry, in a way that the physical behavior relies on the propagation of reaction forces produced inside the tissue. Finally, we proposed a 3D model of a joint combining the three elements: anatomical skeleton motion, biomechanical soft tissues deformation, and contact management. On the top of that we built a virtual hip joint and implemented a set of medical applications prototypes. Such applications allow for assessment of stress distribution on the articular surfaces, range of motion estimation based on ligament constraint, ligament elasticity estimation from clinically measured range of motion, and pre- and post-operative evaluation of stress distribution. Although our model provides physicians with a number of useful variables for diagnosis and surgery planning, it should be improved for effective clinical use. Validation has been done partially. However, a global clinical validation is necessary. Patient specific data are still difficult to obtain, especially individualized mechanical properties of tissues. The characterization of material properties in our soft tissues model can also be improved by including control over the shear modulus