Haptics-Enabled Teleoperation for Robotics-Assisted Minimally Invasive Surgery

The lack of force feedback (haptics) in robotic surgery can be considered to be a safety risk leading to accidental tissue damage and puncturing of blood vessels due to excessive forces being applied to tissue and vessels or causing inefficient control over the instruments because of insufficient applied force. This project focuses on providing a satisfactory solution for introducing haptic feedback in robotics-assisted minimally invasive surgical (RAMIS) systems. The research addresses several key issues associated with the incorporation of haptics in a master-slave (teleoperated) robotic environment for minimally invasive surgery (MIS). In this project, we designed a haptics-enabled dual-arm (two masters - two slaves) robotic MIS testbed to investigate and validate various single-arm as well as dual-arm teleoperation scenarios. The most important feature of this setup is the capability of providing haptic feedback in all 7 degrees of freedom (DOF) required for RAMIS (3 translations, 3 rotations and pinch motion of the laparoscopic tool). The setup also enables the evaluation of the effect of replacing haptic feedback by other sensory cues such as visual representation of haptic information (sensory substitution) and the hypothesis that surgical outcomes may be improved by substituting or augmenting haptic feedback by such sensory cues

Force Measurement Methods in Telerobotic Surgery: Implications for End-Effector Manufacture

Haptic feedback in telesurgical applications refers to the relaying of position and force information from a remote surgical site to the surgeon in real-time during a surgical procedure. This feedback, coupled with visual information via microscopic cameras, has the potential to provide the surgeon with additional ‘feel’ for the manipulations being performed at the instrument-biological tissue interface. This increased sensitivity has many associated benefits which include, but are not limited to; minimal tissue damage, reduced recuperation periods, and less patient trauma. The inclusion of haptic feedback leads to reduction in surgeon fatigue which contributes to enhanced performance during operation. Commercially available Minimally Invasive Robotic Surgical (MIRS) systems are being widely used, the best-known examples being from the daVinci® by Intuitive Surgical Inc. However, currently these systems do not possess force feedback capability which therefore restricts their use during many delicate and complex procedures. The ideal system would consist of a multi-degree-of-freedom framework which includes end-effector instruments with embedded force sensing included. A force sensing characterisation platform has been developed by this group which facilitates the evaluation of force sensing technologies. Surgical scissors have been chosen as the instrument and biological tissue phantom specimens have been used during testing. This test-bed provides accurate, repeatable measurements of the forces produced at the interface between the tissue and the scissor blades during cutting using conventional sensing technologies. The primary focus of this paper is to provide a review of the traditional and developing force sensing technologies with a view to establishing the most appropriate solution for this application. The impact that an appropriate sensing technology has on the manufacturability of the instrument end-effector is considered. Particular attention is given to the issues of embedding the force sensing transducer into the instrument tip

Using High-Level Processing of Low-Level Signals to Actively Assist Surgeons with Intelligent Surgical Robots

Robotic surgical systems are increasingly used for minimally-invasive surgeries. As such, there is opportunity for these systems to fundamentally change the way surgeries are performed by becoming intelligent assistants rather than simply acting as the extension of surgeons' arms. As a step towards intelligent assistance, this thesis looks at ways to represent different aspects of robot-assisted surgery (RAS). We identify three main components: the robot, the surgeon actions, and the patient scene dynamics. Traditional learning algorithms in these domains are predominantly supervised methods. This has several drawbacks. First many of these domains are non-categorical, like how soft-tissue deforms. This makes labeling difficult. Second, surgeries vary greatly. Estimation of the robot state may be affected by how the robot is docked and cable tensions in the instruments. Estimation of the patient anatomy and its dynamics are often inaccurate, and in any case, may change throughout a surgery. To obtain the most accurate information, these aspects must be learned during the procedure. This limits the amount of labeling that could be done. On the surgeon side, different surgeons may perform the same procedure differently and the algorithm should provide personalized estimations for surgeons. All of these considerations motivated the use of self-supervised learning throughout this thesis. We first build a representation of the robot system. In particular, we looked at learning the dynamics model of the robot. We evaluate the model by using it to estimate forces. Once we can estimate forces in free space, we extend the algorithm to take into account patient-specific interactions, namely with the trocar and the cannula seal. Accounting for surgery-specific interactions is possible because our method does not require additional sensors and can be trained in less than five minutes, including time for data collection. Next, we use cross-modal training to understand surgeon actions by looking at the bottleneck layer when mapping video to kinematics. This should contain information about the latent space of surgeon-actions, while discarding some medium-specific information about either the video or the kinematics. Lastly, to understand the patient scene, we start with modeling interactions between a robot instrument and a soft-tissue phantom. Models are often inaccurate due to imprecise material parameters and boundary conditions, particularly in clinical scenarios. Therefore, we add a depth camera to observe deformations to correct the results of simulations. We also introduce a network that learns to simulate soft-tissue deformation from physics simulators in order to speed up the estimation. We demonstrate that self-supervised learning can be used for understanding each part of RAS. The representations it learns contain information about signals that are not directly measurable. The self-supervised nature of the methods presented in this thesis lends itself well to learning throughout a surgery. With such frameworks, we can overcome some of the main barriers to adopting learning methods in the operating room: the variety in surgery and the difficulty in labeling enough training data for each case

Cooperative Object Manipulation with Force Tracking on the da Vinci Research Kit

The da Vinci Surgical System is one of the most established robot-assisted surgery device commended for its dexterity and ergonomics in minimally invasive surgery. Conversely, it inherits disadvantages which are lack of autonomy and haptic feedback. In order to address these issues, this work proposes an industry-inspired solution to the field of force control in medical robotics. This approach contributes to shared autonomy by developing a controller for cooperative object manipulation with force tracking utilizing available manipulators and force feedback. To achieve simultaneous position and force tracking of the object, master and slave manipulators were assigned then controlled with Cartesian position control and impedance control respectively. Because impedance control requires a model-based feedforward compensation, we identified the lumped base parameters of mass, inertias, and frictions of a three degree-of-freedom double four-bar linkage mechanism with least squares and weighted least squares regression methods. Additionally, semidefinite programming was used to constrain the parameters to a feasible physical solution in standard parameter space. Robust stick-slip static friction compensation was applied where linear Viscous and Coulomb friction was inadequate in modeling the prismatic third joint. The Robot Operating System based controller was tested in RViz to check the cooperative kinematics of up to three manipulators. Additionally, simulation with the dynamic engine Gazebo verified the cooperative controller applying a constant tension force on a massless spring-damper virtual object. With adequate model feedback linearization, the cooperative impedance controller tested on the da Vinci Research Kit yielded stable tension force tracking while simultaneously moving in Cartesian space. The maximum force tracking error was +/- 0.5 N for both a compliant and stiff manipulated object

A Sensorized Instrument for Minimally Invasive Surgery for the Measurement of Forces during Training and Surgery: Development and Applications

The reduced access conditions present in Minimally Invasive Surgery (MIS) affect the feel of interaction forces between the instruments and the tissue being treated. This loss of haptic information compromises the safety of the procedure and must be overcome through training. Haptics in MIS is the subject of extensive research, focused on establishing force feedback mechanisms and developing appropriate sensors. This latter task is complicated by the need to place the sensors as close as possible to the instrument tip, as the measurement of forces outside of the patient\u27s body does not represent the true tool--tissue interaction. Many force sensors have been proposed, but none are yet available for surgery. The objectives of this thesis were to develop a set of instruments capable of measuring tool--tissue force information in MIS, and to evaluate the usefulness of force information during surgery and for training and skills assessment. To address these objectives, a set of laparoscopic instruments was developed that can measure instrument position and tool--tissue interaction forces in multiple degrees of freedom. Different design iterations and the work performed towards the development of a sterilizable instrument are presented. Several experiments were performed using these instruments to establish the usefulness of force information in surgery and training. The results showed that the combination of force and position information can be used in the development of realistic tissue models or haptic interfaces specifically designed for MIS. This information is also valuable in order to create tactile maps to assist in the identification of areas of different stiffness. The real-time measurement of forces allows visual force feedback to be presented to the surgeon. When applied to training scenarios, the results show that experience level correlates better with force-based metrics than those currently used in training simulators. The proposed metrics can be automatically computed, are completely objective, and measure important aspects of performance. The primary contribution of this thesis is the design and development of highly versatile instruments capable of measuring force and position during surgery. A second contribution establishes the importance and usefulness of force data during skills assessment, training and surgery

sCAM: An Untethered Insertable Laparoscopic Surgical Camera Robot

Fully insertable robotic imaging devices represent a promising future of minimally invasive laparoscopic vision. Emerging research efforts in this field have resulted in several proof-of-concept prototypes. One common drawback of these designs derives from their clumsy tethering wires which not only cause operational interference but also reduce camera mobility. Meanwhile, these insertable laparoscopic cameras are manipulated without any pose information or haptic feedback, which results in open loop motion control and raises concerns about surgical safety caused by inappropriate use of force.This dissertation proposes, implements, and validates an untethered insertable laparoscopic surgical camera (sCAM) robot. Contributions presented in this work include: (1) feasibility of an untethered fully insertable laparoscopic surgical camera, (2) camera-tissue interaction characterization and force sensing, (3) pose estimation, visualization, and feedback with sCAM, and (4) robotic-assisted closed-loop laparoscopic camera control. Borrowing the principle of spherical motors, camera anchoring and actuation are achieved through transabdominal magnetic coupling in a stator-rotor manner. To avoid the tethering wires, laparoscopic vision and control communication are realized with dedicated wireless links based on onboard power. A non-invasive indirect approach is proposed to provide real-time camera-tissue interaction force measurement, which, assisted by camera-tissue interaction modeling, predicts stress distribution over the tissue surface. Meanwhile, the camera pose is remotely estimated and visualized using complementary filtering based on onboard motion sensing. Facilitated by the force measurement and pose estimation, robotic-assisted closed-loop control has been realized in a double-loop control scheme with shared autonomy between surgeons and the robotic controller.The sCAM has brought robotic laparoscopic imaging one step further toward less invasiveness and more dexterity. Initial ex vivo test results have verified functions of the implemented sCAM design and the proposed force measurement and pose estimation approaches, demonstrating the technical feasibility of a tetherless insertable laparoscopic camera. Robotic-assisted control has shown its potential to free surgeons from low-level intricate camera manipulation workload and improve precision and intuitiveness in laparoscopic imaging

Realization of a demonstrator slave for robotic minimally invasive surgery

Robots for Minimally Invasive Surgery (MIS) can improve the surgeon’s work conditions with respect to conventional MIS and to enable MIS with more complex procedures. This requires to provide the surgeon with tactile feedback to feel forces executed on e.g. tissue and sutures, which is partially lost in conventional MIS. Additionally use of a robot should improve the approach possibilities of a target organ by means of instrument degrees of freedom (DoFs) and of the entry points with a compact set-up. These requirements add to the requirements set by the most common commercially available system, the da Vinci which are: (i) dexterity, (ii) natural hand-eye coordination, (iii) a comfortable body posture, (iv) intuitive utilization, and (v) a stereoscopic ’3D’ view of the operation site. The purpose of Sofie (Surgeon’s operating force-feedback interface Eindhoven) is to evaluate the possible benefit of force-feedback and the approach of both patient and target organ. Sofie integrates master, slave, electronic hardware and control. This thesis focusses on the design and realization of a technology demonstrator of the Slave. To provide good accuracy and valuable force-feedback, good dynamic behavior and limited hysteresis are required. To this end the Slave includes (i) a relatively short force-path between its instrument-tips and between tip and patient, and (ii) a passive instrument-support by means of a remote kinematically fixed point of rotation. The incision tissue does not support the instrument. The Slave is connected directly to the table. It provides a 20 DoF highly adaptable stiff frame (pre-surgical set-up) with a short force-path between the instrumenttips and between instrument-tip and patient. During surgery this frame supports three 4 DoF manipulators, two for exchangeable 4 DoF instruments and one for an endoscope. The pre-surgical set-up of the Slave consists of a 5 DoF platform-adjustment with a platform. This platform can hold three 5 DoF manipulator-adjustments in line-up. The set-up is compact to avoid interference with the team, entirely mechanical and allows fast manual adjustment and fixation of the joints. It provides a stiff frame during surgery. A weight-compensation mechanism for the platformadjustment has been proposed. Measurements indicate all natural frequencies are above 25 Hz. The manipulator moves the instrument in 4 DoFs (??, , ?? and Z). Each manipulator passively supports its instrument with a parallelogram mechanism, providing a kinematically fixed point of rotation. Two manipulators have been designed in consecutive order. The first manipulator drives with a worm-wormwheel, the second design uses a ball-screw drive. This ball-screw drive reduces friction, which is preferred for next generations of the manipulator, since the worm-wormwheel drive shows a relatively low coherence at low frequencies. The compact ??Zmanipulator moves the instrument in ?? by rotating a drum. Friction wheels in the drum provide Z. Eventually, the drum will be removable from the manipulator for sterilization. This layout of the manipulator results in a small motion-envelope and least obstructs the team at the table. Force sensors measuring forces executed with the instrument, are integrated in the manipulator instead of at the instrument tip, to avoid all risks of electrical signals being introduced into the patient. Measurements indicate the separate sensors function properly. Integrated in the manipulator the sensors provide a good indication of the force but do suffer from some hysteresis which might be caused by moving wires. The instrument as realized consists of a drive-box, an instrument-tube and a 4 DoF tip. It provides the surgeon with three DoFs additional to the gripper of conventional MIS instruments. These DoFs include two lateral rotations (pitch and pivot) to improve the approach possibilities and the roll DoF will contribute in stitching. Pitch and roll are driven by means of bevelgears, driven with concentric tubes. Cables drive the pivot and close DoFs of the gripper. The transmissions are backdriveable for safety. Theoretical torques that can be achieved with this instrument approximate the requirements closely. Further research needs to reveal the torques achieved in practice and whether the requirements obtained from literature actually are required for these 4 DoF instruments. Force-sensors are proposed and can be integrated. Sofie currently consists of a master prototype with two 5 DoF haptic interfaces, the Slave and an electronic hardware cabinet. The surgeon uses the haptic interfaces of the Master to manipulate the manipulators and instruments of the Slave, while the actuated DoFs of the Master provide the surgeon with force-feedback. This project resulted in a demonstrator of the slave with force sensors incorporated, compact for easy approach of the patient and additional DoFs to increase approach possibilities of the target organ. This slave and master provide a good starting point to implement haptic controllers. These additional features may ultimately benefit both surgeon and patient

Design of Novel Sensors and Instruments for Minimally Invasive Lung Tumour Localization via Palpation

Minimally Invasive Thoracoscopic Surgery (MITS) has become the treatment of choice for lung cancer. However, MITS prevents the surgeons from using manual palpation, thereby often making it challenging to reliably locate the tumours for resection. This thesis presents the design, analysis and validation of novel tactile sensors, a novel miniature force sensor, a robotic instrument, and a wireless hand-held instrument to address this limitation. The low-cost, disposable tactile sensors have been shown to easily detect a 5 mm tumour located 10 mm deep in soft tissue. The force sensor can measure six degrees of freedom forces and torques with temperature compensation using a single optical fiber. The robotic instrument is compatible with the da Vinci surgical robot and allows the use of tactile sensing, force sensing and ultrasound to localize the tumours. The wireless hand-held instrument allows the use of tactile sensing in procedures where a robot is not available

Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery

International audienceThis paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery