From teleoperation to autonomous robot-assisted microsurgery: A survey

Robot-assisted microsurgery (RAMS) has many benefits compared to traditional microsurgery. Microsurgical platforms with advanced control strategies, high-quality micro-imaging modalities and micro-sensing systems are worth developing to further enhance the clinical outcomes of RAMS. Within only a few decades, microsurgical robotics has evolved into a rapidly developing research field with increasing attention all over the world. Despite the appreciated benefits, significant challenges remain to be solved. In this review paper, the emerging concepts and achievements of RAMS will be presented. We introduce the development tendency of RAMS from teleoperation to autonomous systems. We highlight the upcoming new research opportunities that require joint efforts from both clinicians and engineers to pursue further outcomes for RAMS in years to come

Optical Coherence Tomography Distal Sensor Based Handheld Microsurgical Tools

Microsurgery is typically differentiated from a general surgery in that it requires a precise sub-millimeter manipulation that could only be achievable under optical magnification. For instance, microsurgeons use surgical microscopes to view surgical sites and train themselves several years to acquire surgical skills to perform the delicate procedures. However, such microsurgical approach imposes considerable physical stress and mental fatigue on the surgeons and these could be sources for surgical risks and complications. For these reasons, a variety of robotic based surgical guidance methods have been developed and studied with the hope of providing safer and more precise microsurgery. These robotic arm based systems have been developed to provide precise tool movement and to remove physiological hand tremor, which is one of the main limiting factors that prevents precise tool manipulation. In another approaches use simpler system that adds robotic functions to existing handheld surgical tools. It is a hybrid system that incorporates the advantages of conventional manual system and robot-assist system. The advantages of such hybrid handheld systems include portability, disposability, and elimination of the large robotic-assist systems in complex surgical environment. The most critical benefit of the hybrid handheld system is its ease of use since it allows surgeons to manipulate tools mostly using their hand. However due to the imprecise nature of tool control using hands, tool tracking is more critical in handheld microsurgical tool systems than that of robotic arm systems. In general, the accuracy of the tool control is largely determined by the resolution of the sensors and the actuators. Therefore, it is essential to develop a real-time high resolution sensor in order to develop a practical microsurgical tools. For this reason, a novel intuitive targeting and tracking scheme that utilizes a common-path swept source optical coherence tomography (CP-SSOCT) distal sensor was developed integrated with handheld microsurgical tools. To achieve micron-order precision control, a reliable and accurate OCT distal sensing method was developed. The method uses a prediction algorithm is necessary to compensate for the system delay associated with the computational, mechanical and electronic latencies. Due to the multi-layered structure of retina, it was also necessary to develop effective surface detection methods rather than simple peak detection. The OCT distal sensor was integrated into handheld motion-guided micro-forceps system for highly accurate depth controlled epiretinal membranectomy. A touch sensor and two motors were used in the forceps design to minimize the motion artifact induced by squeezing, and to independently control the depth guidance of the tool-tip and the grasping action. We also built a depth guided micro-injector system that enables micro-injection with precise injection depth control. For these applications, a smart motion monitoring and a guiding algorithm were developed to provide precise and intuitive freehand control. Finally, phantom and ex-vivo bovine eye experiments were performed to evaluate the performance of the proposed OCT distal sensor and validate the effectiveness of the depth-guided micro-forceps and micro-injector over the freehand performance

Développement et validation de sondes en fibre optique miniaturisées pour le guidage intra-opératoire d’interventions intraoculaires

Les procédures chirurgicales intraoculaires sont des procédures difficiles par la précision qu’elles demandent, on parle de microchirurgie, mais aussi par la difficulté et la faible qualité de visualisation des tissus à traiter. En effet, dans la plupart des procédures intraoculaires le chirurgien utilise uniquement un microscope ophtalmologique qui ne permet la visualisation des tissus que par la pupille du patient et offre une perception limitée de la profondeur. La Tomographie en Cohérence Optique (OCT) fournit des images en profondeur des tissus sains de manière non invasive, elle est utilisée couramment en diagnostic ophtalmologique et est de plus en plus utilisée intra-opérativement. Dans cette thèse nous allons présenter deux systèmes OCT intra-opératifs qui visent à assister les chirurgiens sur deux procédures intraoculaires, la vitrectomie et l’injection sous-rétinienne. Pour ces deux projets nous avons utilisé le matériel chirurgical utilisé cliniquement pour plusieurs raisons : s’assurer d’utiliser des outils adéquats (dimensions, efficacité, sécurité) pour la procédure, garder des outils que les chirurgiens utilisent régulièrement et avec lesquels ils sont familiers et limiter les coûts de développement. Pour le système OCT nous avons utilisé des sondes OCT en fibre optique car elles sont flexibles, bon marché et de petit diamètre. Leur focalisation peut également être modifiée dépendamment de l’application avec une fibre optique GRIN à leur extrémité pour augmenter le signal OCT. Nous avons ainsi attaché à ces outils chirurgicaux des sondes OCT en fibre optique. Pour le projet portant sur les injections sous-rétiniennes il a fallu dans un premier temps développer des sondes OCT avec des diamètres plus petits que ceux existant. Pour ce faire nous avons développé une méthode permettant de réduire le diamètre des sondes avec de l’acide fluorhydrique et grâce à un design permettant de conserver les propriétés optiques des sondes. Ce travail est présenté dans le premier article. Le second article présente un système permettant de guider les injections sous-rétiniennes. L’injection sous-rétinienne est une intervention chirurgicale de haute précision visant à restaurer et/ou préserver la vision des patients souffrant de maladies rétiniennes. Néanmoins, l’injection sous-rétinienne reste à la limite des capacités physiologiques humaines en raison des tremblements de la main et peut être compromise par le reflux du médicament si l’injection n’est pas assez profonde dans la rétine. Nous avons développé un système pour guider l’injection avec un micromanipulateur et donner des informations précises sur la profondeur au chirurgien avec l’OCT intra-opératif. Après avoir miniaturisé une sonde OCT en fibre optique avec la méthode présentée dans l’article 1 nous avons pu l’insérer dans une canule utilisée cliniquement. La sonde couplée à un système OCT que nous avons développé acquiert un signal A-scan qui va permettre de connaitre la distance entre la canule et la rétine mais aussi de sélectionner la profondeur de l’injection dans les couches rétiniennes. La canule est attachée à un micromanipulateur qui assure son déplacement dans l’œil. Une image M-scan est construite avec le signal OCT et le chirurgien peut directement sélectionner sur l’image la profondeur de l’injection. Nous avons développé l’interface sur Labview. Après avoir sélectionné la cible de l’injection le programme de guidage va déplacer la canule et injecter le volume adéquat grâce à une pompe contrôlable. Nous avons validé notre système de guidage sur des yeux de porcs ex-vivo. Sur 40 injections 38 présentaient un décollement rétinien ciblé et localisé, preuve de la réussie de l’injection rétinienne ce qui représente un taux de succès de 95% (CI : 83.1 – 99.4). Nous avons aussi grâce à un algorithme de traitement de l’image calculé le volume présent sous la rétine après l’injection que nous avons comparé au volume injecté. Nous avons ainsi trouvé que 75% du volume initialement injecté se retrouve bien sous la rétine. Le troisième article présente un système permettant d’arrêter automatiquement le vitrecteur lors d’une vitrectomie pour réduire les dommages accidentels sur la rétine. La survenue de déchirures rétiniennes iatrogèniques dans la vitrectomie par la pars plane est une complication qui compromet l’efficacité globale de la chirurgie. Un certain nombre de déchirures rétiniennes iatrogènes se produisent lorsque la rétine est coupée accidentellement par le vitrecteur. Nous avons développé un vitrecteur intelligent capable de détecter en temps réel une coupure rétinienne accidentelle et de désactiver rapidement la machine de vitrectomie pour les prévenir. Ce vitrecteur intelligent est composé d’une sonde OCT attachée au vitrecteur et va avoir comme rôle de détecter si le vitrecteur aspire la rétine et va endommager ces tissus sains. La sonde OCT agit comme un détecteur de présence devant l’ouverture du vitrecteur, ceci en comparant un signal de référence avec le signal en direct. Cette comparaison de signal OCT va commander un bras robotique pour actionner la pédale d’arrêt du vitrecteur. Ainsi le chirurgien n’a pas besoin d’interpréter un signal, la décision d’arrêt du vitrecteur dû à la présence de la rétine est prise automatiquement. Ceci va permettre de réduire grandement, de 300 ms à 29 ms, le délai de la prise de décision d’arrêt du vitrecteur précédemment limité par le temps de réaction du chirurgien. Nous avons développé les sondes OCT, le système OCT ainsi que l’algorithme d’arrêt automatique de ce système. Nous avons validé sur des yeux porcins in-vivo, deux chirurgiens ont utilisé notre système en essayant d’endommager les tissus rétiniens. 70% (CI : 56.39 – 82.02) des tentatives de dommages rétiniens des chirurgiens furent atténuées ou empêchées par notre système. Ce projet a abouti au dépôt d’un brevet ("Smart Vitrector", Provisional patent application, US 63109040).Intraocular surgical procedures are difficult procedures because of the precision they require, they are often referred as microsurgery, but also by the little information available to the surgeon. In most intraocular procedures the surgeon only uses an ophthalmic microscope which allows visualization of tissue just through the patient’s pupil and offers limited depth perception. Optical Coherence Tomography (OCT) provides in-depth images of healthy tissue in a non-invasive manner, is commonly used in ophthalmologic diagnostics, and is increasingly used intraoperatively. In this thesis we will present two intraoperative OCT systems that aim to assist surgeons with two intraocular procedures, vitrectomy and subretinal injection. For these two projects we used the surgical equipment used clinically for several reasons : to make sure to use adequate tools (dimensions, efficiency, safety) for the procedure, to keep tools that surgeons use regularly and with which they are familiar and limit development costs. For the OCT system we used fiber optic OCT probes as they are flexible, cheap and small in diameter. Their focus can also be modified, depending the application, with a GRIN fiber at their tip to increase the OCT signal. We have attached optical fiber OCT probes to these surgical tools. For the subretinal injections project it was first necessary to develop OCT probes with smaller diameters than existing ones. To do this, we have developed a method to reduce the diameter of the probes with hydrofluoric acid and a design to maintain the optical properties of the probes. This work is presented in the first article. The second article presents a system for guiding subretinal injections. Subretinal injection of drugs is a challenging surgical intervention aiming to restore and/or preserve the vision of patients suffering from retinal diseases. Nevertheless, the subretinal injection remains at the edge of human physiological capacity because of hand tremor and can be mitigated by drug reflux if the injection is not deep enough in the retina. We developed a system to guide the injection with a micromanipulator and give precise depth information to the surgeon with intraoperative OCT. To do so we first miniaturized an optical fiber OCT probe with the method presented in article 1, we were able to insert it into a cannula used clinically. The probe coupled to an OCT system that we have developed acquires an A-scan signal which enables to know the distance between the cannula and the retina but also to select the depth of the injection into the retinal layers. The cannula is attached to a micromanipulator that moves it inside the eye. An M-scan image is built with the OCT signal and the surgeon can directly select on the image the depth of the injection. We developed the interface on Labview. After selecting the injection target, the guidance program will move the cannula and inject the appropriate volume using a controllable pump.We have validated our guidance system on pig eyes ex-vivo. Out of 40 injections, 38 presented a retinal detachment, proof of a successful retinal injection, which represents a success rate of 95% (CI : 83.1 – 99.4). Thanks to an image processing algorithm, we also calculated the bleb volume under the retina after the injection, which we compared to the initial injected volume. We have found that 75% of the injected volume ends in the subretinal space. The third article presents for automatically stopping the vitrector during a vitrectomy. The occurrence of iatrogenic retinal breaks in pars plana vitrectomy is a complication that compromises the overall efficacy of the surgery. A subset of iatrogenic retinal break occurs when the retina is cut accidentally by the vitrector. We developed a smart vitrector that can detect in real-time potential accidental retinal cut and activate promptly a vitrectomy machine to prevent them. To do so an OCT probe is attached to the vitrector and will have the role of detecting if the vitrector sucks the retina and will damage these healthy tissues. The OCT probe acts as a presence detector in front of the vitrector opening, by comparing a reference signal with the live signal. This OCT signal comparison will control a robotic arm to operate the vitrector stop pedal. Thus, the surgeon does not need to interpret a signal, the decision to stop the vitrector due to the presence of the retina is taken automatically. This will greatly reduce, from 300 ms to 29 ms, the delay to stop the vitrector previously limited by the reaction time of the surgeon. We have developed the OCT probes, the OCT system, and the automatic shutdown algorithm for this system. We validated our system on in-vivo porcine eyes, two surgeons used the modified vitrector trying to damage retinal tissue. 70% (CI : 56.39 – 82.02) of surgeons’ retinal damage attempts were mitigated or prevented by our system. This project resulted in a patent ("Smart Vitrector", Provisional patent application, US 63109040)

Force-Sensing-Based Multi-Platform Robotic Assistance for Vitreoretinal Surgery

Vitreoretinal surgery aims to treat disorders of the retina, vitreous body, and macula, such as retinal detachment, diabetic retinopathy, macular hole, epiretinal membrane and retinal vein occlusion. Challenged by several technical and human limitations, vitreoretinal practice currently ranks amongst the most demanding fields in ophthalmic surgery. Of vitreoretinal procedures, membrane peeling is the most common to be performed, over 0.5 million times annually, and among the most prone to complications. It requires an extremely delicate tissue manipulation by various micron scale maneuvers near the retina despite the physiological hand tremor of the operator. In addition, to avoid injuries, the applied forces on the retina need to be kept at a very fine level, which is often well below the tactile sensory threshold of the surgeon. Retinal vein cannulation is another demanding procedure where therapeutic agents are injected into occluded retinal veins. The feasibility of this treatment is limited due to challenges in identifying the moment of venous puncture, achieving cannulation and maintaining it throughout the drug delivery period. Recent advancements in medical robotics have significant potential to address most of the challenges in vitreoretinal practice, and therefore to prevent traumas, lessen complications, minimize intra-operative surgeon effort, maximize surgeon comfort, and promote patient safety. This dissertation presents the development of novel force-sensing tools that can easily be used on various robotic platforms, and robot control methods to produce integrated assistive surgical systems that work in partnership with surgeons against the current limitations in vitreoretinal surgery, specifically focusing on membrane peeling and vein cannulation procedures. Integrating high sensitivity force sensing into the ophthalmic instruments enables precise quantitative monitoring of applied forces. Auditory feedback based upon the measured forces can inform (and warn) the surgeon quickly during the surgery and help prevent injury due to excessive forces. Using these tools on a robotic platform can attenuate hand tremor of the surgeon, which effectively promotes tool manipulation accuracy. In addition, based upon certain force signatures, the robotic system can precisely identify critical instants, such as the venous puncture in retinal vein cannulation, and actively guide the tool towards clinical targets, compensate any involuntary motion of the surgeon, or generate additional motion that will make the surgical task easier. The experimental results using two distinct robotic platforms, the Steady-Hand Eye Robot and Micron, in combination with the force-sensing ophthalmic instruments, show significant performance improvement in artificial dry phantoms and ex vivo biological tissues

Control and Estimation Methods Towards Safe Robot-assisted Eye Surgery

Vitreoretinal surgery is among the most delicate surgical tasks in which physiological hand tremor may severely diminish surgeon performance and put the eye at high risk of injury. Unerring targeting accuracy is required to perform precise operations on micro-scale tissues. Tool tip to tissue interaction forces are usually below human tactile perception, which may result in exertion of excessive forces to the retinal tissue leading to irreversible damages. Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing and intelligent control algorithms of important surgery states and situations (e.g. instrument tip position measurements and control of interaction forces). This dissertation provides novel control and state estimation methods to improve safety during robot-assisted eye surgery. The integration of robotics into retinal microsurgery leads to a reduction in surgeon perception of tool-to-tissue forces at sclera. This blunting of human tactile sensory input, which is due to the inflexible inertia of the robot, is a potential iatrogenic risk during robotic eye surgery. To address this issue, a sensorized surgical instrument equipped with Fiber Bragg Grating (FBG) sensors, which is capable of measuring the sclera forces and instrument insertion depth into the eye, is integrated to the Steady-Hand Eye Robot (SHER). An adaptive control scheme is then customized and implemented on the robot that is intended to autonomously mitigate the risk of unsafe scleral forces and excessive insertion of the instrument. Various preliminary and multi-user clinician studies are then conducted to evaluate the effectiveness of the control method during mock retinal surgery procedures. In addition, due to inherent flexibility and the resulting deflection of eye surgical instruments as well as the need for targeting accuracy, we have developed a method to enhance deflected instrument tip position estimation. Using an iterative method and microscope data, we develop a calibration- and registration-independent (RI) framework to provide online estimates of the instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model for tip position evolution obtained based on the forward kinematics (FWK) of the robot and FBG sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the instrument tip position estimation during robotic surgery. The entire framework is independent of camera-to-robot coordinate frame registration and is evaluated during various phantom experiments to demonstrate its effectiveness

EyeLS: Shadow-Guided Instrument Landing System for Intraocular Target Approaching in Robotic Eye Surgery

Robotic ophthalmic surgery is an emerging technology to facilitate high-precision interventions such as retina penetration in subretinal injection and removal of floating tissues in retinal detachment depending on the input imaging modalities such as microscopy and intraoperative OCT (iOCT). Although iOCT is explored to locate the needle tip within its range-limited ROI, it is still difficult to coordinate iOCT's motion with the needle, especially at the initial target-approaching stage. Meanwhile, due to 2D perspective projection and thus the loss of depth information, current image-based methods cannot effectively estimate the needle tip's trajectory towards both retinal and floating targets. To address this limitation, we propose to use the shadow positions of the target and the instrument tip to estimate their relative depth position and accordingly optimize the instrument tip's insertion trajectory until the tip approaches targets within iOCT's scanning area. Our method succeeds target approaching on a retina model, and achieves an average depth error of 0.0127 mm and 0.3473 mm for floating and retinal targets respectively in the surgical simulator without damaging the retina.Comment: 10 page

Augmentation Of Human Skill In Microsurgery

Surgeons performing highly skilled microsurgery tasks can benefit from information and manual assistance to overcome technological and physiological limitations to make surgery safer, efficient, and more successful. Vitreoretinal surgery is particularly difficult due to inherent micro-scale and fragility of human eye anatomy. Additionally, surgeons are challenged by physiological hand tremor, poor visualization, lack of force sensing, and significant cognitive load while executing high-risk procedures inside the eye, such as epiretinal membrane peeling. This dissertation presents the architecture and the design principles for a surgical augmentation environment which is used to develop innovative functionality to address the fundamental limitations in vitreoretinal surgery. It is an inherently information driven modular system incorporating robotics, sensors, and multimedia components. The integrated nature of the system is leveraged to create intuitive and relevant human-machine interfaces and generate a particular system behavior to provide active physical assistance and present relevant sensory information to the surgeon. These include basic manipulation assistance, audio-visual and haptic feedback, intraoperative imaging and force sensing. The resulting functionality, and the proposed architecture and design methods generalize to other microsurgical procedures. The system's performance is demonstrated and evaluated using phantoms and in vivo experiments

Deep Learning Guided Autonomous Surgery: Guiding Small Needles into Sub-Millimeter Scale Blood Vessels

We propose a general strategy for autonomous guidance and insertion of a needle into a retinal blood vessel. The main challenges underpinning this task are the accurate placement of the needle-tip on the target vein and a careful needle insertion maneuver to avoid double-puncturing the vein, while dealing with challenging kinematic constraints and depth-estimation uncertainty. Following how surgeons perform this task purely based on visual feedback, we develop a system which relies solely on \emph{monocular} visual cues by combining data-driven kinematic and contact estimation, visual-servoing, and model-based optimal control. By relying on both known kinematic models, as well as deep-learning based perception modules, the system can localize the surgical needle tip and detect needle-tissue interactions and venipuncture events. The outputs from these perception modules are then combined with a motion planning framework that uses visual-servoing and optimal control to cannulate the target vein, while respecting kinematic constraints that consider the safety of the procedure. We demonstrate that we can reliably and consistently perform needle insertion in the domain of retinal surgery, specifically in performing retinal vein cannulation. Using cadaveric pig eyes, we demonstrate that our system can navigate to target veins within 22$\mu m$ XY accuracy and perform the entire procedure in less than 35 seconds on average, and all 24 trials performed on 4 pig eyes were successful. Preliminary comparison study against a human operator show that our system is consistently more accurate and safer, especially during safety-critical needle-tissue interactions. To the best of the authors' knowledge, this work accomplishes a first demonstration of autonomous retinal vein cannulation at a clinically-relevant setting using animal tissues

Intraoperative Fourier domain optical coherence tomography for microsurgery guidance and assessment

In this dissertation, advanced high-speed Fourier domain optical coherence tomography (FD-OCT)systems were investigated and developed. Several real-time, high resolution functional Spectral-domain OCT (SD-OCT) systems capable of imaging and sensing blood flow and motion were designed and developed. The system were designed particularly for microsurgery guidance and assessment. The systems were tested for their ability to assessing microvascular anastomosis and vulnerable plaque development. An all fiber-optic common-path optical coherence tomography (CP-OCT) system capable of measuring high-resolution optical distances, was built and integrated into di fferent imaging modalities. First, a novel non-contact accurate in-vitro intra-ocular lens power measurement method was proposed and validated based on CP-OCT. Second, CP-OCT was integrated with a ber bundle based confocal microscope to achieve motion-compensated imaging. Distance between the probe and imaged target was monitored by the CP-OCT system in real-time.The distance signal from the CP-OCT system was routed to a high speed, high resolution linear motor to compensate for the axial motion of the sample in a closed-loop control. Finally a motion-compensated hand-held common-path Fourier domain optical coherence tomography probe was developed for image-guided intervention. Both phantom and ex vivo models were used to test and evaluate the probe. As the data acquisition speed of current OCT systems continue to increase, the means to process the data in real-time are in critically needed. Previous graphics processing unit accelerated OCT signal processing methods have shown their potential to achieve real-time imaging. In this dissertation, algorithms to perform real-time reference A-line subtraction and saturation artifact removal were proposed, realized and integrated into previously developed FD-OCT system CPU-GPU heterogeneous structure. Fourier domain phase resolved Doppler OCT (PRDOCT) system capable of real-time simultaneous structure and flow imaging based on dual GPUs was also developed and implemented. Finally, systematic experiments were conducted to validate the system for surgical applications. FD-OCT system was used to detect atherosclerotic plaque and drug effi ciency test in mouse model. Application of PRDOCT for both suture and cu ff based microvascular anastomosis guidance and assessment was extensively stuided in rodent model