Alessandro Casella

Alice Valeria Iordache

Elena De Momi

Elisa Iovene

Federico Pessina

Giancarlo Ferrigno

Junling Fu

Leonardo S. Mattos

Marco Riva

Archivio istituzionale della ricerca - Politecnico di Milano

Envisioning Robotic Exoscope: Concept and Preliminary ResultsAlice Valeria Iordache1, *, Alessandro Casella1, 2, *, Elisa Iovene1, Junling Fu1, Federico Pessina3,Marco Riva3, Giancarlo Ferrigno1, Leonardo S. Mattos2, and Elena De Momi11Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy2Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy3Humanitas Research Hospital, Milan, Italy*These authors contributed equally.INTRODUCTIONThe introduction of a surgical microscope in neurosurgeryincreased the spectrum and safety of interventions. Thesurgeon operates with both hands on the patient whilealso controlling the visualization system to provide theproper view. Despite the benefits of the microscope, lowergonomics may lead to long-term effects on the muscu-loskeletal system. The bulky binocular system may limitthe setup’s flexibility, hindering robotics integration intothe operating theatre.Recently, the introduction of the exoscopes overcame thoselimitations by enabling a flexible and ergonomic workingenvironment, combined with improved image quality. Theexoscope is fixed on the surgical field through pneumaticarm holders that the surgeon can move manually orthrough foot control. Although providing the surgeon withenhanced vision, the complexity of its use has emergedas a limiting factor [1]. The continuous switching fromoperation theatre and visualization system reduces thesmoothness of the surgical procedure, and the steep learn-ing curve of the foot pedal [2] leads to switching to theconventional microscope in most cases. Visual servoingtechniques [3] have been primarily studied in the contextof camera automation in minimally invasive laparoscopy[4]. Its potential has become even more significant in thecontext of neurosurgery, as the only solution for trackinginstruments is to employ vision sensors. However, visualservoing techniques for exoscope automation are currentlylimited due to open challenges in neurosurgical practices.This work proposes a framework for an autonomousvision-guided camera holder that tracks and follows aselected surgical instrument based on a visual servoingtechnique. We envision this solution to automate theexoscope in neurosurgery, providing intuitive control ofthe system and reducing surgeon workload and operatingtime.MATERIALS AND METHODSA. Visual Servoing FrameworkIn this study, a markerless position-based visual servoingframework was implemented as illustrated in Fig. 1. A 7DOF serial manipulator (KUKA LwR 4+) was used witha stereo camera mounted in an eye-in-hand configurationto simulate the exoscope system. An object-detectionFig. 1 System’s overview: tool coordinates (𝑥𝑅, 𝑦𝑅) and (𝑥𝐿 , 𝑦𝐿) are identified in the image space. The position𝑃(𝑋,𝑌, 𝑍) in the 3D space is estimated and filtered by the Kalman Filter 𝑻𝒄𝒂𝒎𝒐𝒃𝒋𝑲𝑭. This, together with the desiredposition 𝑻𝒄𝒂𝒎∗𝒐𝒃𝒋, are transformed and used to compute the desired position of the camera 𝒙𝒃𝒂𝒔𝒆𝒅. The desired and actualposition of the camera, 𝒙𝒃𝒂𝒔𝒆𝒂 , are used to calculate the error signal 𝒆. The controller computes the required jointpositions 𝒒𝒅 that compensate for the error.Fig. 2 Design of the task, only one ring is seen by thecamera, camera need to be moved to reach the pegboard.neural network Yolov3[5] was trained to identify the tipof a surgical instrument from the images acquired bythe 3D camera. The training dataset was composed of5000 images, 1800 were extracted from the EndoVis 2017challenge dataset [6] while the remaining were recordedand manually annotated.The position reconstruction was computed by triangu-lation using DLT, while the Kalman filter was applied,considering a three-dimensional constant velocity modelof the instrument’s motion. The desired position of theinstrument with respect to the camera 𝑻𝒄𝒂𝒎∗𝒐𝒃𝒋was definedin such a way that the instrument is kept at the centerof the image. Only motions along the 𝑋𝑌 -plane havebeen implemented since, in neurosurgery, movement alongthe depth is only required for focus adjustments. Finally,a position controller was employed to compensate forthe error between the desired and actual position of thecamera.B. Experimental protocolIn order to test the usability of the system, a user studywas carried out on ten non-medical subjects. The designedtask was a pick-and-place in which users were asked touse a surgical instrument to pick up, one at a time, fourrandomly distributed rings in a defined workspace andplace them on a target pegboard, as shown in Fig. 2.The distance in the 𝑍 direction between the camera andthe task space was 0.2𝑚 and was kept fixed to provide areduced Field of View (FoV) and force the user to movethe camera to complete the task. Users were asked toperform the task exclusively by observing the scene on anexternal monitor where the camera FoV was displayed.The task was executed in two different modalities:1) Autonomous Camera Control (ACC): the user wasable to activate and deactivate the autonomous mo-tion of the camera by pressing a foot pedal.2) Joystick Control (JC): the user was able to move thecamera using a foot-controlled joystick each time adifferent view of the scene was needed.Three repetitions were performed for both modalities.RESULTS AND DISCUSSIONThe systems’ functionality was evaluated in terms of tool’sposition density inside the image plane and execution time.Fig. 3 shows that in ACC, the density is higher at thecenter of the image than in JC. This proves that the ACCis more effective at keeping the instrument in the center ofthe image, providing a better view of the scene, which iscrucial for optimal task performance. As for the executionFig. 3 Normalized instrument’s tip density inside theimage plane for the JC and ACC modalitiestime, ACC showed better results, as illustrated in Table I.The mean and standard deviation of the execution timefor the three repetitions are reported.TABLE I Mean execution time, 𝑠𝑡𝑑 and p-value for thethree repetitions of the JC and ACC modalities.Repetition 1 2 3Modality JC ACC JC ACC JC ACCExec. time[s] 112 82.6 105.3 80.2 104.9 73.1±11.8 ±24.6 ±14.4 ±26.2 ±18.1 ±18.5p-value ** ** **This demonstrates that ACC reduces completion timeby allowing the user to focus only on the main taskand consequently perform better. Statistical analysis wasconducted for all repetitions using the Wilcoxon signed-rank test with a 5% level of significance.CONCLUSIONS AND DISCUSSIONPreliminary results have shown that the proposed marker-less visual servoing system has significant advantages overthe currently used system. However, translating this systeminto operating room requires further improvements. Futurestudies will focus on the improvement of the tracking andcontrol module.REFERENCES[1] Mamelak, et al. A high-definition exoscope system for neurosurgeryand other microsurgical disciplines: Preliminary report. SurgicalInnovation, 15(1):38–46, 2008. PMID: 18388000.[2] Maurer, et al. Evaluation of a novel three-dimensional robotic digitalmicroscope (aeos) in neurosurgery. Cancers, 13(17), 2021.[3] Chaumette and Hutchinson. Visual servo control. i. basic ap-proaches. Robotics Automation Magazine, IEEE, 13:82 – 90, 012007.[4] Gruijthuijsen, et al. Robotic endoscope control via autonomousinstrument tracking, 2021.[5] Redmon and Farhadi. Yolov3: An incremental improvement. 042018.[6] Allan, et al. 2017 robotic instrument segmentation challenge, 2019.

English

Envisioning Robotic Exoscope: Concept and Preliminary Results

Envisioning Robotic Exoscope: Concept and Preliminary Results

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Politecnico di Milano