Stereo vision-based tracking of soft tissue motion with application to online ablation control in laser microsurgery

Recent research has revealed that image-based methods can enhance accuracy and safety in laser microsurgery. In this study, non-rigid tracking using surgical stereo imaging and its application to laser ablation is discussed. A recently developed motion estimation framework based on piecewise affine deformation modeling is extended by a mesh refinement step and considering texture information. This compensates for tracking inaccuracies potentially caused by inconsistent feature matches or drift. To facilitate online application of the method, computational load is reduced by concurrent processing and affine-invariant fusion of tracking and refinement results. The residual latency-dependent tracking error is further minimized by Kalman filter-based upsampling, considering a motion model in disparity space. Accuracy is assessed in laparoscopic, beating heart, and laryngeal sequences with challenging conditions, such as partial occlusions and significant deformation. Performance is compared with that of state-of-the-art methods. In addition, the online capability of the method is evaluated by tracking two motion patterns performed by a high-precision parallel-kinematic platform. Related experiments are discussed for tissue substitute and porcine soft tissue in order to compare performances in an ideal scenario and in a setup mimicking clinical conditions. Regarding the soft tissue trial, the tracking error can be significantly reduced from 0.72 mm to below 0.05 mm with mesh refinement. To demonstrate online laser path adaptation during ablation, the non-rigid tracking framework is integrated into a setup consisting of a surgical Er:YAG laser, a three-axis scanning unit, and a low-noise stereo camera. Regardless of the error source, such as laser-to-camera registration, camera calibration, image-based tracking, and scanning latency, the ablation root mean square error is kept below 0.21 mm when the sample moves according to the aforementioned patterns. Final experiments regarding motion-compensated laser ablation of structurally deforming tissue highlight the potential of the method for vision-guided laser surgery.EU/FP/-ICT/28866

Methods for a fusion of Optical Coherence Tomography and stereo camera image data

This work investigates combination of Optical Coherence Tomography and two cameras, observing a microscopic scene. Stereo vision provides realistic images, but is limited in terms of penetration depth. Optical Coherence Tomography (OCT) enables access to subcutaneous structures, but 3D-OCT volume data do not give the surgeon a familiar view. The extension of the stereo camera setup with OCT imaging combines the benefits of both modalities. In order to provide the surgeon with a convenient integration of OCT into the vision interface, we present an automated image processing analysis of OCT and stereo camera data as well as combined imaging as augmented reality visualization. Therefore, we care about OCT image noise, perform segmentation as well as develop proper registration objects and methods. The registration between stereo camera and OCT results in a Root Mean Square error of 284 μm as average of five measurements. The presented methods are fundamental for fusion of both imaging modalities. Augmented reality is shown as application of the results. Further developments lead to fused visualization of subcutaneous structures, as information of OCT images, into stereo vision. © 2015 SPIE

Endoluminal non-contact soft tissue ablation using fiber-based Er:YAG laser delivery

The introduction of Er:YAG lasers for soft and hard tissue ablation has proven promising results over the last decades due to strong absorption at 2.94 μm wavelength by water molecules. An extension to endoluminal applications demands laser delivery without mirror arms due to dimensional constraints. Therefore, fiber-based solutions are advanced to provide exible access while keeping space requirements to a minimum. Conventional fiber-based treatments aim at laser-tissue interactions in contact mode. However, this procedure is associated with disadvantages such as advancing decrease in power delivery due to particle coverage of the fiber tip, tissue carbonization, and obstructed observation of the ablation progress. The objective of this work is to overcome aforementioned limitations with a customized fiber-based module for non-contact robot-assisted endoluminal surgery and its associated experimental evaluation. Up to the authors knowledge, this approach has not been presented in the context of laser surgery at 2.94 μm wavelength. The preliminary system design is composed of a 3D Er:YAG laser processing unit enabling automatic laser to fiber coupling, a GeO2 solid core fiber, and a customized module combining collimation and focusing unit (focal length of 20 mm, outer diameter of 8 mm). The performance is evaluated with studies on tissue substitutes (agar-agar) as well as porcine samples that are analysed by optical coherence tomography measurements. Cuts (depths up to 3mm) with minimal carbonization have been achieved under adequate moistening and sample movement (1.5mms-1). Furthermore, an early cadaver study is presented. Future work aims at module miniaturization and integration into an endoluminal robot for scanning and focus adaptation. © 2016 SPIE

Stereo vision-guided laser microsurgery

Transoral laser microsurgery (TLM) provides the most advanced microscopic technique for contact-less, atraumatic treatment of oral, pharyngeal, and in especially laryngeal carcinomas. The primary goal of the laser ablation is radical removal of the lesion. This necessitates the selection of sufficiently large resection margins and thus conflicts with a further principle of surgery which is function preservation. Preserving as much healthy tissue as possible requires very accurate laser positioning that, as of today, demands a highly experienced surgeon. Even though the advantages of TLM are manifold, there are substantial technical limitations compromising the post-operative outcome and hence the quality of life of the patient. Major challenges do not solely arise from a limited field of view and range of motion, but in particular from inaccurate laser focusing, inadequate imaging of the submucosal extent, suboptimal incision planning and loss of ablation accuracy due to soft tissue motion. To overcome these limitations, this dissertation presents a novel approach for stereo vision-guided laser microsurgery with special emphasis on laryngeal interventions. A variety of laser technologies and computer vision algorithms exist; however, holistic integration of real-time stereoscopic image processing into soft tissue laser surgery, considering the laser characteristics on the one hand and surgical scene information, e.g., structure and non-rigid motion, on the other hand, has not been addressed in its entity thus far. A computational method for stereo vision-based real-time surface estimation is developed as a prerequisite to multimodal registration. The reconstruction accuracy is assessed on an in vivo dataset and a variety of stereo imaging devices considered for microsurgery. A method for laser-to-camera registration is proposed facilitating distance-based laser focus adjustment and thus optimal ablation. To assist the manual focus repositioning process, color-encoded distance superimposed to the live view is implemented as part of the surgeon interface. A further contribution aims at the fusion of optical coherence tomography imaging with stereo vision. A registration and segmentation framework enables the detection and visualization of submucosal changes of laryngeal tissue. Moreover, three-dimensional surface information and laser-to-camera registration are integrated into a stylus-tablet-based interface. Several path planning strategies are evaluated in terms of accuracy and usability. Finally, laser ablation in a dynamic soft tissue environment is addressed with an algorithm for non-rigid tracking. Experiments demonstrate the benefit of live view stabilization during incision planning and closed loop control for motion compensation during laser ablation

\u3bcRALP and Beyond: Micro-Technologies and Systems for Robot-Assisted Endoscopic Laser Microsurgery

Laser microsurgery is the current gold standard surgical technique for the treatment of selected diseases in delicate organs such as the larynx. However, the operations require large surgical expertise and dexterity, and face significant limitations imposed by available technology, such as the requirement for direct line of sight to the surgical field, restricted access, and direct manual control of the surgical instruments. To change this status quo, the European project mu RALP pioneered research towards a complete redesign of current laser microsurgery systems, focusing on the development of robotic micro-technologies to enable endoscopic operations. This has fostered awareness and interest in this field, which presents a unique set of needs, requirements and constraints, leading to research and technological developments beyond mu RALP and its research consortium. This paper reviews the achievements and key contributions of such research, providing an overview of the current state of the art in robot-assisted endoscopic laser microsurgery. The primary target application considered is phonomicrosurgery, which is a representative use case involving highly challenging microsurgical techniques for the treatment of glottic diseases. The paper starts by presenting the motivations and rationale for endoscopic laser microsurgery, which leads to the introduction of robotics as an enabling technology for improved surgical field accessibility, visualization and management. Then, research goals, achievements, and current state of different technologies that can build-up to an effective robotic system for endoscopic laser microsurgery are presented. This includes research in micro-robotic laser steering, flexible robotic endoscopes, augmented imaging, assistive surgeon-robot interfaces, and cognitive surgical systems. Innovations in each of these areas are shown to provide sizable progress towards more precise, safer and higher quality endoscopic laser microsurgeries. Yet, major impact is really expected from the full integration of such individual contributions into a complete clinical surgical robotic system, as illustrated in the end of this paper with a description of preliminary cadaver trials conducted with the integrated mu RALP system. Overall, the contribution of this paper lays in outlining the current state of the art and open challenges in the area of robot-assisted endoscopic laser microsurgery, which has important clinical applications even beyond laryngology