Visual Tracking in Robotic Minimally Invasive Surgery

Intra-operative imaging and robotics are some of the technologies driving forward better and more effective minimally invasive surgical procedures. To advance surgical practice and capabilities further, one of the key requirements for computationally enhanced interventions is to know how instruments and tissues move during the operation. While endoscopic video captures motion, the complex appearance dynamic effects of surgical scenes are challenging for computer vision algorithms to handle with robustness. Tackling both tissue and instrument motion estimation, this thesis proposes a combined non-rigid surface deformation estimation method to track tissue surfaces robustly and in conditions with poor illumination. For instrument tracking, a keypoint based 2D tracker that relies on the Generalized Hough Transform is developed to initialize a 3D tracker in order to robustly track surgical instruments through long sequences that contain complex motions. To handle appearance changes and occlusion a patch-based adaptive weighting with segmentation and scale tracking framework is developed. It takes a tracking-by-detection approach and a segmentation model is used to assigns weights to template patches in order to suppress back- ground information. The performance of the method is thoroughly evaluated showing that without any offline-training, the tracker works well even in complex environments. Finally, the thesis proposes a novel 2D articulated instrument pose estimation framework, which includes detection-regression fully convolutional network and a multiple instrument parsing component. The framework achieves compelling performance and illustrates interesting properties includ- ing transfer between different instrument types and between ex vivo and in vivo data. In summary, the thesis advances the state-of-the art in visual tracking for surgical applications for both tissue and instrument motion estimation. It contributes to developing the technological capability of full surgical scene understanding from endoscopic video

Stereo vision-based tracking of soft tissue motion with application to online ablation control in laser microsurgery

Recent research has revealed that image-based methods can enhance accuracy and safety in laser microsurgery. In this study, non-rigid tracking using surgical stereo imaging and its application to laser ablation is discussed. A recently developed motion estimation framework based on piecewise affine deformation modeling is extended by a mesh refinement step and considering texture information. This compensates for tracking inaccuracies potentially caused by inconsistent feature matches or drift. To facilitate online application of the method, computational load is reduced by concurrent processing and affine-invariant fusion of tracking and refinement results. The residual latency-dependent tracking error is further minimized by Kalman filter-based upsampling, considering a motion model in disparity space. Accuracy is assessed in laparoscopic, beating heart, and laryngeal sequences with challenging conditions, such as partial occlusions and significant deformation. Performance is compared with that of state-of-the-art methods. In addition, the online capability of the method is evaluated by tracking two motion patterns performed by a high-precision parallel-kinematic platform. Related experiments are discussed for tissue substitute and porcine soft tissue in order to compare performances in an ideal scenario and in a setup mimicking clinical conditions. Regarding the soft tissue trial, the tracking error can be significantly reduced from 0.72 mm to below 0.05 mm with mesh refinement. To demonstrate online laser path adaptation during ablation, the non-rigid tracking framework is integrated into a setup consisting of a surgical Er:YAG laser, a three-axis scanning unit, and a low-noise stereo camera. Regardless of the error source, such as laser-to-camera registration, camera calibration, image-based tracking, and scanning latency, the ablation root mean square error is kept below 0.21 mm when the sample moves according to the aforementioned patterns. Final experiments regarding motion-compensated laser ablation of structurally deforming tissue highlight the potential of the method for vision-guided laser surgery.EU/FP/-ICT/28866

A Modular and Open-Source Framework for Virtual Reality Visualisation and Interaction in Bioimaging

Life science today involves computational analysis of a large amount and variety of data, such as volumetric data acquired by state-of-the-art microscopes, or mesh data from analysis of such data or simulations. The advent of new imaging technologies, such as lightsheet microscopy, has resulted in the users being confronted with an ever-growing amount of data, with even terabytes of imaging data created within a day. With the possibility of gentler and more high-performance imaging, the spatiotemporal complexity of the model systems or processes of interest is increasing as well. Visualisation is often the first step in making sense of this data, and a crucial part of building and debugging analysis pipelines. It is therefore important that visualisations can be quickly prototyped, as well as developed or embedded into full applications. In order to better judge spatiotemporal relationships, immersive hardware, such as Virtual or Augmented Reality (VR/AR) headsets and associated controllers are becoming invaluable tools. In this work we present scenery, a modular and extensible visualisation framework for the Java VM that can handle mesh and large volumetric data, containing multiple views, timepoints, and color channels. scenery is free and open-source software, works on all major platforms, and uses the Vulkan or OpenGL rendering APIs. We introduce scenery's main features, and discuss its use with VR/AR hardware and in distributed rendering. In addition to the visualisation framework, we present a series of case studies, where scenery can provide tangible benefit in developmental and systems biology: With Bionic Tracking, we demonstrate a new technique for tracking cells in 4D volumetric datasets via tracking eye gaze in a virtual reality headset, with the potential to speed up manual tracking tasks by an order of magnitude. We further introduce ideas to move towards virtual reality-based laser ablation and perform a user study in order to gain insight into performance, acceptance and issues when performing ablation tasks with virtual reality hardware in fast developing specimen. To tame the amount of data originating from state-of-the-art volumetric microscopes, we present ideas how to render the highly-efficient Adaptive Particle Representation, and finally, we present sciview, an ImageJ2/Fiji plugin making the features of scenery available to a wider audience.:Abstract Foreword and Acknowledgements Overview and Contributions Part 1 - Introduction 1 Fluorescence Microscopy 2 Introduction to Visual Processing 3 A Short Introduction to Cross Reality 4 Eye Tracking and Gaze-based Interaction Part 2 - VR and AR for System Biology 5 scenery — VR/AR for Systems Biology 6 Rendering 7 Input Handling and Integration of External Hardware 8 Distributed Rendering 9 Miscellaneous Subsystems 10 Future Development Directions Part III - Case Studies C A S E S T U D I E S 11 Bionic Tracking: Using Eye Tracking for Cell Tracking 12 Towards Interactive Virtual Reality Laser Ablation 13 Rendering the Adaptive Particle Representation 14 sciview — Integrating scenery into ImageJ2 & Fiji Part IV - Conclusion 15 Conclusions and Outlook Backmatter & Appendices A Questionnaire for VR Ablation User Study B Full Correlations in VR Ablation Questionnaire C Questionnaire for Bionic Tracking User Study List of Tables List of Figures Bibliography Selbstständigkeitserklärun

Medical Imaging of Microrobots: Toward In Vivo Applications

Medical microrobots (MRs) have been demonstrated for a variety of non-invasive biomedical applications, such as tissue engineering, drug delivery, and assisted fertilization, among others. However, most of these demonstrations have been carried out in in vitro settings and under optical microscopy, being significantly different from the clinical practice. Thus, medical imaging techniques are required for localizing and tracking such tiny therapeutic machines when used in medical-relevant applications. This review aims at analyzing the state of the art of microrobots imaging by critically discussing the potentialities and limitations of the techniques employed in this field. Moreover, the physics and the working principle behind each analyzed imaging strategy, the spatiotemporal resolution, and the penetration depth are thoroughly discussed. The paper deals with the suitability of each imaging technique for tracking single or swarms of MRs and discusses the scenarios where contrast or imaging agent's inclusion is required, either to absorb, emit, or reflect a determined physical signal detected by an external system. Finally, the review highlights the existing challenges and perspective solutions which could be promising for future in vivo applications

OPTICAL NAVIGATION TECHNIQUES FOR MINIMALLY INVASIVE ROBOTIC SURGERIES

Minimally invasive surgery (MIS) involves small incisions in a patient's body, leading to reduced medical risk and shorter hospital stays compared to open surgeries. For these reasons, MIS has experienced increased demand across different types of surgery. MIS sometimes utilizes robotic instruments to complement human surgical manipulation to achieve higher precision than can be obtained with traditional surgeries. Modern surgical robots perform within a master-slave paradigm, in which a robotic slave replicates the control gestures emanating from a master tool manipulated by a human surgeon. Presently, certain human errors due to hand tremors or unintended acts are moderately compensated at the tool manipulation console. However, errors due to robotic vision and display to the surgeon are not equivalently addressed. Current vision capabilities within the master-slave robotic paradigm are supported by perceptual vision through a limited binocular view, which considerably impacts the hand-eye coordination of the surgeon and provides no quantitative geometric localization for robot targeting. These limitations lead to unexpected surgical outcomes, and longer operating times compared to open surgery. To improve vision capabilities within an endoscopic setting, we designed and built several image guided robotic systems, which obtained sub-millimeter accuracy. With this improved accuracy, we developed a corresponding surgical planning method for robotic automation. As a demonstration, we prototyped an autonomous electro-surgical robot that employed quantitative 3D structural reconstruction with near infrared registering and tissue classification methods to localize optimal targeting and suturing points for minimally invasive surgery. Results from validation of the cooperative control and registration between the vision system in a series of in vivo and in vitro experiments are presented and the potential enhancement to autonomous robotic minimally invasive surgery by utilizing our technique will be discussed

IR Laser-Induced Gene Expression for Tracking Development of Single Embryonic Neurons and Glia in C. Elegans

The assembly of neural circuits requires a complex choreography of developmental events: neurons must be generated, extend neurites at the correct time and location, and then integrate extracellular information, like long-range guidance cues or cellular contacts, with an internal developmental program to make correct wiring decisions. Visualizing neural-circuit assembly in vivo can provide insight into how these events are coordinated. The C. elegans embryo, which contains only 222 neurons and 56 glia, is an attractive setting to study nervous system development comprehensively in an intact, living organism. However, methods to label and track optically-resolvable neurites or manipulate single neurons through gene expression do not exist, as most embryonic reporters are broadly expressed. Here, I present a method for expressing fluorescent reporters or any gene of interest in specific C. elegans embryonic neurons, glia, or other cell types, without cell specific drivers. Our method is based on a previous setup (Kamei et al., 2009), and uses an infrared (IR) laser to localize heat to the volume of a single precursor cell in the embryo. This induces gene expression in the progeny of that cell (1-4 cells/embryo) through heat-shock-response regulatory elements. I perform significant optimizations to adapt this strategy to cells in the C. elegans embryo, which are highly sensitive to heat toxicity. Direct temperature measurements of IR heating in the embryo reveal that cells are heated to physiological temperatures (320C) for 5 minute durations using our modified irradiation protocol. These conditions lead to high rates of gene induction (\u3e60%) with no signs of damage. First, I use our system to label and track single neurons during early nervous system assembly. These studies reveal a retrograde extension mechanism for axon growth in specific interneurons. I also study the etiology of axon-guidance defects in sax-3/Robo and vab-1/EphR mutants; these studies suggest that a timing/competence mechanism controls axon-outgrowth dynamics in the nerve ring. Next, I demonstrate the versatility of IR irradiation by performing cell-specific rescues, determining DAF-6/Patched-related site of action during sensory-organ development. Finally, I demonstrate that IR cell irradiation can be used to perform simultaneous ablation and labeling of cells in the same embryo. I use this system to uncover a role for the amphid sheath glia in dendrite extension. As IR induction can be used for targeted labeling, gene expression, and ablation without the need for cell-specific drivers, this tool opens to door to high resolution systematic analyses of C. elegans morphogenesis

Uterine transplantation: the move from the animal model into the human setting – surgical, reproductive and clinical aspects

Women with absolute uterine factor infertility (AUFI) are considered as being ‘unconditionally infertile’. Uterine transplantation (UTx) may be a possible treatment option in the future for such women. This thesis describes a number of key areas of research that are important in order to move closer to a successful and crucially, safe transplant in the human setting. Nine allogeneic transplants were carried out in a rabbit model to investigate anatomical and surgical aspects necessary for a successful UTx. An attempt to characterise and quantify the immunological mechanisms involved in allogeneic UTx (rejection patterns) was made. Out of the nine recipients, one was a long-term survivor. Embryo transfer was performed in this one doe with the aim of establishing pregnancy. Performing UTx in a large-animal model is necessary as the pelvis resembles a woman’s reproductive system more closely. In addition, the anastomotic technique is similar. Five sheep autotransplants were performed to further define surgical techniques. The anastomotic model was internal to external iliac vessel. Out of the five transplants, three sheep demonstrated adequate perfusion in the immediate post-operative period. Furthermore, the suitability of two different imaging modalities, pulse oximetry and multispectral imaging, for assessing uterine perfusion and extent of ischaemia were been studied in both the rabbit and sheep models. Biophotonics was also applied in the form of Endoscopic Laser Speckle Contrast Analysis to characterize blood flow in the two models. Both Multispectral Imaging and Endoscopic Laser Speckle Contrast Analysis have never been assessed before in a gynecological context. In order to transfer the concept of UTx to the human, we carried out a retrospective study of abdominal radical trachelectomy (ART) as a potential replacement for radical hysterectomy in patients with early stage cervical cancer desiring a fertility-sparing procedure. ART forms the foundation of the original work into aspects of UTx. This original body of research revolved around the potential blood supply to a uterus. Furthermore, an attempt has been made to analyse the motivations, aims and feelings of patients diagnosed with AUFI towards UTx. Forty patients were interviewed. The final study involved the evaluation of the perceptions of health care professionals towards UTx, with 528 participants surveyed.Open Acces

Aerospace medicine and biology: A cumulative index to a continuing bibliography (supplement 345)

This publication is a cumulative index to the abstracts contained in Supplements 333 through 344 of Aerospace Medicine and Biology: A Continuing Bibliography. Seven indexes are included -- subject, personal author, corporate source, foreign technology, contract number, report number, and accession number

MEMS Technology for Biomedical Imaging Applications

Biomedical imaging is the key technique and process to create informative images of the human body or other organic structures for clinical purposes or medical science. Micro-electro-mechanical systems (MEMS) technology has demonstrated enormous potential in biomedical imaging applications due to its outstanding advantages of, for instance, miniaturization, high speed, higher resolution, and convenience of batch fabrication. There are many advancements and breakthroughs developing in the academic community, and there are a few challenges raised accordingly upon the designs, structures, fabrication, integration, and applications of MEMS for all kinds of biomedical imaging. This Special Issue aims to collate and showcase research papers, short commutations, perspectives, and insightful review articles from esteemed colleagues that demonstrate: (1) original works on the topic of MEMS components or devices based on various kinds of mechanisms for biomedical imaging; and (2) new developments and potentials of applying MEMS technology of any kind in biomedical imaging. The objective of this special session is to provide insightful information regarding the technological advancements for the researchers in the community