Visual Tracking in Robotic Minimally Invasive Surgery

Intra-operative imaging and robotics are some of the technologies driving forward better and more effective minimally invasive surgical procedures. To advance surgical practice and capabilities further, one of the key requirements for computationally enhanced interventions is to know how instruments and tissues move during the operation. While endoscopic video captures motion, the complex appearance dynamic effects of surgical scenes are challenging for computer vision algorithms to handle with robustness. Tackling both tissue and instrument motion estimation, this thesis proposes a combined non-rigid surface deformation estimation method to track tissue surfaces robustly and in conditions with poor illumination. For instrument tracking, a keypoint based 2D tracker that relies on the Generalized Hough Transform is developed to initialize a 3D tracker in order to robustly track surgical instruments through long sequences that contain complex motions. To handle appearance changes and occlusion a patch-based adaptive weighting with segmentation and scale tracking framework is developed. It takes a tracking-by-detection approach and a segmentation model is used to assigns weights to template patches in order to suppress back- ground information. The performance of the method is thoroughly evaluated showing that without any offline-training, the tracker works well even in complex environments. Finally, the thesis proposes a novel 2D articulated instrument pose estimation framework, which includes detection-regression fully convolutional network and a multiple instrument parsing component. The framework achieves compelling performance and illustrates interesting properties includ- ing transfer between different instrument types and between ex vivo and in vivo data. In summary, the thesis advances the state-of-the art in visual tracking for surgical applications for both tissue and instrument motion estimation. It contributes to developing the technological capability of full surgical scene understanding from endoscopic video

Bayesian Estimation of Intrinsic Tissue Oxygenation and Perfusion from RGB Images

Multispectral imaging (MSI) can potentially assist the intra-operative assessment of tissue structure, function and viability, by providing information about oxygenation. In this paper, we present a novel technique for recovering intrinsic MSI measurements from endoscopic RGB images without custom hardware adaptations. The advantage of this approach is that it requires no modification to existing surgical and diagnostic endoscopic imaging systems. Our method uses a radiometric colour calibration of the endoscopic camera's sensor in conjunction with a Bayesian framework to recover a per-pixel measurement of the total blood volume (THb) and oxygen saturation (SO2) in the observed tissue. The sensor's pixel measurements are modelled as weighted sums over a mixture of Poisson distributions and we optimise the variables SO2 and THb to maximise the likelihood of the observations. To validate our technique, we use synthetic images generated from Monte Carlo (MC) physics simulation of light transport through soft tissue containing sub-surface blood vessels. We also validate our method on in vivo data by comparing it to a MSI dataset acquired with a hardware system that sequentially images multiple spectral bands without overlap. Our results are promising and show that we are able to provide surgeons with additional relevant information by processing endoscopic images with our modelling and inference framework

SCALING ARTIFICIAL INTELLIGENCE IN ENDOSCOPY: FROM MODEL DEVELOPMENT TO MACHINE LEARNING OPERATIONS FRAMEWORKS

Questa tesi esplora l'integrazione dell'intelligenza artificiale (IA) in Otorinolaringoiatria – Chirurgia di Testa e Collo, concentrandosi sui progressi della computer vision per l’endoscopia e le procedure chirurgiche. La ricerca inizia con una revisione completa dello stato dell’arte dell'IA e della computer vision in questo campo, identificando aree per ulteriori sviluppi. L'obiettivo principale è stato quello di sviluppare un sistema di computer vision per l'analisi di immagini e video endoscopici. La ricerca ha coinvolto la progettazione di strumenti per la rilevazione e segmentazione di neoplasie nelle vie aerodigestive superiori (VADS) e la valutazione della motilità delle corde vocali, cruciale nella stadiazione del carcinoma laringeo. Inoltre, lo studio si è focalizzato sul potenziale dei foundation vision models, vision transformers basati su self-supervised learning, per ridurre la necessità di annotazione da parte di esperti, approccio particolarmente vantaggioso in campi con dati limitati. Inoltre, la ricerca ha incluso lo sviluppo di un'applicazione web per migliorare e velocizzare il processo di annotazione in endoscopia delle VADS, nell’ambito generale delle tecniche di MLOps. La tesi copre varie fasi della ricerca, a partire dalla definizione del quadro concettuale e della metodologia, denominata "Videomics". Include una revisione della letteratura sull'IA in endoscopia clinica, focalizzata sulla Narrow Band Imaging (NBI) e sulle reti neurali convoluzionali (CNN). Lo studio progredisce attraverso diverse fasi, dalla valutazione della qualità delle immagini endoscopiche alla caratterizzazione approfondita delle lesioni neoplastiche. Si affronta anche la necessità di standard nel reporting degli studi di computer vision in ambito medico e si valuta l'applicazione dell'IA in setting dinamici come la motilità delle corde vocali. Una parte significativa della ricerca indaga l'uso di algoritmi di computer vision generalizzati (“foundation models”) e la “commoditization” degli algoritmi di machine learning, utilizzando polipi nasali e il carcinoma orofaringeo come casi studio. Infine, la tesi discute lo sviluppo di ENDO-CLOUD, un sistema basato su cloud per l’analisi della videolaringoscopia, evidenziando le sfide e le soluzioni nella gestione dei dati e l’utilizzo su larga scala di modelli di IA nell'imaging medico.This thesis explores the integration of artificial intelligence (AI) in Otolaryngology – Head and Neck Surgery, focusing on advancements in computer vision for endoscopy and surgical procedures. It begins with a comprehensive review of AI and computer vision advancements in this field, identifying areas for further exploration. The primary aim was to develop a computer vision system for endoscopy analysis. The research involved designing tools for detecting and segmenting neoplasms in the upper aerodigestive tract (UADT) and assessing vocal fold motility, crucial in laryngeal cancer staging. Further, the study delves into the potential of vision foundation models, like vision transformers trained via self-supervision, to reduce the need for expert annotations, particularly beneficial in fields with limited cases. Additionally, the research includes the development of a web application for enhancing and speeding up the annotation process in UADT endoscopy, under the umbrella of Machine Learning Operations (MLOps). The thesis covers various phases of research, starting with defining the conceptual framework and methodology, termed "Videomics". It includes a literature review on AI in clinical endoscopy, focusing on Narrow Band Imaging (NBI) and convolutional neural networks (CNNs). The research progresses through different stages, from quality assessment of endoscopic images to in-depth characterization of neoplastic lesions. It also addresses the need for standards in medical computer vision study reporting and evaluates the application of AI in dynamic vision scenarios like vocal fold motility. A significant part of the research investigates the use of "general purpose" vision algorithms and the commoditization of machine learning algorithms, using nasal polyps and oropharyngeal cancer as case studies. Finally, the thesis discusses the development of ENDO-CLOUD, a cloud-based system for videolaryngoscopy, highlighting the challenges and solutions in data management and the large-scale deployment of AI models in medical imaging

Medical Imaging of Microrobots: Toward In Vivo Applications

Medical microrobots (MRs) have been demonstrated for a variety of non-invasive biomedical applications, such as tissue engineering, drug delivery, and assisted fertilization, among others. However, most of these demonstrations have been carried out in in vitro settings and under optical microscopy, being significantly different from the clinical practice. Thus, medical imaging techniques are required for localizing and tracking such tiny therapeutic machines when used in medical-relevant applications. This review aims at analyzing the state of the art of microrobots imaging by critically discussing the potentialities and limitations of the techniques employed in this field. Moreover, the physics and the working principle behind each analyzed imaging strategy, the spatiotemporal resolution, and the penetration depth are thoroughly discussed. The paper deals with the suitability of each imaging technique for tracking single or swarms of MRs and discusses the scenarios where contrast or imaging agent's inclusion is required, either to absorb, emit, or reflect a determined physical signal detected by an external system. Finally, the review highlights the existing challenges and perspective solutions which could be promising for future in vivo applications

IR Laser-Induced Gene Expression for Tracking Development of Single Embryonic Neurons and Glia in C. Elegans

The assembly of neural circuits requires a complex choreography of developmental events: neurons must be generated, extend neurites at the correct time and location, and then integrate extracellular information, like long-range guidance cues or cellular contacts, with an internal developmental program to make correct wiring decisions. Visualizing neural-circuit assembly in vivo can provide insight into how these events are coordinated. The C. elegans embryo, which contains only 222 neurons and 56 glia, is an attractive setting to study nervous system development comprehensively in an intact, living organism. However, methods to label and track optically-resolvable neurites or manipulate single neurons through gene expression do not exist, as most embryonic reporters are broadly expressed. Here, I present a method for expressing fluorescent reporters or any gene of interest in specific C. elegans embryonic neurons, glia, or other cell types, without cell specific drivers. Our method is based on a previous setup (Kamei et al., 2009), and uses an infrared (IR) laser to localize heat to the volume of a single precursor cell in the embryo. This induces gene expression in the progeny of that cell (1-4 cells/embryo) through heat-shock-response regulatory elements. I perform significant optimizations to adapt this strategy to cells in the C. elegans embryo, which are highly sensitive to heat toxicity. Direct temperature measurements of IR heating in the embryo reveal that cells are heated to physiological temperatures (320C) for 5 minute durations using our modified irradiation protocol. These conditions lead to high rates of gene induction (\u3e60%) with no signs of damage. First, I use our system to label and track single neurons during early nervous system assembly. These studies reveal a retrograde extension mechanism for axon growth in specific interneurons. I also study the etiology of axon-guidance defects in sax-3/Robo and vab-1/EphR mutants; these studies suggest that a timing/competence mechanism controls axon-outgrowth dynamics in the nerve ring. Next, I demonstrate the versatility of IR irradiation by performing cell-specific rescues, determining DAF-6/Patched-related site of action during sensory-organ development. Finally, I demonstrate that IR cell irradiation can be used to perform simultaneous ablation and labeling of cells in the same embryo. I use this system to uncover a role for the amphid sheath glia in dendrite extension. As IR induction can be used for targeted labeling, gene expression, and ablation without the need for cell-specific drivers, this tool opens to door to high resolution systematic analyses of C. elegans morphogenesis

OPTICAL NAVIGATION TECHNIQUES FOR MINIMALLY INVASIVE ROBOTIC SURGERIES

Minimally invasive surgery (MIS) involves small incisions in a patient's body, leading to reduced medical risk and shorter hospital stays compared to open surgeries. For these reasons, MIS has experienced increased demand across different types of surgery. MIS sometimes utilizes robotic instruments to complement human surgical manipulation to achieve higher precision than can be obtained with traditional surgeries. Modern surgical robots perform within a master-slave paradigm, in which a robotic slave replicates the control gestures emanating from a master tool manipulated by a human surgeon. Presently, certain human errors due to hand tremors or unintended acts are moderately compensated at the tool manipulation console. However, errors due to robotic vision and display to the surgeon are not equivalently addressed. Current vision capabilities within the master-slave robotic paradigm are supported by perceptual vision through a limited binocular view, which considerably impacts the hand-eye coordination of the surgeon and provides no quantitative geometric localization for robot targeting. These limitations lead to unexpected surgical outcomes, and longer operating times compared to open surgery. To improve vision capabilities within an endoscopic setting, we designed and built several image guided robotic systems, which obtained sub-millimeter accuracy. With this improved accuracy, we developed a corresponding surgical planning method for robotic automation. As a demonstration, we prototyped an autonomous electro-surgical robot that employed quantitative 3D structural reconstruction with near infrared registering and tissue classification methods to localize optimal targeting and suturing points for minimally invasive surgery. Results from validation of the cooperative control and registration between the vision system in a series of in vivo and in vitro experiments are presented and the potential enhancement to autonomous robotic minimally invasive surgery by utilizing our technique will be discussed

Characterising pattern asymmetry in pigmented skin lesions

Abstract. In clinical diagnosis of pigmented skin lesions asymmetric pigmentation is often indicative of melanoma. This paper describes a method and measures for characterizing lesion symmetry. The estimate of mirror symmetry is computed first for a number of axes at different degrees of rotation with respect to the lesion centre. The statistics of these estimates are the used to assess the overall symmetry. The method is applied to three different lesion representations showing the overall pigmentation, the pigmentation pattern, and the pattern of dermal melanin. The best measure is a 100% sensitive and 96% specific indicator of melanoma on a test set of 33 lesions, with a separate training set consisting of 66 lesions

MEMS Technology for Biomedical Imaging Applications

Biomedical imaging is the key technique and process to create informative images of the human body or other organic structures for clinical purposes or medical science. Micro-electro-mechanical systems (MEMS) technology has demonstrated enormous potential in biomedical imaging applications due to its outstanding advantages of, for instance, miniaturization, high speed, higher resolution, and convenience of batch fabrication. There are many advancements and breakthroughs developing in the academic community, and there are a few challenges raised accordingly upon the designs, structures, fabrication, integration, and applications of MEMS for all kinds of biomedical imaging. This Special Issue aims to collate and showcase research papers, short commutations, perspectives, and insightful review articles from esteemed colleagues that demonstrate: (1) original works on the topic of MEMS components or devices based on various kinds of mechanisms for biomedical imaging; and (2) new developments and potentials of applying MEMS technology of any kind in biomedical imaging. The objective of this special session is to provide insightful information regarding the technological advancements for the researchers in the community

Uterine transplantation: the move from the animal model into the human setting – surgical, reproductive and clinical aspects

Women with absolute uterine factor infertility (AUFI) are considered as being ‘unconditionally infertile’. Uterine transplantation (UTx) may be a possible treatment option in the future for such women. This thesis describes a number of key areas of research that are important in order to move closer to a successful and crucially, safe transplant in the human setting. Nine allogeneic transplants were carried out in a rabbit model to investigate anatomical and surgical aspects necessary for a successful UTx. An attempt to characterise and quantify the immunological mechanisms involved in allogeneic UTx (rejection patterns) was made. Out of the nine recipients, one was a long-term survivor. Embryo transfer was performed in this one doe with the aim of establishing pregnancy. Performing UTx in a large-animal model is necessary as the pelvis resembles a woman’s reproductive system more closely. In addition, the anastomotic technique is similar. Five sheep autotransplants were performed to further define surgical techniques. The anastomotic model was internal to external iliac vessel. Out of the five transplants, three sheep demonstrated adequate perfusion in the immediate post-operative period. Furthermore, the suitability of two different imaging modalities, pulse oximetry and multispectral imaging, for assessing uterine perfusion and extent of ischaemia were been studied in both the rabbit and sheep models. Biophotonics was also applied in the form of Endoscopic Laser Speckle Contrast Analysis to characterize blood flow in the two models. Both Multispectral Imaging and Endoscopic Laser Speckle Contrast Analysis have never been assessed before in a gynecological context. In order to transfer the concept of UTx to the human, we carried out a retrospective study of abdominal radical trachelectomy (ART) as a potential replacement for radical hysterectomy in patients with early stage cervical cancer desiring a fertility-sparing procedure. ART forms the foundation of the original work into aspects of UTx. This original body of research revolved around the potential blood supply to a uterus. Furthermore, an attempt has been made to analyse the motivations, aims and feelings of patients diagnosed with AUFI towards UTx. Forty patients were interviewed. The final study involved the evaluation of the perceptions of health care professionals towards UTx, with 528 participants surveyed.Open Acces