Doctor of Philosophy

dissertationClosed-loop control of wireless capsule endoscopes is an active area of research because it would drastically improve screening of the gastrointestinal tract. Traditional endoscopic procedures are unable to view the entire gastrointestinal tract and current commercial wireless capsule endoscopes are limited in their effectiveness due to their passive nature. This dissertation advances the field of active capsule endoscopy by developing methods to localize the full six-degree-of-freedom (6-DOF) pose of a screw-type magnetic capsule while it is being propelled through a lumen (such as the small intestines) using an external rotating magnetic dipole. The same external magnetic dipole is utilized for both propulsion and localization. Hardware was designed and constructed to enable testing of the magnetic localization and propulsion methods, including a robotic end-effector used as the external actuator magnet, and a prototype capsule embedded with Hall-effect sensors. Due to the use of a rotating magnetic field for propulsion, at any given time, the capsule can be in one of three regimes: synchronously rotating with the applied field, in "step-out" where it is free to move but the external field is rotating too quickly for the capsule to remain synchronously rotating, or completely stationary. We show that it is only necessary to distinguish whether or not the capsule is synchronously rotating (i.e., a single localization method can be used for a capsule in either the step-out or stationary regimes). Two magnetic localization methods are developed. The first uses nonlinear least squares to estimate the capsule's pose when it has no (or approximately no) net motion (e.g., to find the initial capsule pose or when it is stuck in an intestinal fold). The second method estimates the 6-DOF capsule pose as it synchronously rotates with the applied magnetic field using a square-root variant of the Unscented Kalman filter. A simple process model is adopted that restricts the capsule's movement to translation along and rotation about its principle axis. The capsule is actively propelled forward or backward, but it is not actively steered, rather, steering is provided by the lumen. The propulsion parameters that transform magnetic force and torque to the capsule's spatial velocity and angular velocity are estimated with an additional square-root Unscented Kalman filter to enable the capsule to navigate heterogeneous environments such as the small intestines. An optimized localization-propulsion system is described using the two localization algorithms and prior work in screw-type magnetic capsule propulsion with a single rotating dipole field. The capsule's regime is determined and the corresponding localization method is employed. Based on the capsule's estimated pose and the current estimates of its propulsion parameters, the actuator magnet's pose relative to the capsule is optimized to maximize the capsule's forward propulsion. Using this system, our prototype magnetic capsule successfully completed U-shaped and S-shaped trajectories in fresh bovine intestines with an average forward velocity of 5.5mm/s and 3.5 mm/s, respectively. At this rate it would take approximately 18-30 minutes to traverse the 6 meters of a typical human small intestine

Control of Magnetic Continuum Robots for Endoscopy

The present thesis discusses the problem of magnetic actuation and control applied to millimetre-scale robots for endoluminal procedures. Magnetic actuation, given its remote manipulation capabilities, has the potential to overcome several limitations of current endoluminal procedures, such as the relatively large size, high sti�ness and limited dexterity of existing tools. The application of functional forces remotely facilitates the development of softer and more dexterous endoscopes, which can navigate with reduced discomfort for the patient. However, the solutions presented in literature are not always able to guarantee smooth navigation in complex and convoluted anatomical structures. This thesis aims at improving the navigational capabilities of magnetic endoluminal robots, towards achieving full autonomy. This is realized by introducing novel design, sensing and control approaches for magnetically actuated soft endoscopes and catheters. First, the application of accurate closed-loop control to a 1 Internal Permanent Magnet (IPM) endoscope was analysed. The proposed approach can guarantee better navigation capabilities, thanks to the manipulation of every mechanical Degree of Freedom (DOF) - 5 DOFs. Speci�cally, it was demonstrated that gravity can be balanced with su�cient accuracy to guarantee tip levitation. In this way contact is minimized and obstacle avoidance improved. Consequently, the overall navigation capabilities of the endoscope were enhanced for given application. To improve exploration of convoluted anatomical pathways, the design of magnetic endoscopes with multiple magnetic elements along their length was introduced. This approach to endoluminal device design can ideally allow manipulation along the full length; facilitating full shape manipulation, as compared to tip-only control. To facilitate the control of multiple magneto-mechanical DOFs along the catheters' length, a magnetic actuation method was developed based on the collaborative robotic manipulation of 2 External Permanent Magnets (EPMs). This method, compared to the state-of-the-art, facilitates large workspace and applied �eld, while guaranteeing dexterous actuation. Using this approach, it was demonstrated that it is possible to actuate up to 8 independent magnetic DOFs. In the present thesis, two di�erent applications are discussed and evaluated, namely: colonoscopy and navigational bronchoscopy. In the former, a single-IPM endoscopic approach is utilized. In this case, the anatomy is large enough to permit equipping the endoscope with a camera; allowing navigation by direct vision. Navigational bronchoscopy, on-the-other-hand, is performed in very narrow peripheral lumina, and navigation is informed via pre-operative imaging. The presented work demonstrates how the design of the magnetic catheters, informed by a pre-operative Computed Tomography (CT) scan, can mitigate the need for intra-operative imaging and, consequently, reduce radiation exposure for patients and healthcare workers. Speci�cally, an optimization routine to design the catheters is presented, with the aim of achieving follow-the-leader navigation without supervision. In both scenarios, analysis of how magnetic endoluminal devices can improve the current practice and revolutionize the future of medical diagnostics and treatment is presented and discussed

Learning-based depth and pose prediction for 3D scene reconstruction in endoscopy

Colorectal cancer is the third most common cancer worldwide. Early detection and treatment of pre-cancerous tissue during colonoscopy is critical to improving prognosis. However, navigating within the colon and inspecting the endoluminal tissue comprehensively are challenging, and success in both varies based on the endoscopist's skill and experience. Computer-assisted interventions in colonoscopy show much promise in improving navigation and inspection. For instance, 3D reconstruction of the colon during colonoscopy could promote more thorough examinations and increase adenoma detection rates which are associated with improved survival rates. Given the stakes, this thesis seeks to advance the state of research from feature-based traditional methods closer to a data-driven 3D reconstruction pipeline for colonoscopy. More specifically, this thesis explores different methods that improve subtasks of learning-based 3D reconstruction. The main tasks are depth prediction and camera pose estimation. As training data is unavailable, the author, together with her co-authors, proposes and publishes several synthetic datasets and promotes domain adaptation models to improve applicability to real data. We show, through extensive experiments, that our depth prediction methods produce more robust results than previous work. Our pose estimation network trained on our new synthetic data outperforms self-supervised methods on real sequences. Our box embeddings allow us to interpret the geometric relationship and scale difference between two images of the same surface without the need for feature matches that are often unobtainable in surgical scenes. Together, the methods introduced in this thesis help work towards a complete, data-driven 3D reconstruction pipeline for endoscopy

ADVANCED INTRAVASCULAR MAGNETIC RESONANCE IMAGING WITH INTERACTION

Intravascular (IV) Magnetic Resonance Imaging (MRI) is a specialized class of interventional MRI (iMRI) techniques that acquire MRI images through blood vessels to guide, identify and/or treat pathologies inside the human body which are otherwise difficult to locate and treat precisely. Here, interactions based on real-time computations and feedback are explored to improve the accuracy and efficiency of IVMRI procedures. First, an IV MRI-guided high-intensity focused ultrasound (HIFU) ablation method is developed for targeting perivascular pathology with minimal injury to the vessel wall. To take advantage of real-time feedback, a software interface is developed for monitoring thermal dose with real-time MRI thermometry, and an MRI-guided ablation protocol developed and tested on muscle and liver tissue ex vivo. It is shown that, with cumulative thermal dose monitored with MRI thermometry, lesion location and dimensions can be estimated consistently, and desirable thermal lesions can be achieved in animals in vivo. Second, to achieve fully interactive IV MRI, high-resolution real-time 10 frames-per-second (fps) MRI endoscopy is developed as an advance over prior methods of MRI endoscopy. Intravascular transmit-receive MRI endoscopes are fabricated for highly under-sampled radial-projection MRI in a clinical 3Tesla MRI scanner. Iterative nonlinear reconstruction is accelerated using graphics processor units (GPU) to achieve true real-time endoscopy visualization at the scanner. The results of high-speed MRI endoscopy at 6-10 fps are consistent with fully-sampled MRI endoscopy and histology, with feasibility demonstrated in vivo in a large animal model. Last, a general framework for automatic imaging contrast tuning over MRI protocol parameters is explored. The framework reveals typical signal patterns over different protocol parameters from calibration imaging data and applies this knowledge to design efficient acquisition strategies and predicts contrasts under unacquired protocols. An external computer in real-time communication with the MRI console is utilized for online processing and controlling MRI acquisitions. This workflow enables machine learning for optimizing acquisition strategies in general, and provides a foundation for efficiently tuning MRI protocol parameters to perform interventional MRI in the highly varying and interactive environments commonly in play. This work is loosely inspired by prior research on extremely accelerated MRI relaxometry using the minimal-acquisition linear algebraic modeling (SLAM) method

MEMS Technology for Biomedical Imaging Applications

Biomedical imaging is the key technique and process to create informative images of the human body or other organic structures for clinical purposes or medical science. Micro-electro-mechanical systems (MEMS) technology has demonstrated enormous potential in biomedical imaging applications due to its outstanding advantages of, for instance, miniaturization, high speed, higher resolution, and convenience of batch fabrication. There are many advancements and breakthroughs developing in the academic community, and there are a few challenges raised accordingly upon the designs, structures, fabrication, integration, and applications of MEMS for all kinds of biomedical imaging. This Special Issue aims to collate and showcase research papers, short commutations, perspectives, and insightful review articles from esteemed colleagues that demonstrate: (1) original works on the topic of MEMS components or devices based on various kinds of mechanisms for biomedical imaging; and (2) new developments and potentials of applying MEMS technology of any kind in biomedical imaging. The objective of this special session is to provide insightful information regarding the technological advancements for the researchers in the community

Towards Quantitative Endoscopy with Vision Intelligence

In this thesis, we work on topics related to quantitative endoscopy with vision-based intelligence. Specifically, our works revolve around the topic of video reconstruction in endoscopy, where many challenges exist, such as texture scarceness, illumination variation, multimodality, etc., and these prevent prior works from working effectively and robustly. To this end, we propose to combine the strength of expressivity of deep learning approaches and the rigorousness and accuracy of non-linear optimization algorithms to develop a series of methods to confront such challenges towards quantitative endoscopy. We first propose a retrospective sparse reconstruction method that can estimate a high-accuracy and density point cloud and high-completeness camera trajectory from a monocular endoscopic video with state-of-the-art performance. To enable this, replacing the role of a hand-crafted local descriptor, a deep image feature descriptor is developed to boost the feature matching performance in a typical sparse reconstruction algorithm. A retrospective surface reconstruction pipeline is then proposed to estimate a textured surface model from a monocular endoscopic video, where self-supervised depth and descriptor learning and surface fusion technique is involved. We show that the proposed method performs superior to a popular dense reconstruction method and the estimate reconstructions are in good agreement with the surface models obtained from CT scans. To align video-reconstructed surface models with pre-operative imaging such as CT, we introduce a global point cloud registration algorithm that is robust to resolution mismatch that often happens in such multi-modal scenarios. Specifically, a geometric feature descriptor is developed where a novel network normalization technique is used to help a 3D network produce more consistent and distinctive geometric features for samples with different resolutions. The proposed geometric descriptor achieves state-of-the-art performance, based on our evaluation. Last but not least, a real-time SLAM system that estimates a surface geometry and camera trajectory from a monocular endoscopic video is developed, where deep representations for geometry and appearance and non-linear factor graph optimization are used. We show that the proposed SLAM system performs favorably compared with a state-of-the-art feature-based SLAM system

A Novel Bio-Inspired Insertion Method for Application to Next Generation Percutaneous Surgical Tools

The use of minimally invasive techniques can dramatically improve patient outcome from neurosurgery, with less risk, faster recovery, and better cost effectiveness when compared to conventional surgical intervention. To achieve this, innovative surgical techniques and new surgical instruments have been developed. Nevertheless, the simplest and most common interventional technique for brain surgery is needle insertion for either diagnostic or therapeutic purposes. The work presented in this thesis shows a new approach to needle insertion into soft tissue, focussing on soft tissue-needle interaction by exploiting microtextured topography and the unique mechanism of a reciprocating motion inspired by the ovipositor of certain parasitic wasps. This thesis starts by developing a brain-like phantom which I was shown to have mechanical properties similar to those of neurological tissue during needle insertion. Secondly, a proof-of-concept of the bio-inspired insertion method was undertaken. Based on this finding, the novel method of a multi-part probe able to penetrate a soft substrate by reciprocal motion of each segment is derived. The advantages of the new insertion method were investigated and compared with a conventional needle insertion in terms of needle-tissue interaction. The soft tissue deformation and damage were also measured by exploiting the method of particle image velocimetry. Finally, the thesis proposes the possible clinical application of a biologically-inspired surface topography for deep brain electrode implantation. As an adjunct to this work, the reciprocal insertion method described here fuelled the research into a novel flexible soft tissue probe for percutaneous intervention, which is able to steer along curvilinear trajectories within a compliant medium. Aspects of this multi-disciplinary research effort on steerable robotic surgery are presented, followed by a discussion of the implications of these findings within the context of future work