Towards Robot Autonomy in Medical Procedures Via Visual Localization and Motion Planning

Robots performing medical procedures with autonomous capabilities have the potential to positively effect patient care and healthcare system efficiency. These benefits can be realized by autonomous robots facilitating novel procedures, increasing operative efficiency, standardizing intra- and inter-physician performance, democratizing specialized care, and focusing the physician’s time on subtasks that best leverage their expertise. However, enabling medical robots to act autonomously in a procedural environment is extremely challenging. The deforming and unstructured nature of the environment, the lack of features in the anatomy, and sensor size constraints coupled with the millimeter level accuracy required for safe medical procedures introduce a host of challenges not faced by robots operating in structured environments such as factories or warehouses. Robot motion planning and localization are two fundamental abilities for enabling robot autonomy. Motion planning methods compute a sequence of safe and feasible motions for a robot to accomplish a specified task, where safe and feasible are defined by constraints with respect to the robot and its environment. Localization methods estimate the position and orientation of a robot in its environment. Developing such methods for medical robots that overcome the unique challenges in procedural environments is critical for enabling medical robot autonomy. In this dissertation, I developed and evaluated motion planning and localization algorithms towards robot autonomy in medical procedures. A majority of my work was done in the context of an autonomous medical robot built for enhanced lung nodule biopsy. First, I developed a dataset of medical environments spanning various organs and procedures to foster future research into medical robots and automation. I used this data in my own work described throughout this dissertation. Next, I used motion planning to characterize the capabilities of the lung nodule biopsy robot compared to existing clinical tools and I highlighted trade-offs in robot design considerations. Then, I conducted a study to experimentally demonstrate the benefits of the autonomous lung robot in accessing otherwise hard-to-reach lung nodules. I showed that the robot enables access to lung regions beyond the reach of existing clinical tools with millimeter-level accuracy sufficient for accessing the smallest clinically operable nodules. Next, I developed a localization method to estimate the bronchoscope’s position and orientation in the airways with respect to a preoperatively planned needle insertion pose. The method can be used by robotic bronchoscopy systems and by traditional manually navigated bronchoscopes. The method is designed to overcome challenges with tissue motion and visual homogeneity in the airways. I demonstrated the success of this method in simulated lungs undergoing respiratory motion and showed the method’s ability to generalize across patients.Doctor of Philosoph

Re-localisation of microscopic lesions in their macroscopic context for surgical instrument guidance

Optical biopsies interrogate microscopic structure in vivo with a 2mm diameter miniprobe placed in contact with the tissue for detection of lesions and assessment of disease progression. After detection, instruments are guided to the lesion location for a new optical interrogation, or for treatment, or for tissue excision during the same or a future examination. As the optical measurement can be considered as a point source of information at the surface of the tissue of interest, accurate guidance can be difficult. A method for re-localisation of the sampling point is, therefore, needed. The method presented in this thesis has been developed for biopsy site re-localisation during a surveillance examination of Barrett’s Oesophagus. The biopsy site, invisible macroscopically during conventional endoscopy, is re-localised in the target endoscopic image using epipolar lines derived from its locations given by the tip of the miniprobe visible in a series of reference endoscopic images. A confidence region can be drawn around the relocalised biopsy site from its uncertainty that is derived analytically. This thesis also presents a method to improve the accuracy of the epipolar lines derived for the biopsy site relocalisation using an electromagnetic tracking system. Simulations and tests on patient data identified the cases when the analytical uncertainty is a good approximation of the confidence region and showed that biopsy sites can be re-localised with accuracies better than 1mm. Studies on phantom and on porcine excised tissue demonstrated that an electromagnetic tracking system contributes to more accurate epipolar lines and re-localised biopsy sites for an endoscope displacement greater than 5mm. The re-localisation method can be applied to images acquired during different endoscopic examinations. It may also be useful for pulmonary applications. Finally, it can be combined with a Magnetic Resonance scanner which can steer cells to the biopsy site for tissue treatment

Activity profiling for minimally invasive surgery

Imperial Users onl

BTLD+:A BAYESIAN APPROACH TO TRACKING LEARNING DETECTION BY PARTS

The contribution proposed in this thesis focuses on this particular instance of the visual tracking problem, referred as Adaptive Ap- iv \ufffcpearance Tracking. We proposed different approaches based on the Tracking Learning Detection (TLD) decomposition proposed in [55]. TLD decomposes visual tracking into three components, namely the tracker, the learner and detector. The tracker and the detector are two competitive processes for target localization based on comple- mentary sources of informations. The former searches for local fea- tures between consecutive frames in order to localize the target; the latter exploits an on-line appearance model to detect confident hy- pothesis over the entire image. The learner selects the final solution among the provided hypothesis. It updates the target appearance model, if necessary, reinitialize the tracker and bootstraps the detec- tor\u2019s appearance model. In particular, we investigated different ap- proaches to enforce the TLD stability. First, we replaced the tracker component with a novel one based on mcmc particle filtering; after- wards, we proposed a robust appearance modeling component able to characterize deformable objects in static images; after all, we inte- grated a modeling component able to integrate local visual features learning into the whole approach, lying to a couple layered represen- tation of the target appearance

Bayesian image restoration and bacteria detection in optical endomicroscopy

Optical microscopy systems can be used to obtain high-resolution microscopic images of tissue cultures and ex vivo tissue samples. This imaging technique can be translated for in vivo, in situ applications by using optical fibres and miniature optics. Fibred optical endomicroscopy (OEM) can enable optical biopsy in organs inaccessible by any other imaging systems, and hence can provide rapid and accurate diagnosis in a short time. The raw data the system produce is difficult to interpret as it is modulated by a fibre bundle pattern, producing what is called the “honeycomb effect”. Moreover, the data is further degraded due to the fibre core cross coupling problem. On the other hand, there is an unmet clinical need for automatic tools that can help the clinicians to detect fluorescently labelled bacteria in distal lung images. The aim of this thesis is to develop advanced image processing algorithms that can address the above mentioned problems. First, we provide a statistical model for the fibre core cross coupling problem and the sparse sampling by imaging fibre bundles (honeycomb artefact), which are formulated here as a restoration problem for the first time in the literature. We then introduce a non-linear interpolation method, based on Gaussian processes regression, in order to recover an interpretable scene from the deconvolved data. Second, we develop two bacteria detection algorithms, each of which provides different characteristics. The first approach considers joint formulation to the sparse coding and anomaly detection problems. The anomalies here are considered as candidate bacteria, which are annotated with the help of a trained clinician. Although this approach provides good detection performance and outperforms existing methods in the literature, the user has to carefully tune some crucial model parameters. Hence, we propose a more adaptive approach, for which a Bayesian framework is adopted. This approach not only outperforms the proposed supervised approach and existing methods in the literature but also provides computation time that competes with optimization-based methods

Image-set, Temporal and Spatiotemporal Representations of Videos for Recognizing, Localizing and Quantifying Actions

This dissertation addresses the problem of learning video representations, which is defined here as transforming the video so that its essential structure is made more visible or accessible for action recognition and quantification. In the literature, a video can be represented by a set of images, by modeling motion or temporal dynamics, and by a 3D graph with pixels as nodes. This dissertation contributes in proposing a set of models to localize, track, segment, recognize and assess actions such as (1) image-set models via aggregating subset features given by regularizing normalized CNNs, (2) image-set models via inter-frame principal recovery and sparsely coding residual actions, (3) temporally local models with spatially global motion estimated by robust feature matching and local motion estimated by action detection with motion model added, (4) spatiotemporal models 3D graph and 3D CNN to model time as a space dimension, (5) supervised hashing by jointly learning embedding and quantization, respectively. State-of-the-art performances are achieved for tasks such as quantifying facial pain and human diving. Primary conclusions of this dissertation are categorized as follows: (i) Image set can capture facial actions that are about collective representation; (ii) Sparse and low-rank representations can have the expression, identity and pose cues untangled and can be learned via an image-set model and also a linear model; (iii) Norm is related with recognizability; similarity metrics and loss functions matter; (v) Combining the MIL based boosting tracker with the Particle Filter motion model induces a good trade-off between the appearance similarity and motion consistence; (iv) Segmenting object locally makes it amenable to assign shape priors; it is feasible to learn knowledge such as shape priors online from Web data with weak supervision; (v) It works locally in both space and time to represent videos as 3D graphs; 3D CNNs work effectively when inputted with temporally meaningful clips; (vi) the rich labeled images or videos help to learn better hash functions after learning binary embedded codes than the random projections. In addition, models proposed for videos can be adapted to other sequential images such as volumetric medical images which are not included in this dissertation

Suivi automatique d'instruments dans les séquences d'images thoracoscopiques

Anatomie de la structure thoracique -- Chirurgie minimalement invasive -- Contexte clinique -- Système de navigation pour le rachis -- Approches au suivi d'instruments chirurgicaux -- Techniques de suivi d'objets à travers une séquence d'images -- Objectifs spécifiques du projet -- Méthodologie -- Extraction des caractéristiques de l'instrument -- Suivi temporel par filtrage particulaire -- Méthode d'évaluation et de validation -- Interface graphique et mise en contexte -- Extraction des caractéristiques de l'instrument -- Suivi automatique des instruments -- Précision de la procédure de suivi -- Temps de calcul -- Limites de la méthode proposée

Recovery from community acquired pneumonia

RECOVERY FROM COMMUNITY ACQUIRED PNEUMONIA. A PhD Thesis by Daniel Gower Wootton Aims To measure symptomatic recovery over a year among an adult cohort recruited from hospital with community acquired pneumonia (CAP). To measure the host recovery mechanism efferocytosis and the diversity of the bacterial microbiota in sputum and relate these to individual characteristics of subjects in the cohort. Methods Patients with CAP were recruited from two hospitals in Liverpool, (UK) and were followed-up for one year. The CAP-sym questionnaire was completed at multiple time-points in order to create a statistical model of symptomatic recovery. DNA was extracted from acute sputum samples and 16S rRNA sequencing revealed the diversity of bacteria in sputum. At one month into recovery subjects volunteered for bronchoalveolar lavage and rates of efferocytosis were measured by co-culturing ex-vivo alveolar macrophages with apoptotic autologous neutrophils. Results The 169 subjects recruited with CAP were found to have high levels of socio-economic deprivation, smoking and COPD and the median age was 64 years. A non-linear, longitudinal, statistical model of symptoms found that smoking impaired recovery but people tended to describe better recovery as they got older. Efferocytosis was impaired by smoking but improved by statins and these effects were modified by body mass index. Those with prior pulmonary disease had lower bacterial diversity in their sputum and in this cohort a species from the genus Haemophilus was dominant. Conclusion This work proves the principal that modelling CAP-sym scores can be used to investigate factors associated with differential recovery from CAP. It highlights the detrimental effects of smoking on both recovery and efferocytosis. This is the first study to show that the bacterial diversity of CAP sputum is influenced by prior lung disease. The translational outcomes are the potential for trials of statins as pro-recovery agents and to study modified empirical antibiotics for those with CAP and prior-lung disease

Numerical Simulation in Biomechanics and Biomedical Engineering

In the first contribution, Morbiducci and co-workers discuss the theoretical and methodological bases supporting the Lagrangian- and Euler-based methods, highlighting their application to cardiovascular flows. The second contribution, by the Ansón and van Lenthe groups, proposes an automated virtual bench test for evaluating the stability of custom shoulder implants without the necessity of mechanical testing. Urdeitx and Doweidar, in the third paper, also adopt the finite element method for developing a computational model aim to study cardiac cell behavior under mechano-electric stimulation. In the fourth contribution, Ayensa-Jiménez et al. develop a methodology to approximate the multidimensional probability density function of the parametric analysis obtained developing a mathematical model of the cancer evolution. The fifth paper is oriented to the topological data analysis; the group of Cueto and Chinesta designs a predictive model capable of estimating the state of drivers using the data collected from motion sensors. In the sixth contribution, the Ohayon and Finet group uses wall shear stress-derived descriptors to study the role of recirculation in the arterial restenosis due to different malapposed and overlapping stent conditions. In the seventh contribution, the research group of Antón demonstrates that the simulation time can be reduced for cardiovascular numerical analysis considering an adequate geometry-reduction strategy applicable to truncated patient specific artery. In the eighth paper, Grasa and Calvo present a numerical model based on the finite element method for simulating extraocular muscle dynamics. The ninth paper, authored by Kahla et al., presents a mathematical mechano-pharmaco-biological model for bone remodeling. Martínez, Peña, and co-workers propose in the tenth paper a methodology to calibrate the dissection properties of aorta layer, with the aim of providing useful information for reliable numerical tools. In the eleventh contribution, Martínez-Bocanegra et al. present the structural behavior of a foot model using a detailed finite element model. The twelfth contribution is centered on the methodology to perform a finite, element-based, numerical model of a hydroxyapatite 3D printed bone scaffold. In the thirteenth paper, Talygin and Gorodkov present analytical expressions describing swirling jets for cardiovascular applications. In the fourteenth contribution, Schenkel and Halliday propose a novel non-Newtonian particle transport model for red blood cells. Finally, Zurita et al. propose a parametric numerical tool for analyzing a silicone customized 3D printable trachea-bronchial prosthesis