Tracking and Mapping in Medical Computer Vision: A Review

As computer vision algorithms are becoming more capable, their applications in clinical systems will become more pervasive. These applications include diagnostics such as colonoscopy and bronchoscopy, guiding biopsies and minimally invasive interventions and surgery, automating instrument motion and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing and applying algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. We then review datasets provided in the field and the clinical needs therein. Then, we delve in depth into the algorithmic side, and summarize recent developments, which should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. Finally, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications in the field. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.Comment: 31 pages, 17 figure

LightDepth: Single-View Depth Self-Supervision from Illumination Decline

Single-view depth estimation can be remarkably effective if there is enough ground-truth depth data for supervised training. However, there are scenarios, especially in medicine in the case of endoscopies, where such data cannot be obtained. In such cases, multi-view self-supervision and synthetic-to-real transfer serve as alternative approaches, however, with a considerable performance reduction in comparison to supervised case. Instead, we propose a single-view self-supervised method that achieves a performance similar to the supervised case. In some medical devices, such as endoscopes, the camera and light sources are co-located at a small distance from the target surfaces. Thus, we can exploit that, for any given albedo and surface orientation, pixel brightness is inversely proportional to the square of the distance to the surface, providing a strong single-view self-supervisory signal. In our experiments, our self-supervised models deliver accuracies comparable to those of fully supervised ones, while being applicable without depth ground-truth data

NR-SLAM: Non-Rigid Monocular SLAM

In this paper we present NR-SLAM, a novel non-rigid monocular SLAM system founded on the combination of a Dynamic Deformation Graph with a Visco-Elastic deformation model. The former enables our system to represent the dynamics of the deforming environment as the camera explores, while the later allows us to model general deformations in a simple way. The presented system is able to automatically initialize and extend a map modeled by a sparse point cloud in deforming environments, that is refined with a sliding-window Deformable Bundle Adjustment. This map serves as base for the estimation of the camera motion and deformation and enables us to represent arbitrary surface topologies, overcoming the limitations of previous methods. To assess the performance of our system in challenging deforming scenarios, we evaluate it in several representative medical datasets. In our experiments, NR-SLAM outperforms previous deformable SLAM systems, achieving millimeter reconstruction accuracy and bringing automated medical intervention closer. For the benefit of the community, we make the source code public.Comment: 12 pages, 7 figures, submited to the IEEE Transactions on Robotics (T-RO

Enhancement of virtual colonoscopy system.

Colorectal cancer is the fourth most common cancer, and the fourth leading cause of cancer related death in the United States. It also happens to be one of the most preventable cancers provided an individual performs a regular screening. For years colonoscopy via colonoscope was the only method for colorectal cancer screening. In the past decade, colonography or virtual colonoscopy (VC) has become an alternative (or supplement) to the traditional colonoscopy. VC has become a much researched topic since its introduction in the mid-nineties. Various visualization methods have been introduced including: traditional flythrough, colon flattening, and unfolded-cube projection. In recent years, the CVIP Lab has introduced a patented visualization method for VC called flyover. This novel visualization method provides complete visualization of the large intestine without significant modification to the rendered three-dimensional model. In this thesis, a CVIP Lab VC interface was developed using Lab software to segment, extract the centerline, split (for flyover), and visualize the large intestine. This system includes adaptive level sets software to perform large intestine segmentation, and CVIP Lab patented curve skeletons software to extract the large intestine centerline. This software suite has not been combined in this manner before so the system stands as a unique contribution to the CVIP Lab colon project. The system is also a novel VC pipeline when compared to other academic and commercial VC methods. The complete system is capable of segmenting, finding the centerline, splitting, and visualizing a large intestine with a limited number of slices (~350 slices) for VC in approximately four and a half minutes. Complete CT scans were also validated with the centerline extraction external to the system (since the curve skeletons code used for centerline extraction cause memory exceptions because of high memory utilization)

Cost-effectiveness analysis of colorectal cancer screening strategies in Germany

Colorectal cancer (CRC) screening has been shown to contribute to the reduction in CRC incidence and mortality. To inform the CRC screening recommendations, it usually relies on models that are flexible to predict the effectiveness of various screening modalities and strategies from a lifetime perspective. To date, there are only two CRC microsimulation models considering the two CRC carcinogenesis pathways (adenoma-carcinoma and serrated neoplasia pathways). However, both are calibrated with grid search optimization methods, limiting their capability to account for parameter uncertainty. Furthermore, there is no cost-effectiveness analysis from a microsimulation model to assess the current German CRC screening program and to explore alternative strategies under different invitation approaches to improve screening participation. The aims of the present thesis are two-fold: (1) To construct an individual-level model encompassing both CRC development pathways, and to explore a Bayesian calibration method for CRC disease modeling; (2) To conduct an up-to-date cost-effectiveness analysis for evaluating the cost-effectiveness of various CRC screening strategies in the current German organized CRC screening program, and to thereby inform future CRC screening policies in Germany. A discrete event simulation model, DECAS, was thus developed in the R software. DECAS simulates the CRC natural history from the state of no lesions to precancerous lesions (adenoma or serrated polyps) and to pre-clinical and clinical CRCs in individuals with an average CRC risk and follows them up from the age of 20 to 90 or death, whichever occurs first. The rates of event happening were lesion-, age-, sex- and location-specific, and they were calibrated with a likelihood-free approximate Bayesian computation method, adaptive population Monte Carlo (APMC). The calibration took advantage of 74 prevalence data points from the German screening colonoscopy program, which consisted of 5.2 million average-risk screening participants in 2003-2014. The Bayesian calibration rendered 1,000 sets of posterior parameter samples, with which DECAS successfully reproduced the CRC incidence data from the German national cancer registry. After DECAS natural history model validation, the screening component was added to the DECAS model. If any lesions prior to the clinical cancer state are detected by the screening tests, individuals can be referred to or directly removed by colonoscopy. To further validate the predictive ability of DECAS regarding the CRC screening effects, external validations against two large randomized control trials on flexible sigmoidoscopy and guaiac fecal occult blood test and a large colonoscopy cohort study were performed. Additionally, cross validation against the three most widely used CRC screening models, the CISNET models, was conducted. DECAS demonstrated accurate predictions for CRC incidence and mortality reduction in the validation studies. The validated DECAS model was then used to evaluate the benefits, burdens, and harms of CRC screening strategies in Germany, including annual fecal immunochemical tests (FIT) for aged 50-54 years followed by two 10-yearly colonoscopies or biennial FIT from age 55-75 years for both sexes, and the new strategy allowing men to start the two 10-yearly colonoscopies from the age of 50 years. Alternative strategies including biennial FITs or 10-yearly colonoscopies from the age of 45 or 50 years, and combined strategies with annual FIT from the age of 45 followed by 10-yearly colonoscopies from the age of 50 were also evaluated. All strategies were evaluated under four scenarios: perfect adherence, low adherence under the current organized program with an invitation letter, improved adherence with an invitation letter and mail-out FITs, high (but imperfect) adherence with an invitation letter, mail-out FITs, and an additional reminder. All strategies were found to be cost-effective compared to no-screening across all four scenarios. Assuming perfect adherence and compared to no-screening, the screening strategies brought about a 34-75% CRC incidence reduction, a 52-80% CRC mortality reduction, 57-97 life-years gained, and 36-98 quality-adjusted life-years gained per 1,000 40-year-olds. All strategies were cost-saving, and they resulted in 809-3,240 colonoscopies needed and 1-4 colonoscopy complication cases per 1,000 40-year-olds. In scenarios with imperfect adherence, the benefits, burdens, and harms decreased with the participation rates. In the two mail-out FIT scenarios, the sent but unused FITs could amount up to 9,967 kits and caused an additional cost of €93,323 per 1,000 40-year-olds in the biennial FIT strategy starting at age 45 in the lower adherence scenario. Additionally, the strategy with sex-differentiated starting age for colonoscopy appeared to be more cost-effective than the equal-starting-age strategy. Both pure colonoscopy and FIT-colonoscopy combined strategies appeared to be more cost-effective than pure FIT ones. Three-time 10-yearly colonoscopies strategy starting from the age of 45 was deemed the most cost-effective across scenarios given the willingness-to-pay thresholds of €5,000-100,000. Overall, strategies starting from the age of 45 provided the best balance between benefits, burdens, and harms, which is consistent with recent recommendation changes from major US guidelines. The modeling evidence from the present thesis can, despite the uncertainty, serve as a basis to inform future policy making for CRC screening in Germany in the absence of long-term evidence for FIT and colonoscopy screening from clinical trials. Future research directions include a recalibration of DECAS with more efficient Bayesian algorithms and with more robust serrated polyp data when available. Moreover, the cost-effectiveness for more risk-stratified screening strategies other than sex-specific ones (e.g., with a priori individual risks) and alternative screening modalities (e.g., multitarget stool DNA test or computed tomography colonography) can be explored. Lastly, DECAS can also be used to analyze the public health and economic impacts of delayed CRC screening due to disruption by external forces, e.g., the COVID-19 pandemic

Segmentation and polyp detection in virtual colonoscopy : a complete system for computer aided diagnosis

El cancer colorectal es una de las mayores causas de muerte por cancer en el mundo. La deteccion temprana de polipos es fundamental para su tratamiento, permitiendo alcanzar tasas del 90% de curabilidad. La tecnica habitual para la deteccion de polipos, debido a su elevada performance, es la colonoscopia optica (tecnica invasiva y extremadamente cara). A mediados de los '90 surge la tecnica denominada colonoscopia virtual. Esta tecnica consiste en la reconstruccion 3D del colon a partir de cortes de tomografia computada. Es por ende una tecnica no invasiva, y relativamente barata, pero la cantidad de falsos positivos y falsos negativos producida por estos metodos esta muy por encima de los maximos aceptados en la practica medica. Los avances recientes en las tecnicas de imagenologia parecerian hacer posible la reduccion de estas tasas. Como consecuencia de esto, estamos asistiendo a un nuevo interes por la colonoscopia virtual. En este trabajo se presenta un sistema completo de diagnostico asistido por computadora. La primera etapa del sistema es la segmentacion, que consiste en la reconstruccion 3D de la superficie del colon a partir del volumen tomografico. El aporte principal en este paso es el suavizado de la imagen. A partir de la superficie, se detectan aquellas zonas candidatas de ser polipos mediante una estrategia multi-escala que permite delinear con precision la zona. Luego para cada candidato se extraen caracteristicas geometricas y de textura, que son calculadas tambien en el tejido que rodea la zona a efectos de compararlas. Finalmente las zonas candidatas se clasifican utilizando SVM. Los resultados obtenidos son prometedores, permitiendo detectar un 100% de los polipos mayoresColorectal cancer is the second leading cause of cancer-related death in the United States, and the third cause worldwide. The early detection of polyps is fundamental, allowing to reduce mortality rates up to 90%. Nowadays, optical colonoscopy is the most used detection method due in part to its relative high performance. Virtual Colonoscopy is a promising alternative technique that emerged in the 90's. It uses volumetric Computed Tomographic data of the cleansed and air-distended colon, and the examination is made by a specialist from the images in a computer. Therefore, this technique is less invasive and less expensive than optical colonoscopy, but up to now the false positive and false negative rates are above the accepted medical limits. Recent advances in imaging techniques have the potential to reduce these rates; consequently, we are currently re-experiencing an increasing interest in Virtual Colonoscopy. In this work we propose a complete pipeline for a Computer-Aided Detection algorithm. The system starts with a novel and simple segmentation step. We then introduce geometrical and textural features that take into account not only the candidate polyp region, but the surrounding area at multiple scales as well. This way, our proposed CAD algorithm is able to accurately detect candidate polyps by measuring local variations of these features. Candidate patches are then classi ed using SVM. The whole algorithm is completely automatic and produces state-of-the-art results, achieving 100% sensitivity for polyps greater than 6mm in size with less than one false positive per case, and 100% sensitivity for polyps greater than 3mm in size with 2:2 false positives per case

A Comparative Study on Polyp Classification and Localization from Colonoscopy Videos

Colorectal cancer is one of the most common types of cancer with a high mortality rate. It typically develops from small clumps of benign cells called polyp. The adenomatous polyp has a higher chance of developing into cancer compared to the hyperplastic polyp. Colonoscopy is the preferred procedure for colorectal cancer screening and to minimize its risk by performing a biopsy on found polyps. Thus, a good polyp detection model can assist physicians and increase the effectiveness of colonoscopy. Several models using handcrafted features and deep learning approaches have been proposed for the polyp detection task. In this study, we compare the performances of the previous state-of-the-art general object detection models for polyp detection and classification (into adenomatous and hyperplastic class). Specifically, we compare the performances of FasterRCNN, SSD, YOLOv3, RefineDet, RetinaNet, and FasterRCNN with DetNet backbone. This comparative study serves as an initial analysis of the effectiveness of these models and to choose a base model that we will improve further for polyp detection

3D Textured Surface Reconstruction from Endoscopic Video

Endoscopy enables high-resolution visualization of tissue texture and is a critical step in many clinical workflows, including diagnosis of infections, tumors or diseases and treatment planning for cancers. This includes my target problems of radiation treatment planning in the nasopharynx and pre-cancerous polyps screening and treatment in colonoscopy. However, an endoscopic video does not provide its information in 3D space, making it difficult to use for tumor localization, and it is inefficient to review. In addition, when there are incomplete camera observations of the organ surface, full surface coverage cannot be guaranteed in an endoscopic procedure, and unsurveyed regions can hardly be noticed in a continuous first-person perspective. This dissertation introduces a new imaging approach that we call endoscopography: an endoscopic video is reconstructed into a full 3D textured surface, which we call an endoscopogram. In this dissertation, I present two endoscopography techniques. One method is a combination of a frame-by-frame algorithmic 3D reconstruction method and a groupwise deformable surface registration method. My contribution is the innovative combination of the two methods that improves the temporal consistency of the frame-by-frame 3D reconstruction algorithm and eliminates the manual intervention that was needed in the deformable surface registration method. The combined method reconstructs an endoscopogram in an offline manner, and the information contained in the tissue texture in the endoscopogram can be transferred to a 3D image such as CT through a surface-to-surface registration. Then, through an interactive tool, the physician can draw directly on the endoscopogram surface to specify a tumor, which then can be automatically transferred to CT slices to aid tumor localization. The second method is a novel deep-learning-driven dense SLAM (simultaneous localization and mapping) system, called RNN-SLAM, that in real time can produce an endoscopogram with display of the unsurveyed regions. In particular, my contribution is the deep learning system in the RNN-SLAM, called RNN-DP. RNN-DP is a novel multi-view dense depth map and odometry estimation method that uses Recurrent Neural Networks (RNN) and trains utilizing multi-view image reprojection and forward-backward flow-consistency losses.Doctor of Philosoph

State of the Art in Dense Monocular Non-Rigid 3D Reconstruction

3D reconstruction of deformable (or non-rigid) scenes from a set of monocular2D image observations is a long-standing and actively researched area ofcomputer vision and graphics. It is an ill-posed inverse problem,since--without additional prior assumptions--it permits infinitely manysolutions leading to accurate projection to the input 2D images. Non-rigidreconstruction is a foundational building block for downstream applicationslike robotics, AR/VR, or visual content creation. The key advantage of usingmonocular cameras is their omnipresence and availability to the end users aswell as their ease of use compared to more sophisticated camera set-ups such asstereo or multi-view systems. This survey focuses on state-of-the-art methodsfor dense non-rigid 3D reconstruction of various deformable objects andcomposite scenes from monocular videos or sets of monocular views. It reviewsthe fundamentals of 3D reconstruction and deformation modeling from 2D imageobservations. We then start from general methods--that handle arbitrary scenesand make only a few prior assumptions--and proceed towards techniques makingstronger assumptions about the observed objects and types of deformations (e.g.human faces, bodies, hands, and animals). A significant part of this STAR isalso devoted to classification and a high-level comparison of the methods, aswell as an overview of the datasets for training and evaluation of thediscussed techniques. We conclude by discussing open challenges in the fieldand the social aspects associated with the usage of the reviewed methods.<br