Demostrador VSLAM con partes rígidas y deformables

Se ha implementado un sistema de VSLAM (Simultaneous Location and Mapping with Visual sensor) que procesa secuencias de vídeo que incluyen una escena con elementos tanto rígidos como deformables. El objetivo es un programa para procesar la secuencia de vídeo obteniendo un modelo 3D de la escena y la posición de la cámara respecto del modelo construido, todo ello en tiempo real a frecuencia de vídeo. Para ello se ha construido una escena física para el demostrador que contiene un material textil de gran riqueza visual y deformable bajo la acción de una fuerza. El marco del demostrador es rígido y se encuentra estático en la escena. Para construir el software se ha partido del sistema ORBSLAM, que es un sistema de VSLAM para escenas rígidas, extendiéndolo para que pueda procesar las escenas que contienen el demostrador. La parte no rígida se inicializa mediante una segmentación entre los puntos rígidos y deformables del mapa 3D del demostrador en reposo. Con los puntos identificados como deformables se construye el modelo de deformación, que consiste en una malla regular triangular. Cada frame se procesa secuencialmente, buscando y emparejando los puntos no rígidos del mapa en cada nueva imagen. Si el demostrador sufre deformaciones, aparecerá un error de reproyección en estos emparejamientos. Una optimización no lineal reduce este error modificando la posición de los nodos de la malla, estimando así la deformación ocurrida. El sistema ha sido validado experimentalmente y es capaz de estimar pequeñas deformaciones a frecuencia de vídeo

Dense optical flow estimation in colonoscopy images using an unsupervised learning approach

[Resumen] La colonoscopia es la técnica de referencia en la detección del cáncer colorrectal. Sin embargo, los métodos asistidos por ordenador no se usan mucho en estos procedimientos. Este trabajo se enmarca dentro del proyecto EndoMapper, que tiene como objetivo crear reconstrucciones del colon que puedan ser utilizadas para ayudar a los médicos o para llevar a cabo cirugía robotizada. La asociación de datos es un elemento clave en estos sistemas, realizando la asociación entre píxeles de la imagen para permitir posteriores reconstrucciones 3D. En este trabajo, evaluamos un método de estimación de flujo óptico denso sobre imágenes de colonoscopia, adaptándolo al dominio del colon mediante aprendizaje no supervisado. Diseñamos un conjunto de datos para training, validation y test a partir de las secuencias reales de colonoscopia del conjunto de datos de Endomapper. El modelo se ha re-entrenado obteniendo una versión adaptada al dominio del colon. La validación experimental muestra cómo el modelo entrenado puede estimar el flujo de manera robusta bajo cambios de iluminación. También muestra una capacidad excepcional para estimar el flujo entre imágenes de colonoscopia muy separadas con grandes rotaciones.[Abstract] Colonoscopy is the gold standard in colorectal cancer screening. However, computer-assisted interventional methods are not widely used in these procedures. The Endomapper project, in which this work is embedded, aims to create reconstructions of the colon that can be used to assist doctors or for robotic surgery. Data association is a key element in these systems, performing the association between image pixels to enable subsequent 3D reconstructions. In this work, we evaluated a dense optical flow method on colonoscopy images, adapting it to the colon domain by using unsupervised learning. We built a dataset for training, validation and test from the real colonoscopy sequences of the Endomapper dataset. The model has been re-trained obtaining a version adapted to the colon domain. Experimental validation shows how the trained model is able to estimate flow robustly under illumination changes. It also shows an exceptional ability to estimate flow between widely separated colonoscopy images with large rotations.Ministerio de Educación y Formación Profesional; 22CO1/006926Ministerio de Ciencia e Innovación; PID2021-127685NB-I00Gobierno de Aragón; DGA_T45-17

Evaluación y procesamiento de escenas médicas con sistema VSLAM no rígido

En este proyecto se ha adaptado y evaluado un sistema de VSLAM (Simultaneous Location and Mapping with Visual sensor) deformable monocular para el procesamiento de escenas médicas. El objetivo es implementar un algoritmo que procese secuencias médicas, obteniendo un modelo 3D deformable de la escena, así como la posición relativa de la cámara respecto del modelo. Para construir el software se modifica el algoritmo DefSLAM, un sistema VSLAM deformable monocular en etapa de desarrollo y que ha sido probado en una tela que se deforma. Se modifica y sintoniza para que pueda procesar secuencias médicas donde hay deformación.Para ello se realiza una primera sintonía, de manera que se tenga el sistema en funcionamiento en escenas médicas. Se define una métrica propia para evaluar la calidad de la geometría estimada, por medio de la estimación estéreo de la superficie, calculada en cada imagen de la secuencia.El sistema presenta algunas limitaciones que serán resueltas. Para reducir la deriva de escala, se propone modificar la política de inserción de keyframes. Para gestionar las oclusiones y reflejos que se puedan dar, se propone por un lado, incluir un prior de movimiento suave para la cámara, y por otro, la generación de unas máscaras que detecten las regiones ocluidas o que contengan reflejos, y eliminar los puntos detectados en ellas.El sistema ha sido validado experimentalmente y es capaz de procesar las secuencias médicas seleccionadas, estimando las deformaciones.La implementación se ha hecho en C++ y está disponible en un repositorio privado de GitHub.<br /

SimCol3D -- 3D Reconstruction during Colonoscopy Challenge

Colorectal cancer is one of the most common cancers in the world. While colonoscopy is an effective screening technique, navigating an endoscope through the colon to detect polyps is challenging. A 3D map of the observed surfaces could enhance the identification of unscreened colon tissue and serve as a training platform. However, reconstructing the colon from video footage remains unsolved due to numerous factors such as self-occlusion, reflective surfaces, lack of texture, and tissue deformation that limit feature-based methods. Learning-based approaches hold promise as robust alternatives, but necessitate extensive datasets. By establishing a benchmark, the 2022 EndoVis sub-challenge SimCol3D aimed to facilitate data-driven depth and pose prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022 in Singapore. Six teams from around the world and representatives from academia and industry participated in the three sub-challenges: synthetic depth prediction, synthetic pose prediction, and real pose prediction. This paper describes the challenge, the submitted methods, and their results. We show that depth prediction in virtual colonoscopy is robustly solvable, while pose estimation remains an open research question

Reuse your features: unifying retrieval and feature-metric alignment

We propose a compact pipeline to unify all the steps of Visual Localization: image retrieval, candidate re-ranking and initial pose estimation, and camera pose refinement. Our key assumption is that the deep features used for these individual tasks share common characteristics, so we should reuse them in all the procedures of the pipeline. Our DRAN (Deep Retrieval and image Alignment Network) is able to extract global descriptors for efficient image retrieval, use intermediate hierarchical features to re-rank the retrieval list and produce an intial pose guess, which is finally refined by means of a feature-metric optimization based on learned deep multi-scale dense features. DRAN is the first single network able to produce the features for the three steps of visual localization. DRAN achieves a competitive performance in terms of robustness and accuracy specially in extreme day-night changes.Comment: 8 pages, 6 figures. Submitted to RA-L with option to IROS 202

EndoMapper dataset of complete calibrated endoscopy procedures

Computer-assisted systems are becoming broadly used in medicine. In endoscopy, most research focuses on automatic detection of polyps or other pathologies, but localization and navigation of the endoscope is completely performed manually by physicians. To broaden this research and bring spatial Artificial Intelligence to endoscopies, data from complete procedures are needed. This data will be used to build a 3D mapping and localization systems that can perform special task like, for example, detect blind zones during exploration, provide automatic polyp measurements, guide doctors to a polyp found in a previous exploration and retrieve previous images of the same area aligning them for easy comparison. These systems will provide an improvement in the quality and precision of the procedures while lowering the burden on the physicians. This paper introduces the Endomapper dataset, the first collection of complete endoscopy sequences acquired during regular medical practice, including slow and careful screening explorations, making secondary use of medical data. Its original purpose is to facilitate the development and evaluation of VSLAM (Visual Simultaneous Localization and Mapping) methods in real endoscopy data. The first release of the dataset is composed of 59 sequences with more than 15 hours of video. It is also the first endoscopic dataset that includes both the computed geometric and photometric endoscope calibration with the original calibration videos. Meta-data and annotations associated to the dataset varies from anatomical landmark and description of the procedure labeling, tools segmentation masks, COLMAP 3D reconstructions, simulated sequences with groundtruth and meta-data related to special cases, such as sequences from the same patient. This information will improve the research in endoscopic VSLAM, as well as other research lines, and create new research lines.Comment: 11 pages, 7 figures, 4 table

SimCol3D - 3D Reconstruction during Colonoscopy Challenge

Colorectal cancer is one of the most common cancers in the world. While colonoscopy is an effective screening technique, navigating an endoscope through the colon to detect polyps is challenging. A 3D map of the observed surfaces could enhance the identification of unscreened colon tissue and serve as a training platform. However, reconstructing the colon from video footage remains unsolved due to numerous factors such as self-occlusion, reflective surfaces, lack of texture, and tissue deformation that limit feature-based methods. Learning-based approaches hold promise as robust alternatives, but necessitate extensive datasets. By establishing a benchmark, the 2022 EndoVis sub-challenge SimCol3D aimed to facilitate data-driven depth and pose prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022 in Singapore. Six teams from around the world and representatives from academia and industry participated in the three sub-challenges: synthetic depth prediction, synthetic pose prediction, and real pose prediction. This paper describes the challenge, the submitted methods, and their results. We show that depth prediction in virtual colonoscopy is robustly solvable, while pose estimation remains an open research question