A Template-Based 3D Reconstruction of Colon Structures and Textures from Stereo Colonoscopic Images

This article presents a framework for 3D reconstruction of colonic surface using stereo colonoscopic images. Due to the limited overlaps between consecutive frames and the nonexistence of large loop closures during a normal screening colonoscopy, the state-of-art simultaneous localization and mapping (SLAM) algorithms cannot be directly applied to this scenario, thus a colon model segmented from CT scans is used together with the colonosocopic images to achieve the colon 3D reconstruction with high accuracy. The proposed framework includes 3D scan (point cloud with RGB information) reconstruction from stereo images, a visual odometry (VO) based camera pose initialization module, a 3D registration scheme for matching texture scans to the segmented colon model, and a barycentric-based texture rendering module for mapping textures from colonoscopic images to the reconstructed colonic surface. A realistic simulator is developed using Unity to simulate the procedures of colonoscopy and used to provide experimental datasets in different scenarios. Experimental results demonstrate the good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness. Currently, the framework requires a pre-operative colon model as the template for colon reconstruction and can reconstruct 3D colon maps when the colon has no large deformation and the colon structure is clearly visible. The datasets used in this article and the developed simulator are made publicly available for other researchers to use (https://github.com/zsustc/colon_reconstruction_dataset)

Learning-based depth and pose prediction for 3D scene reconstruction in endoscopy

Colorectal cancer is the third most common cancer worldwide. Early detection and treatment of pre-cancerous tissue during colonoscopy is critical to improving prognosis. However, navigating within the colon and inspecting the endoluminal tissue comprehensively are challenging, and success in both varies based on the endoscopist's skill and experience. Computer-assisted interventions in colonoscopy show much promise in improving navigation and inspection. For instance, 3D reconstruction of the colon during colonoscopy could promote more thorough examinations and increase adenoma detection rates which are associated with improved survival rates. Given the stakes, this thesis seeks to advance the state of research from feature-based traditional methods closer to a data-driven 3D reconstruction pipeline for colonoscopy. More specifically, this thesis explores different methods that improve subtasks of learning-based 3D reconstruction. The main tasks are depth prediction and camera pose estimation. As training data is unavailable, the author, together with her co-authors, proposes and publishes several synthetic datasets and promotes domain adaptation models to improve applicability to real data. We show, through extensive experiments, that our depth prediction methods produce more robust results than previous work. Our pose estimation network trained on our new synthetic data outperforms self-supervised methods on real sequences. Our box embeddings allow us to interpret the geometric relationship and scale difference between two images of the same surface without the need for feature matches that are often unobtainable in surgical scenes. Together, the methods introduced in this thesis help work towards a complete, data-driven 3D reconstruction pipeline for endoscopy

Enhanced computer assisted detection of polyps in CT colonography

This thesis presents a novel technique for automatically detecting colorectal polyps in computed tomography colonography (CTC). The objective of the documented computer assisted diagnosis (CAD) technique is to deal with the issue of false positive detections without adversely affecting polyp detection sensitivity. The thesis begins with an overview of CTC and a review of the associated research areas, with particular attention given to CAD-CTC. This review identifies excessive false positive detections as a common problem associated with current CAD-CTC techniques. Addressing this problem constitutes the major contribution of this thesis. The documented CAD-CTC technique is trained with, and evaluated using, a series of clinical CTC data sets These data sets contain polyps with a range of different sizes and morphologies. The results presented m this thesis indicate the validity of the developed CAD-CTC technique and demonstrate its effectiveness m accurately detecting colorectal polyps while significantly reducing the number of false positive detections