One-sided transparency : a revolution in visualization.

Colorectal cancer is one of the leading causes of death in the world. Colonoscopy, the traditional procedure for detecting colorectal cancer, is very effective. It does have downsides, however - it is invasive, uncomfortable for the patient, and not available to some patients with certain conditions. Virtual colonoscopy has been developed in order to address these issues. A virtual colonoscopy (VC) is a non-invasive method for performing a colonoscopy by using medical imaging data to create a virtual representation of the colon. Previous virtual colonoscopy methods include fly-through, fly-over, flattening, and the unfolded cube method. Fly-through moves the camera through the inside of the colon, following a centerline from the length of the colon. Fly-over splits the colon into halves longitudinally, and flies a camera over each half. Flattening reduces the 3D colon model to a 2D image. The unfolded cube method flies a set of cameras along the centerline as in flythrough, but where flythrough had one camera looking along the centerline, the unfolded cube method presents views from six cameras. The six camera views are positioned in the pattern of an unfolded cube, which gives rise to the method’s name. This thesis will present a new method called one-sided transparency (OST). This is a method for visualizing virtual objects so that the interior surfaces can be viewed from the outside. OST has numerous improvements over existing VC methods, particularly when combined with fly-over methods. However, this thesis will also demonstrate that OST is not limited to fly-over nor even to VC, as it has applications in multiple fields. For quantitative evaluation, this thesis focused on comparing specific scenarios that OST excels in visualizing. Fly-through navigation has difficulties with polyps between haustral folds, and prior fly-over work had visual artifacts that degraded the quality of the final visualization. These and other specific cases are visualized using OST in order to highlight the power of this new technique. Additionally, the previous FO method had some significant drawbacks that are solved by the application of OST. These problems and their origins will be addressed, along with the way that OST solves them. This thesis will also explore potential applications for OST outside of VC. This will include a more general visualization of tubular objects. It will be shown that OST has the ability to highlight structural issues and deformities such as cracks and bumps. This has potential applications in medical fields outside of VC as well as in structural engineering. This will demonstrate OST’s usefulness as a general technique, even outside the context of VC. Finally, this thesis will present results regarding OST for VC. It will show that OST presents several advantages over previous VC methods. OST allows easier viewing of polyps in difficult locations and offers a more complete view of the colon. OST has a number of advantages over the existing fly-over method, including faster time-to-viewing, less sensitivity to centerline error, and improved accuracy in the separation of halves

Advanced Endoscopic Navigation:Surgical Big Data,Methodology,and Applications

随着科学技术的飞速发展，健康与环境问题日益成为人类面临的最重大问题之一。信息科学、计算机技术、电子工程与生物医学工程等学科的综合应用交叉前沿课题，研究现代工程技术方法，探索肿瘤癌症等疾病早期诊断、治疗和康复手段。本论文综述了计算机辅助微创外科手术导航、多模态医疗大数据、方法论及其临床应用：从引入微创外科手术导航概念出发，介绍了医疗大数据的术前与术中多模态医学成像方法、阐述了先进微创外科手术导航的核心流程包括计算解剖模型、术中实时导航方案、三维可视化方法及交互式软件技术，归纳了各类微创外科手术方法的临床应用。同时，重点讨论了全球各种手术导航技术在临床应用中的优缺点，分析了目前手术导航领域内的最新技术方法。在此基础上，提出了微创外科手术方法正向数字化、个性化、精准化、诊疗一体化、机器人化以及高度智能化的发展趋势。【Abstract】Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.X.L. acknowledges funding from the Fundamental Research Funds for the Central Universities. T.M.P. acknowledges funding from the Canadian Foundation for Innovation, the Canadian Institutes for Health Research, the National Sciences and Engineering Research Council of Canada, and a grant from Intuitive Surgical Inc

Surgical Subtask Automation for Intraluminal Procedures using Deep Reinforcement Learning

Intraluminal procedures have opened up a new sub-field of minimally invasive surgery that use flexible instruments to navigate through complex luminal structures of the body, resulting in reduced invasiveness and improved patient benefits. One of the major challenges in this field is the accurate and precise control of the instrument inside the human body. Robotics has emerged as a promising solution to this problem. However, to achieve successful robotic intraluminal interventions, the control of the instrument needs to be automated to a large extent. The thesis first examines the state-of-the-art in intraluminal surgical robotics and identifies the key challenges in this field, which include the need for safe and effective tool manipulation, and the ability to adapt to unexpected changes in the luminal environment. To address these challenges, the thesis proposes several levels of autonomy that enable the robotic system to perform individual subtasks autonomously, while still allowing the surgeon to retain overall control of the procedure. The approach facilitates the development of specialized algorithms such as Deep Reinforcement Learning (DRL) for subtasks like navigation and tissue manipulation to produce robust surgical gestures. Additionally, the thesis proposes a safety framework that provides formal guarantees to prevent risky actions. The presented approaches are evaluated through a series of experiments using simulation and robotic platforms. The experiments demonstrate that subtask automation can improve the accuracy and efficiency of tool positioning and tissue manipulation, while also reducing the cognitive load on the surgeon. The results of this research have the potential to improve the reliability and safety of intraluminal surgical interventions, ultimately leading to better outcomes for patients and surgeons

Virtualna kolonoskopija i 3d rekonstrukcije u bolesnika s karcinomom kolona i rektuma

Introduction: Virtual endoscopy of the colon or virtual colonoscopy (VC) is a technique for postprocessing high resolution helical CT datasets. 3D surface rendering volume reconstruction that simulates endoscopic views of the inner surface of the colon was performed in real time by means of computer generated ray casting. VC may be performed together with 3D volume rendering (3DVR). VC can be applied in teaching, diagnostics, intervention planning or intraoperative navigation and as a non-invasive technique it is particularly useful when the patient cannot tolerate classical fiberoptic endoscopy. Objectives: To implement VC as a new technique of high resolution helical CT data postprocessing in screening, preoperative diagnostics and management and to discuss advantages and disadvantages of the method. Methods: VC and 3DVR in space analysis were performed using Syngo 2006G platform by Siemens Medical Systems in: a 46-year-old man with a spastic colon and some intestinal diverticula and polyps, a 65-year-old man who underwentendoscopic removal of a large colorectal polyp, a 64-year-old women with large circular cancer stenosis of the rectum and right hepatic lobe metastatic lesion, and a 52-year-old man with a large endoluminal cancer mass in the sigmoid colon. Virtual colonoscopy was performed in real time using ray casting algorithm with space leaping acceleration method. Siemens Somatom Emotion 16 scanner was used for image acquisition. Postprocessing of the images stored in DICOM format was done by dual Xeon workstation. Results: Fly through algorithm and 3DVR were performed on data sets created from axial CT images collected from multislice helical CT scanner archived in DICOM format. Fly through and 3DVR postprocessing produced a series of images that were analyzed by one radiologist, two surgeons and two physicians experienced in virtual endoscopy. Conclusion: VC is a useful non-invasive method in the assessment of malignant and benign lesions of colon. It may provide useful additional information for a surgeon during preoperative management. Furthermore, it allows three dimensional visualization in the lumen beyond areas of narrowing or stenosis and it gives a highly accurate representation of colorectal lesions. Its disadvantages are that it does not provide histology, it requires an air-mucosa interface to produce an image and it cannot identify functional lesions. Using VC, clinicians can appreciate not only the intraluminal proliferation of the tumor but also the extraluminal extension of the mass and its relation to the surrounding organs.Uvod: Virtualna endoskopija kolona ili virtualna kolonoskopija (VC) je metoda postprocesiranja podataka dobivenih kompjutoriziranom tomografijom visoke razlučivosti. VC simulira pogled pravog endoskopa na unutrašnju površinu debelog crijeva, a izvodi se u realnom vremenu na računalnim radnim stanicama s pomoću računalnog algoritma “ray casting”. VC se može izvoditi zajedno sa trodimenzionalnim volumnim renderiranjem (3DVR) koje daje 3D prikaz promatranog dijela tijela. VC se može koristiti u edukaciji, dijagnostici, planiranju endoskopskih zahvata ili operacija, te intraoperacijskoj navigaciji. Kao neinvazivna tehnika VC je naročito korisna ako bolesnik ne može podnijeti klasičnu fiberoptičku endoskopiju ili je ova kontraindicirana.. Ciljevi rada: Uvođenje VC kao nove metode postprocesiranja podataka dobivenih spiralnom kompjutoriziranom tomografijom visoke razlučivosti u ranu dijagnostiku (screening), prijeoperacijsku dijagnostiku i obradu bolesnika, te prikaz prednosti i nedostataka ove metode. Metode rada: VC i 3DVR učinjene su s pomoću specijaliziranog programskog paketa Syngo 2006G Platform proizvođača Siemens Medical Systems u: četrdesetšestogodišnjeg muškarca sa spastičnim kolonom te nekoliko intestinalnih divertikula i polipa, šezdesetpetogodišnjeg muškarca koji je podvrgnut endoskopskom uklanjanju velikog kolorektalnog polipa, šezdesetčetverogodišnje žene s opsežnom cirkularnom stenozom rektuma i metastazama u desnom hepatalnom režnju, te pedesetdvogodišnjeg muškarca s velikom endoluminalnim karcinomom u sigmoidnom kolonu. Virtualna kolonoskopija napravljena je u realnom vremenu s pomoću “ray casting” algoritma uz “space leaping” metodu akceleracije. Siemens Somatom Emotion 16 CT uređaj primijenjen je za prikupljanje podataka. Postprocesiranje slikovnih CT prikaza pohranjenih u DICOM formatu napravljeno je na radnoj stanici s dva Xeon procesora. Rezultati: “Fly through” algoritam i 3DVR primijenjeni su nad skupom slikovnih podataka iz aksijalnih CT prikaza dobivenih višeslojnim CT uređajem i pohranjenim u DICOM formatu. VC i 3DVR postprocesiranje rezultiralo je serijom slikovnih prikaza koje su analizirali jedan radiolog, dva kirurga i dva liječnika s iskustvom u virtualnoj endoskopiji. Zaključak: VC je korisna neinvazivna tehnika za probir i procjenu benignih i malignih lezija kolona. VC može pružiti korisne dodatne informacije kirurgu prilikom prijeoperacijske obrade bolesnika. Nadalje, virtualna kolonoskopija pruža mogućnost trodimenzionalne vizualizacije lumena crijeva iza područja suženja ili stenoze te pruža vrlo pouzdan prikaz kolorektalnih lezija. Njezin nedostatak je da nije moguće napraviti biopsiju, zahtijeva granicu zrak-sluznica za generiranje slikovnog prikaza te ne može identificirati funkcionalne lezije. Primjenom ovih tehnika kliničari mogu procijeniti ne samo intraluminalnu proliferaciju tumora već i ekstraluminalno širenje tumorske mase te odnos prema okolnim anatomskim strukturama

Learning-based depth and pose prediction for 3D scene reconstruction in endoscopy

Colorectal cancer is the third most common cancer worldwide. Early detection and treatment of pre-cancerous tissue during colonoscopy is critical to improving prognosis. However, navigating within the colon and inspecting the endoluminal tissue comprehensively are challenging, and success in both varies based on the endoscopist's skill and experience. Computer-assisted interventions in colonoscopy show much promise in improving navigation and inspection. For instance, 3D reconstruction of the colon during colonoscopy could promote more thorough examinations and increase adenoma detection rates which are associated with improved survival rates. Given the stakes, this thesis seeks to advance the state of research from feature-based traditional methods closer to a data-driven 3D reconstruction pipeline for colonoscopy. More specifically, this thesis explores different methods that improve subtasks of learning-based 3D reconstruction. The main tasks are depth prediction and camera pose estimation. As training data is unavailable, the author, together with her co-authors, proposes and publishes several synthetic datasets and promotes domain adaptation models to improve applicability to real data. We show, through extensive experiments, that our depth prediction methods produce more robust results than previous work. Our pose estimation network trained on our new synthetic data outperforms self-supervised methods on real sequences. Our box embeddings allow us to interpret the geometric relationship and scale difference between two images of the same surface without the need for feature matches that are often unobtainable in surgical scenes. Together, the methods introduced in this thesis help work towards a complete, data-driven 3D reconstruction pipeline for endoscopy

Enhancing endoscopic navigation and polyp detection using artificial intelligence

Colorectal cancer (CRC) is one most common and deadly forms of cancer. It has a very high mortality rate if the disease advances to late stages however early diagnosis and treatment can be curative is hence essential to enhancing disease management. Colonoscopy is considered the gold standard for CRC screening and early therapeutic treatment. The effectiveness of colonoscopy is highly dependent on the operator’s skill, as a high level of hand-eye coordination is required to control the endoscope and fully examine the colon wall. Because of this, detection rates can vary between different gastroenterologists and technology have been proposed as solutions to assist disease detection and standardise detection rates. This thesis focuses on developing artificial intelligence algorithms to assist gastroenterologists during colonoscopy with the potential to ensure a baseline standard of quality in CRC screening. To achieve such assistance, the technical contributions develop deep learning methods and architectures for automated endoscopic image analysis to address both the detection of lesions in the endoscopic image and the 3D mapping of the endoluminal environment. The proposed detection models can run in real-time and assist visualization of different polyp types. Meanwhile the 3D reconstruction and mapping models developed are the basis for ensuring that the entire colon has been examined appropriately and to support quantitative measurement of polyp sizes using the image during a procedure. Results and validation studies presented within the thesis demonstrate how the developed algorithms perform on both general scenes and on clinical data. The feasibility of clinical translation is demonstrated for all of the models on endoscopic data from human participants during CRC screening examinations