Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions

Global rigid registration of CT to video in laparoscopic liver surgery

PURPOSE: Image-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively. METHODS: We propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data. RESULTS: We validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms. CONCLUSION: The proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery

Applying artificial intelligence to big data in hepatopancreatic and biliary surgery: a scoping review

Aim: Artificial Intelligence (AI) and its applications in healthcare are rapidly developing. The healthcare industry generates ever-increasing volumes of data that should be used to improve patient care. This review aims to examine the use of AI and its applications in hepatopancreatic and biliary (HPB) surgery, highlighting studies leveraging large datasets.Methods: A PRISMA-ScR compliant scoping review using Medline and Google Scholar databases was performed (5th August 2022). Studies focusing on the development and application of AI to HPB surgery were eligible for inclusion. We undertook a conceptual mapping exercise to identify key areas where AI is under active development for use in HPB surgery. We considered studies and concepts in the context of patient pathways - before surgery (including diagnostics), around the time of surgery (supporting interventions) and after surgery (including prognostication).Results: 98 studies were included. Most studies were performed in China or the USA (n = 45). Liver surgery was the most common area studied (n = 51). Research into AI in HPB surgery has increased rapidly in recent years, with almost two-thirds published since 2019 (61/98). Of these studies, 11 have focused on using “big data” to develop and apply AI models. Nine of these studies came from the USA and nearly all focused on the application of Natural Language Processing. We identified several critical conceptual areas where AI is under active development, including improving preoperative optimization, image guidance and sensor fusion-assisted surgery, surgical planning and simulation, natural language processing of clinical reports for deep phenotyping and prediction, and image-based machine learning.Conclusion: Applications of AI in HPB surgery primarily focus on image analysis and computer vision to address diagnostic and prognostic uncertainties. Virtual 3D and augmented reality models to support complex HPB interventions are also under active development and likely to be used in surgical planning and education. In addition, natural language processing may be helpful in the annotation and phenotyping of disease, leading to new scientific insights

Evaluation of a calibration rig for stereo laparoscopes

BACKGROUND: Accurate camera and hand-eye calibration are essential to ensure high quality results in image guided surgery applications. The process must also be able to be undertaken by a non-expert user in a surgical setting. PURPOSE: This work seeks to identify a suitable method for tracked stereo laparoscope calibration within theatre. METHODS: A custom calibration rig, to enable rapid calibration in a surgical setting, was designed. The rig was compared against freehand calibration. Stereo reprojection, stereo reconstruction, tracked stereo reprojection and tracked stereo reconstruction error metrics were used to evaluate calibration quality. RESULTS: Use of the calibration rig reduced mean errors: reprojection (1.47mm [SD 0.13] vs 3.14mm [SD 2.11], p-value 1e-8), reconstruction (1.37px [SD 0.10] vs 10.10px [SD 4.54], p-value 6e-7) and tracked reconstruction (1.38mm [SD 0.10] vs 12.64mm [SD 4.34], p-value 1e-6) compared with freehand calibration. The use of a ChArUco pattern yielded slightly lower reprojection errors, while a dot grid produced lower reconstruction errors and was more robust under strong global illumination. CONCLUSION: The use of the calibration rig results in a statistically significant decrease in calibration error metrics, versus freehand calibration, and represents the preferred approach for use in the operating theatre. This article is protected by copyright. All rights reserved

Near infrared fluorescence imaging in surgery:seeing more?

All operations involve a risk of accidental damage by the surgeon. The most common cause for this damage is incorrect distinction of the various tissues. This is often difficult because the different tissues are very similar and the anatomy in the living person does not always follow the anatomy book. This thesis explores the use of fluorescent imaging during surgery. This technique allows the surgeon to see certain parts of the anatomy light up as the camera settings of a viewing operation are adjusted, making it possible to better distinguish the different tissues. This dissertation investigates this technique for three common operations, namely 1. Gallbladder surgery 2. Colon surgery 3. Thyroid surgery This thesis investigates the practical applicability, added value and points for improvement of near infrared fluorescence imaging

Advanced Endoscopic Navigation:Surgical Big Data,Methodology,and Applications

随着科学技术的飞速发展，健康与环境问题日益成为人类面临的最重大问题之一。信息科学、计算机技术、电子工程与生物医学工程等学科的综合应用交叉前沿课题，研究现代工程技术方法，探索肿瘤癌症等疾病早期诊断、治疗和康复手段。本论文综述了计算机辅助微创外科手术导航、多模态医疗大数据、方法论及其临床应用：从引入微创外科手术导航概念出发，介绍了医疗大数据的术前与术中多模态医学成像方法、阐述了先进微创外科手术导航的核心流程包括计算解剖模型、术中实时导航方案、三维可视化方法及交互式软件技术，归纳了各类微创外科手术方法的临床应用。同时，重点讨论了全球各种手术导航技术在临床应用中的优缺点，分析了目前手术导航领域内的最新技术方法。在此基础上，提出了微创外科手术方法正向数字化、个性化、精准化、诊疗一体化、机器人化以及高度智能化的发展趋势。【Abstract】Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.X.L. acknowledges funding from the Fundamental Research Funds for the Central Universities. T.M.P. acknowledges funding from the Canadian Foundation for Innovation, the Canadian Institutes for Health Research, the National Sciences and Engineering Research Council of Canada, and a grant from Intuitive Surgical Inc

Novel imaging and image-guided therapy of prostate cancer

Whole-gland prostate surgery and radiotherapy, the established approaches to localised prostate cancer (PCa), usually cause substantial adverse effects. Targeted image-guided cancer therapy has gained acceptance through improved PCa detection, localization and characterization by magnetic resonance imaging (MRI) and prostate-specific membrane antigen positron emission tomography-computed tomography (PSMA PET-CT). Focal therapy offers a potentially better trade-off between disease control and preservation of genitourinary and bowel function. MRI-guided transurethral ultrasound ablation (TULSA), a recently introduced treatment modality, uses therapeutic ultrasound directed through the urethra to thermally ablate the prostate under real-time MRI control. The applicability of TULSA to focal therapy of primary PCa, palliative therapy of symptomatic locally advanced PCa, and treatment of locally radiorecurrent PCa was investigated in a prospective setting. TULSA was shown to be a safe and effective method for local PCa control. Thermal injury was restricted to the planned treatment volume. This method enabled whole-gland ablation and focal ablation anywhere in the prostate. Furthermore, TULSA achieved local symptom relief in palliative care and encouraging preliminary oncological control in salvage care. These promising phase 1 study results enabled progression to phase 2 studies of patients with localised PCa and salvage of patients with radiorecurrent PCa. The diagnostic accuracy of MRI and PSMA PET-CT was studied to determine the extent of primary PCa, to plan TULSA treatment and evaluate treatment response. PSMA PET-CT was found to be a more sensitive method for detecting metastatic disease and appeared to accurately reflect the extent of local disease before and after TULSA treatment. PSMA PET-CT appears to detect some falsepositive bone lesions. The advantages of using MRI and PSMA PET-CT in treatment planning and monitoring treatment response are under further investigation. These studies have shown 18F-PSMA-1007 PET-CT to be effective in PCadiagnosis and TULSA to be effective in PCa therapyModernit kuvantamismenetelmät ja kuvantamisohjatut hoidot eturauhassyövässä Vakiintuneet paikallisen eturauhassyövän (PCa) hoitomenetelmät, leikkaus ja sädehoito, kohdistuvat koko rauhaseen ja aiheuttavat merkittäviä haittavaikutuksia. Magneettikuvantamisella (MRI) ja eturauhassyövän entsyymikuvantamisella (PSMA PET-TT) PCa:n havaitseminen, paikallistaminen ja karakterisointi ovat tarkentuneet. Kohdennetut kuvantamisohjatut syöpähoidot ovat siksi saaneet hyväksynnän ja tarjoavat mahdollisesti optimaalisemman vaihtoehdon hoidon hyödyn ja sen virtsa- ja sukupuolielimiin kohdistuvien haittojen suhdetta ajatellen. MRI-ohjattu eturauhasen kuumennushoito (TULSA) on uusi menetelmä, jossa virtsaputken kautta kudosta tuhoavaa ultraääntä ohjataan eturauhaseen reaaliaikaisessa MRI-ohjauksessa ja -valvonnassa. TULSA:n käyttökelpoisuutta primaarin PCa:n kohdennetussa hoidossa, paikallisesti edenneen PCa:n palliatiivisessa hoidossa ja sädehoidon jälkeen paikallisesti uusiutuneen PCa:n hoidossa tutkittiin prospektiivisessa tutkimusasetelmassa. TULSA-menetelmän todettiin tuhoavan turvallisesti ja tehokkaasti eturauhaskudosta. Lämpövaurio rajautui suunnitellulle hoitoalueelle. Menetelmä mahdollisti kuumennushoidon käytön kaikkialla eturauhasessa, koko rauhasessa tai paikallisemmin. Lisäksi TULSA-hoito lievensi paikallisoireita palliatiivisilla potilailla ja oli tehokas sädehoidon jälkeen paikallisesti uusiutuneessa PCa:ssä. Lupaavien ensimmäisen vaiheen tutkimustulosten takia olemme siirtyneet toisen vaiheen tutkimuksiin näillä uusilla indikaatioilla. MRI:n ja PSMA PET-TT:n diagnostista tarkuutta tutkittiin primaarin PCa:n levinneisyyden selvittelyssä ja TULSA-hoidon suunnittelussa sekä hoitovasteen arvioinnissa. PSMA PET-TT:n havaittiin olevan herkempi menetelmä etäpesäkkeiden tunnistamisessa ja se näytti tarkasti taudin laajuuden ennen ja jälkeen TULSAhoidon. PSMA PET-TT tunnistaa myös vääriä positiivisia luustomuutoksia. MRI:n ja PSMA PET-TT:n kliinistä hyötyä TULSA-hoidon suunnittelussa ja hoitovasteen seurannassa tutkitaan edelleen. Tutkimuksemme ovat osoittaneet PSMA PET-TT:n hyödyllisyyden PCa:n diagnostiikassa ja TULSA:n turvallisuuden ja tehon PCa:n hoidossa

Development of an image guidance system for laparoscopic liver surgery and evaluation of optical and computer vision techniques for the assessment of liver tissue

Introduction: Liver resection is increasingly being carried out via the laparoscopic approach (keyhole surgery) because there is mounting evidence that it benefits patients by reducing pain and length of hospitalisation. There are however ongoing concerns about oncological radicality (i.e. ability to completely remove cancer) and an inability to control massive haemorrhage. These issues can partially be attributed to a loss of sensation such as depth perception, tactile feedback and a reduced field of view. Utilisation of optical imaging and computer vision may be able to compensate for some of the lost sensory input because these modalities can facilitate visualisation of liver tissue and structural anatomy. Their use in laparoscopy is attractive because it is easy to adapt or integrate with existing technology. The aim of this thesis is to explore to what extent this technology can aid in the detection of normal and abnormal liver tissue and structures. / Methods: The current state of the art for optical imaging and computer vision in laparoscopic liver surgery is assessed in a systematic review. Evaluation of confocal laser endomicroscopy is carried out on a murine and porcine model of liver disease. Multispectral near infrared imaging is evaluated on ex-vivo liver specimen. Video magnification is assessed on a mechanical flow phantom and a porcine model of liver disease. The latter model was also employed to develop a computer vision based image guidance system for laparoscopic liver surgery. This image guidance system is further evaluated in a clinical feasibility study. Where appropriate, experimental findings are substantiated with statistical analysis. / Results: Use of confocal laser endomicroscopy enabled discrimination between cancer and normal liver tissue with a sub-millimetre precision. This technology also made it possible to verify the adequacy of thermal liver ablation. Multispectral imaging, at specific wavelengths was shown to have the potential to highlight the presence of colorectal and hepatocellular cancer. An image reprocessing algorithm is proposed to simplify visual interpretation of the resulting images. It is shown that video magnification can determine the presence of pulsatile motion but that it cannot reliably determine the extent of motion. Development and performance metrics of an image guidance system for laparoscopic liver surgery are outlined. The system was found to improve intraoperative orientation more development work is however required to enable reliable prediction of oncological margins. / Discussion: The results in this thesis indicate that confocal laser endomicroscopy and image guidance systems have reached a development stage where their intraoperative use may benefit surgeons by visualising features of liver anatomy and tissue characteristics. Video magnification and multispectral imaging require more development and suggestions are made to direct this work. It is also highlighted that it is crucial to standardise assessment methods for these technologies which will allow a more direct comparison between the outcomes of different groups. Limited imaging depth is a major restriction of these technologies but this may be overcome by combining them with preoperatively obtained imaging data. Just like laparoscopy, optical imaging and computer vision use functions of light, a shared characteristic that makes their combined use complementary

The Challenge of Augmented Reality in Surgery

Imaging has revolutionized surgery over the last 50 years. Diagnostic imaging is a key tool for deciding to perform surgery during disease management; intraoperative imaging is one of the primary drivers for minimally invasive surgery (MIS), and postoperative imaging enables effective follow-up and patient monitoring. However, notably, there is still relatively little interchange of information or imaging modality fusion between these different clinical pathway stages. This book chapter provides a critique of existing augmented reality (AR) methods or application studies described in the literature using relevant examples. The aim is not to provide a comprehensive review, but rather to give an indication of the clinical areas in which AR has been proposed, to begin to explain the lack of clinical systems and to provide some clear guidelines to those intending pursue research in this area