Augmented reality technology in image-guided therapy: State-of-the-art review.

Image-guided therapies have been on the rise in recent years as they can achieve higher accuracy and are less invasive than traditional methods. By combining augmented reality technology with image-guided therapy, more organs, and tissues can be observed by surgeons to improve surgical accuracy. In this review, 233 publications (dated from 2015 to 2020) on the design and application of augmented reality-based systems for image-guided therapy, including both research prototypes and commercial products, were considered for review. Based on their functions and applications. Sixteen studies were selected. The engineering specifications and applications were analyzed and summarized for each study. Finally, future directions and existing challenges in the field were summarized and discussed

Global rigid registration of CT to video in laparoscopic liver surgery

PURPOSE: Image-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively. METHODS: We propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data. RESULTS: We validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms. CONCLUSION: The proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery

Software Framework for Customized Augmented Reality Headsets in Medicine

The growing availability of self-contained and affordable augmented reality headsets such as the Microsoft HoloLens is encouraging the adoption of these devices also in the healthcare sector. However, technological and human-factor limitations still hinder their routine use in clinical practice. Among them, the major drawbacks are due to their general-purpose nature and to the lack of a standardized framework suited for medical applications and devoid of platform-dependent tracking techniques and/or complex calibration procedures. To overcome such limitations, in this paper we present a software framework that is designed to support the development of augmented reality applications for custom-made head-mounted displays designed to aid high-precision manual tasks. The software platform is highly configurable, computationally efficient, and it allows the deployment of augmented reality applications capable to support in situ visualization of medical imaging data. The framework can provide both optical and video see-through-based augmentations and it features a robust optical tracking algorithm. An experimental study was designed to assess the efficacy of the platform in guiding a simulated task of surgical incision. In the experiments, the user was asked to perform a digital incision task, with and without the aid of the augmented reality headset. The task accuracy was evaluated by measuring the similarity between the traced curve and the planned one. The average error in the augmented reality tests was < 1 mm. The results confirm that the proposed framework coupled with the new-concept headset may boost the integration of augmented reality headsets into routine clinical practice

Advanced Endoscopic Navigation:Surgical Big Data,Methodology,and Applications

随着科学技术的飞速发展，健康与环境问题日益成为人类面临的最重大问题之一。信息科学、计算机技术、电子工程与生物医学工程等学科的综合应用交叉前沿课题，研究现代工程技术方法，探索肿瘤癌症等疾病早期诊断、治疗和康复手段。本论文综述了计算机辅助微创外科手术导航、多模态医疗大数据、方法论及其临床应用：从引入微创外科手术导航概念出发，介绍了医疗大数据的术前与术中多模态医学成像方法、阐述了先进微创外科手术导航的核心流程包括计算解剖模型、术中实时导航方案、三维可视化方法及交互式软件技术，归纳了各类微创外科手术方法的临床应用。同时，重点讨论了全球各种手术导航技术在临床应用中的优缺点，分析了目前手术导航领域内的最新技术方法。在此基础上，提出了微创外科手术方法正向数字化、个性化、精准化、诊疗一体化、机器人化以及高度智能化的发展趋势。【Abstract】Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.X.L. acknowledges funding from the Fundamental Research Funds for the Central Universities. T.M.P. acknowledges funding from the Canadian Foundation for Innovation, the Canadian Institutes for Health Research, the National Sciences and Engineering Research Council of Canada, and a grant from Intuitive Surgical Inc

Augmented reality in open surgery

Augmented reality (AR) has been successfully providing surgeons an extensive visual information of surgical anatomy to assist them throughout the procedure. AR allows surgeons to view surgical field through the superimposed 3D virtual model of anatomical details. However, open surgery presents new challenges. This study provides a comprehensive overview of the available literature regarding the use of AR in open surgery, both in clinical and simulated settings. In this way, we aim to analyze the current trends and solutions to help developers and end/users discuss and understand benefits and shortcomings of these systems in open surgery. We performed a PubMed search of the available literature updated to January 2018 using the terms (1) “augmented reality” AND “open surgery”, (2) “augmented reality” AND “surgery” NOT “laparoscopic” NOT “laparoscope” NOT “robotic”, (3) “mixed reality” AND “open surgery”, (4) “mixed reality” AND “surgery” NOT “laparoscopic” NOT “laparoscope” NOT “robotic”. The aspects evaluated were the following: real data source, virtual data source, visualization processing modality, tracking modality, registration technique, and AR display type. The initial search yielded 502 studies. After removing the duplicates and by reading abstracts, a total of 13 relevant studies were chosen. In 1 out of 13 studies, in vitro experiments were performed, while the rest of the studies were carried out in a clinical setting including pancreatic, hepatobiliary, and urogenital surgeries. AR system in open surgery appears as a versatile and reliable tool in the operating room. However, some technological limitations need to be addressed before implementing it into the routine practice

Simultaneous computer-assisted assessment of mucosal and serosal perfusion in a model of segmental colonic ischemia

BACKGROUND: Fluorescence-based enhanced reality (FLER) enables the quantification of fluorescence signal dynamics, which can be superimposed onto real-time laparoscopic images by using a virtual perfusion cartogram. The current practice of perfusion assessment relies on visualizing the bowel serosa. The aim of this experimental study was to quantify potential differences in mucosal and serosal perfusion levels in an ischemic colon segment. METHODS: An ischemic colon segment was created in 12 pigs. Simultaneous quantitative mucosal and serosal fluorescence imaging was obtained via intravenous indocyanine green injection (0.2 mg/kg), using two near-infrared camera systems, and computer-assisted FLER analysis. Lactate levels were measured in capillary blood of the colonic wall at seven regions of interest (ROIs) as determined with FLER perfusion cartography: the ischemic zone (I), the proximal and distal vascularized areas (PV, DV), and the 50% perfusion threshold proximally and distally at the mucosal and serosal side (P50M, P50S, D50M, D50S). RESULTS: The mean ischemic zone as measured (mm) for the mucosal side was significantly larger than the serosal one (56.3 ± 21.3 vs. 40.8 ± 14.9, p = 0.001) with significantly lower lactate values at the mucosal ROIs. There was a significant weak inverse correlation between lactate and slope values for the defined ROIs (r = - 0.2452, p = 0.0246). CONCLUSIONS: Mucosal ischemic zones were larger than serosal zones. These results suggest that an assessment of bowel perfusion from the serosal side only can underestimate the extent of ischemia. Further studies are required to predict the optimal resection margin and anastomotic site

Automatic registration of 3D models to laparoscopic video images for guidance during liver surgery

Laparoscopic liver interventions offer significant advantages over open surgery, such as less pain and trauma, and shorter recovery time for the patient. However, they also bring challenges for the surgeons such as the lack of tactile feedback, limited field of view and occluded anatomy. Augmented reality (AR) can potentially help during laparoscopic liver interventions by displaying sub-surface structures (such as tumours or vasculature). The initial registration between the 3D model extracted from the CT scan and the laparoscopic video feed is essential for an AR system which should be efficient, robust, intuitive to use and with minimal disruption to the surgical procedure. Several challenges of registration methods in laparoscopic interventions include the deformation of the liver due to gas insufflation in the abdomen, partial visibility of the organ and lack of prominent geometrical or texture-wise landmarks. These challenges are discussed in detail and an overview of the state of the art is provided. This research project aims to provide the tools to move towards a completely automatic registration. Firstly, the importance of pre-operative planning is discussed along with the characteristics of the liver that can be used in order to constrain a registration method. Secondly, maximising the amount of information obtained before the surgery, a semi-automatic surface based method is proposed to recover the initial rigid registration irrespective of the position of the shapes. Finally, a fully automatic 3D-2D rigid global registration is proposed which estimates a global alignment of the pre-operative 3D model using a single intra-operative image. Moving towards incorporating the different liver contours can help constrain the registration, especially for partial surfaces. Having a robust, efficient AR system which requires no manual interaction from the surgeon will aid in the translation of such approaches to the clinics

Precision surgery:the role of intra-operative real-time image guidance - outcomes from a multidisciplinary European consensus conference

Developments within the field of image-guided surgery are ever expanding, driven by collective involvement of clinicians, researchers, and industry. While the general conception of the potential of image-guided surgery is to improve surgical outcome, the specific motives and goals that drive can differ between the different expert groups. To establish the current and future role of intra-operative image guidance within the field of image-guided surgery a Delphi consensus survey was conducted during the 2(nd) European Congress on Image-guided surgery. This multidisciplinary survey included questions on the conceptual potential and clinical value of image-guided surgery and was aimed at defining specific areas of research and development in the field in order to stimulate further advances towards precision surgery. Obtained results based on questionnaires filled in by 56 panel experts (clinicians: N=30, researchers: N=20 and industry: N=6) were discussed during a dedicated expert discussion session during the conference. The outcome of this Delphi consensus is indicative of the potential improvements offered by image-guided surgery and of the need for further research in this emerging field, that can be enriched by the identification of reliable molecular targets

Engineering precision surgery: Design and implementation of surgical guidance technologies

In the quest for precision surgery, this thesis introduces several novel detection and navigation modalities for the localization of cancer-related tissues in the operating room. The engineering efforts have focused on image-guided surgery modalities that use the complementary tracer signatures of nuclear and fluorescence radiation. The first part of the thesis covers the use of “GPS-like” navigation concepts to navigate fluorescence cameras during surgery, based on SPECT images of the patient. The second part of the thesis introduces several new imaging modalities such as a hybrid 3D freehand Fluorescence and freehand SPECT imaging and navigation device. Furthermore, to improve the detection of radioactive tracer-emissions during robot-assisted laparoscopic surgery, a tethered DROP-IN gamma probe is introduced. The clinical indications that are used to evaluate the new technologies were all focused on sentinel lymph node procedures in urology (i.e. prostate and penile cancer). Nevertheless, all presented techniques are of such a nature, that they can be applied to different surgical indications, including sentinel lymph node and tumor-receptor-targeted procedures, localization the primary tumor and metastatic spread. This will hopefully contribute towards more precise, less invasive and more effective surgical procedures in the field of oncology. Crystal Photonics GmbH Eurorad S.A. Intuitive Surgical Inc. KARL STORZ Endoscopie Nederland B.V. MILabs B.V. PI Medical Diagnostic Equipment B.V. SurgicEye GmbH Verb Surgical Inc.LUMC / Geneeskund