Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: phantom studies

Introduction: Presenting visual feedback for image-guided surgery on a monitor requires the surgeon to perform time-consuming comparisons and diversion of sight and attention away from the patient. Deficiencies in previously developed augmented reality systems for image-guided surgery have, however, prevented the general acceptance of any one technique as a viable alternative to monitor displays. This work presents an evaluation of the feasibility and versatility of a novel augmented reality approach for the visualisation of surgical planning and navigation data. The approach, which utilises a portable image overlay device, was evaluated during integration into existing surgical navigation systems and during application within simulated navigated surgery scenarios. Methods: A range of anatomical models, surgical planning data and guidance information taken from liver surgery, cranio-maxillofacial surgery, orthopaedic surgery and biopsy were displayed on patient-specific phantoms, directly on to the patient's skin and on to cadaver tissue. The feasibility of employing the proposed augmented reality visualisation approach in each of the four tested clinical applications was qualitatively assessed for usability, visibility, workspace, line of sight and obtrusiveness. Results: The visualisation approach was found to assist in spatial understanding and reduced the need for sight diversion throughout the simulated surgical procedures. The approach enabled structures to be identified and targeted quickly and intuitively. All validated augmented reality scenes were easily visible and were implemented with minimal overhead. The device showed sufficient workspace for each of the presented applications, and the approach was minimally intrusiveness to the surgical scene. Conclusion: The presented visualisation approach proved to be versatile and applicable to a range of image-guided surgery applications, overcoming many of the deficiencies of previously described AR approaches. The approach presents an initial step towards a widely accepted alternative to monitor displays for the visualisation of surgical navigation dat

Automatic registration of 3D models to laparoscopic video images for guidance during liver surgery

Laparoscopic liver interventions offer significant advantages over open surgery, such as less pain and trauma, and shorter recovery time for the patient. However, they also bring challenges for the surgeons such as the lack of tactile feedback, limited field of view and occluded anatomy. Augmented reality (AR) can potentially help during laparoscopic liver interventions by displaying sub-surface structures (such as tumours or vasculature). The initial registration between the 3D model extracted from the CT scan and the laparoscopic video feed is essential for an AR system which should be efficient, robust, intuitive to use and with minimal disruption to the surgical procedure. Several challenges of registration methods in laparoscopic interventions include the deformation of the liver due to gas insufflation in the abdomen, partial visibility of the organ and lack of prominent geometrical or texture-wise landmarks. These challenges are discussed in detail and an overview of the state of the art is provided. This research project aims to provide the tools to move towards a completely automatic registration. Firstly, the importance of pre-operative planning is discussed along with the characteristics of the liver that can be used in order to constrain a registration method. Secondly, maximising the amount of information obtained before the surgery, a semi-automatic surface based method is proposed to recover the initial rigid registration irrespective of the position of the shapes. Finally, a fully automatic 3D-2D rigid global registration is proposed which estimates a global alignment of the pre-operative 3D model using a single intra-operative image. Moving towards incorporating the different liver contours can help constrain the registration, especially for partial surfaces. Having a robust, efficient AR system which requires no manual interaction from the surgeon will aid in the translation of such approaches to the clinics

Applying artificial intelligence to big data in hepatopancreatic and biliary surgery: a scoping review

Aim: Artificial Intelligence (AI) and its applications in healthcare are rapidly developing. The healthcare industry generates ever-increasing volumes of data that should be used to improve patient care. This review aims to examine the use of AI and its applications in hepatopancreatic and biliary (HPB) surgery, highlighting studies leveraging large datasets.Methods: A PRISMA-ScR compliant scoping review using Medline and Google Scholar databases was performed (5th August 2022). Studies focusing on the development and application of AI to HPB surgery were eligible for inclusion. We undertook a conceptual mapping exercise to identify key areas where AI is under active development for use in HPB surgery. We considered studies and concepts in the context of patient pathways - before surgery (including diagnostics), around the time of surgery (supporting interventions) and after surgery (including prognostication).Results: 98 studies were included. Most studies were performed in China or the USA (n = 45). Liver surgery was the most common area studied (n = 51). Research into AI in HPB surgery has increased rapidly in recent years, with almost two-thirds published since 2019 (61/98). Of these studies, 11 have focused on using “big data” to develop and apply AI models. Nine of these studies came from the USA and nearly all focused on the application of Natural Language Processing. We identified several critical conceptual areas where AI is under active development, including improving preoperative optimization, image guidance and sensor fusion-assisted surgery, surgical planning and simulation, natural language processing of clinical reports for deep phenotyping and prediction, and image-based machine learning.Conclusion: Applications of AI in HPB surgery primarily focus on image analysis and computer vision to address diagnostic and prognostic uncertainties. Virtual 3D and augmented reality models to support complex HPB interventions are also under active development and likely to be used in surgical planning and education. In addition, natural language processing may be helpful in the annotation and phenotyping of disease, leading to new scientific insights

An automatic virtual patient reconstruction from CT-scans for hepatic surgical planning

International audiencePROBLEM/BACKGROUND: In order to help hepatic surgical planning we perfected automatic 3D reconstruction of patients from conventional CT-scan, and interactive visualization and virtual resection tools. TOOLS AND METHODS: From a conventional abdominal CT-scan, we have developed several methods allowing the automatic 3D reconstruction of skin, bones, kidneys, lung, liver, hepatic lesions, and vessels. These methods are based on deformable modeling or thresholding algorithms followed by the application of mathematical morphological operators. From these anatomical and pathological models, we have developed a new framework for translating anatomical knowledge into geometrical and topological constraints. More precisely, our approach allows to automatically delineate the hepatic and portal veins but also to label the portal vein and finally to build an anatomical segmentation of the liver based on Couinaud definition which is currently used by surgeons all over the world. Finally, we have developed a user friendly interface for the 3D visualization of anatomical and pathological structures, the accurate evaluation of volumes and distances and for the virtual hepatic resection along a user-defined cutting plane. RESULTS: A validation study on a 30 patients database gives 2 mm of precision for liver delineation and less than 1 mm for all other anatomical and pathological structures delineation. An in vivo validation performed during surgery also showed that anatomical segmentation is more precise than the delineation performed by a surgeon based on external landmarks. This surgery planning system has been routinely used by our medical partner, and this has resulted in an improvement of the planning and performance of hepatic surgery procedures. CONCLUSION: We have developed new tools for hepatic surgical planning allowing a better surgery through an automatic delineation and visualization of anatomical and pathological structures. These tools represent a first step towards the development of an augmented reality system combined with computer assisted tele-robotical surgery

Essential updates 2020/2021 : Current topics of simulation and navigation in hepatectomy

With the development of three-dimensional (3D) simulation software, preoperative simulation technology is almost completely established. The remaining issue is how to recognize anatomy three-dimensionally. Extended reality is a newly developed technology with several merits for surgical application: no requirement for a sterilized display monitor, better spatial awareness, and the ability to share 3D images among all surgeons. Various technology or devices for intraoperative navigation have also been developed to support the safety and certainty of liver surgery. Consensus recommendations regarding indocyanine green fluorescence were determined in 2021. Extended reality has also been applied to intraoperative navigation, and artificial intelligence (AI) is one of the topics of real-time navigation. AI might overcome the problem of liver deformity with automatic registration. Including the issues described above, this article focuses on recent advances in simulation and navigation in liver surgery from 2020 to 2021

Mixed Reality in Modern Surgical and Interventional Practice: Narrative Review of the Literature

BACKGROUND Mixed reality (MR) and its potential applications have gained increasing interest within the medical community over the recent years. The ability to integrate virtual objects into a real-world environment within a single video-see-through display is a topic that sparks imagination. Given these characteristics, MR could facilitate preoperative and preinterventional planning, provide intraoperative and intrainterventional guidance, and aid in education and training, thereby improving the skills and merits of surgeons and residents alike. OBJECTIVE In this narrative review, we provide a broad overview of the different applications of MR within the entire spectrum of surgical and interventional practice and elucidate on potential future directions. METHODS A targeted literature search within the PubMed, Embase, and Cochrane databases was performed regarding the application of MR within surgical and interventional practice. Studies were included if they met the criteria for technological readiness level 5, and as such, had to be validated in a relevant environment. RESULTS A total of 57 studies were included and divided into studies regarding preoperative and interventional planning, intraoperative and interventional guidance, as well as training and education. CONCLUSIONS The overall experience with MR is positive. The main benefits of MR seem to be related to improved efficiency. Limitations primarily seem to be related to constraints associated with head-mounted display. Future directions should be aimed at improving head-mounted display technology as well as incorporation of MR within surgical microscopes, robots, and design of trials to prove superiority

The Realm of Oncological Lung Surgery: From Past to Present and Future Perspectives

In this chapter, a historical overview as well as an overview of state of the art of the surgical techniques for the treatment of lung cancer is outlined. The chapter focuses on the introduction of open surgery, video-assisted thoracic surgery (VATS), uniportal VATS (UVATS), and robotic-assisted thoracic surgery (RATS) techniques for lung resections. A short introduction on upcoming techniques and modalities is given. The currently available tools as three-dimensional (3D) computed tomography (CT), virtual reality, and endo-bronchial surgery will be discussed. Based on the current development, this chapter attempts to delineate the horizon of oncological lung surgery. The information is generated not only from the available literature, but also from the experiences of surgeons and other physicians as well as co-workers involved in lung cancer treatment around the world. This chapter can be seen as a general introduction to several aspects of oncological lung surgery