Performance of image guided navigation in laparoscopic liver surgery – A systematic review

Background: Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations. Methods: Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format. Results: Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8–15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes. Conclusions: Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard

The value of Augmented Reality in surgery — A usability study on laparoscopic liver surgery

Augmented Reality (AR) is considered to be a promising technology for the guidance of laparoscopic liver surgery. By overlaying pre-operative 3D information of the liver and internal blood vessels on the laparoscopic view, surgeons can better understand the location of critical structures. In an effort to enable AR, several authors have focused on the development of methods to obtain an accurate alignment between the laparoscopic video image and the pre-operative 3D data of the liver, without assessing the benefit that the resulting overlay can provide during surgery. In this paper, we present a study that aims to assess quantitatively and qualitatively the value of an AR overlay in laparoscopic surgery during a simulated surgical task on a phantom setup. We design a study where participants are asked to physically localise pre-operative tumours in a liver phantom using three image guidance conditions — a baseline condition without any image guidance, a condition where the 3D surfaces of the liver are aligned to the video and displayed on a black background, and a condition where video see-through AR is displayed on the laparoscopic video. Using data collected from a cohort of 24 participants which include 12 surgeons, we observe that compared to the baseline, AR decreases the median localisation error of surgeons on non-peripheral targets from 25.8 mm to 9.2 mm. Using subjective feedback, we also identify that AR introduces usability improvements in the surgical task and increases the perceived confidence of the users. Between the two tested displays, the majority of participants preferred to use the AR overlay instead of navigated view of the 3D surfaces on a separate screen. We conclude that AR has the potential to improve performance and decision making in laparoscopic surgery, and that improvements in overlay alignment accuracy and depth perception should be pursued in the future

In vivo estimation of target registration errors during augmented reality laparoscopic surgery

PURPOSE: Successful use of augmented reality for laparoscopic surgery requires that the surgeon has a thorough understanding of the likely accuracy of any overlay. Whilst the accuracy of such systems can be estimated in the laboratory, it is difficult to extend such methods to the in vivo clinical setting. Herein we describe a novel method that enables the surgeon to estimate in vivo errors during use. We show that the method enables quantitative evaluation of in vivo data gathered with the SmartLiver image guidance system. METHODS: The SmartLiver system utilises an intuitive display to enable the surgeon to compare the positions of landmarks visible in both a projected model and in the live video stream. From this the surgeon can estimate the system accuracy when using the system to locate subsurface targets not visible in the live video. Visible landmarks may be either point or line features. We test the validity of the algorithm using an anatomically representative liver phantom, applying simulated perturbations to achieve clinically realistic overlay errors. We then apply the algorithm to in vivo data. RESULTS: The phantom results show that using projected errors of surface features provides a reliable predictor of subsurface target registration error for a representative human liver shape. Applying the algorithm to in vivo data gathered with the SmartLiver image-guided surgery system shows that the system is capable of accuracies around 12 mm; however, achieving this reliably remains a significant challenge. CONCLUSION: We present an in vivo quantitative evaluation of the SmartLiver image-guided surgery system, together with a validation of the evaluation algorithm. This is the first quantitative in vivo analysis of an augmented reality system for laparoscopic surgery

Automatic registration of 3D models to laparoscopic video images for guidance during liver surgery

Laparoscopic liver interventions offer significant advantages over open surgery, such as less pain and trauma, and shorter recovery time for the patient. However, they also bring challenges for the surgeons such as the lack of tactile feedback, limited field of view and occluded anatomy. Augmented reality (AR) can potentially help during laparoscopic liver interventions by displaying sub-surface structures (such as tumours or vasculature). The initial registration between the 3D model extracted from the CT scan and the laparoscopic video feed is essential for an AR system which should be efficient, robust, intuitive to use and with minimal disruption to the surgical procedure. Several challenges of registration methods in laparoscopic interventions include the deformation of the liver due to gas insufflation in the abdomen, partial visibility of the organ and lack of prominent geometrical or texture-wise landmarks. These challenges are discussed in detail and an overview of the state of the art is provided. This research project aims to provide the tools to move towards a completely automatic registration. Firstly, the importance of pre-operative planning is discussed along with the characteristics of the liver that can be used in order to constrain a registration method. Secondly, maximising the amount of information obtained before the surgery, a semi-automatic surface based method is proposed to recover the initial rigid registration irrespective of the position of the shapes. Finally, a fully automatic 3D-2D rigid global registration is proposed which estimates a global alignment of the pre-operative 3D model using a single intra-operative image. Moving towards incorporating the different liver contours can help constrain the registration, especially for partial surfaces. Having a robust, efficient AR system which requires no manual interaction from the surgeon will aid in the translation of such approaches to the clinics

Enhanced Surgeons: Understanding the Design of Augmented Reality Instructions for Keyhole Surgery

It is important to understand how to design AR content for surgical contexts to mitigate the risk of distracting the surgeons. In this work, we test information overlays for AR guidance during keyhole surgery. We performed a preliminary evaluation of a prototype, focusing on the effects of colour, opacity, and information representation. Our work contributes insights into the design of AR guidance in surgery settings and a foundation for future research on visualisation design for surgical AR

Global rigid registration of CT to video in laparoscopic liver surgery

PURPOSE: Image-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively. METHODS: We propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data. RESULTS: We validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms. CONCLUSION: The proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery

Visual Perception and Cognition in Image-Guided Intervention

Surgical image visualization and interaction systems can dramatically affect the efficacy and efficiency of surgical training, planning, and interventions. This is even more profound in the case of minimally-invasive surgery where restricted access to the operative field in conjunction with limited field of view necessitate a visualization medium to provide patient-specific information at any given moment. Unfortunately, little research has been devoted to studying human factors associated with medical image displays and the need for a robust, intuitive visualization and interaction interfaces has remained largely unfulfilled to this day. Failure to engineer efficient medical solutions and design intuitive visualization interfaces is argued to be one of the major barriers to the meaningful transfer of innovative technology to the operating room. This thesis was, therefore, motivated by the need to study various cognitive and perceptual aspects of human factors in surgical image visualization systems, to increase the efficiency and effectiveness of medical interfaces, and ultimately to improve patient outcomes. To this end, we chose four different minimally-invasive interventions in the realm of surgical training, planning, training for planning, and navigation: The first chapter involves the use of stereoendoscopes to reduce morbidity in endoscopic third ventriculostomy. The results of this study suggest that, compared with conventional endoscopes, the detection of the basilar artery on the surface of the third ventricle can be facilitated with the use of stereoendoscopes, increasing the safety of targeting in third ventriculostomy procedures. In the second chapter, a contour enhancement technique is described to improve preoperative planning of arteriovenous malformation interventions. The proposed method, particularly when combined with stereopsis, is shown to increase the speed and accuracy of understanding the spatial relationship between vascular structures. In the third chapter, an augmented-reality system is proposed to facilitate the training of planning brain tumour resection. The results of our user study indicate that the proposed system improves subjects\u27 performance, particularly novices\u27, in formulating the optimal point of entry and surgical path independent of the sensorimotor tasks performed. In the last chapter, the role of fully-immersive simulation environments on the surgeons\u27 non-technical skills to perform vertebroplasty procedure is investigated. Our results suggest that while training surgeons may increase their technical skills, the introduction of crisis scenarios significantly disturbs the performance, emphasizing the need of realistic simulation environments as part of training curriculum

Development of an image guidance system for laparoscopic liver surgery and evaluation of optical and computer vision techniques for the assessment of liver tissue

Introduction: Liver resection is increasingly being carried out via the laparoscopic approach (keyhole surgery) because there is mounting evidence that it benefits patients by reducing pain and length of hospitalisation. There are however ongoing concerns about oncological radicality (i.e. ability to completely remove cancer) and an inability to control massive haemorrhage. These issues can partially be attributed to a loss of sensation such as depth perception, tactile feedback and a reduced field of view. Utilisation of optical imaging and computer vision may be able to compensate for some of the lost sensory input because these modalities can facilitate visualisation of liver tissue and structural anatomy. Their use in laparoscopy is attractive because it is easy to adapt or integrate with existing technology. The aim of this thesis is to explore to what extent this technology can aid in the detection of normal and abnormal liver tissue and structures. / Methods: The current state of the art for optical imaging and computer vision in laparoscopic liver surgery is assessed in a systematic review. Evaluation of confocal laser endomicroscopy is carried out on a murine and porcine model of liver disease. Multispectral near infrared imaging is evaluated on ex-vivo liver specimen. Video magnification is assessed on a mechanical flow phantom and a porcine model of liver disease. The latter model was also employed to develop a computer vision based image guidance system for laparoscopic liver surgery. This image guidance system is further evaluated in a clinical feasibility study. Where appropriate, experimental findings are substantiated with statistical analysis. / Results: Use of confocal laser endomicroscopy enabled discrimination between cancer and normal liver tissue with a sub-millimetre precision. This technology also made it possible to verify the adequacy of thermal liver ablation. Multispectral imaging, at specific wavelengths was shown to have the potential to highlight the presence of colorectal and hepatocellular cancer. An image reprocessing algorithm is proposed to simplify visual interpretation of the resulting images. It is shown that video magnification can determine the presence of pulsatile motion but that it cannot reliably determine the extent of motion. Development and performance metrics of an image guidance system for laparoscopic liver surgery are outlined. The system was found to improve intraoperative orientation more development work is however required to enable reliable prediction of oncological margins. / Discussion: The results in this thesis indicate that confocal laser endomicroscopy and image guidance systems have reached a development stage where their intraoperative use may benefit surgeons by visualising features of liver anatomy and tissue characteristics. Video magnification and multispectral imaging require more development and suggestions are made to direct this work. It is also highlighted that it is crucial to standardise assessment methods for these technologies which will allow a more direct comparison between the outcomes of different groups. Limited imaging depth is a major restriction of these technologies but this may be overcome by combining them with preoperatively obtained imaging data. Just like laparoscopy, optical imaging and computer vision use functions of light, a shared characteristic that makes their combined use complementary

An Augmented Reality Platform for Preoperative Surgical Planning

Researching in new technologies for diagnosis, planning and medical treatment have allowed the development of computer tools that provide new ways of representing data obtained from patient's medical images such as computed tomography (CT) and magnetic resonance imaging (MRI). In this sense, augmented reality (AR) technologies provide a new form of data representation by combining the common analysis using images and the ability to superimpose virtual 3D representations of the organs of the human body in the real environment. In this paper the development of a generic computer platform based on augmented reality technology for surgical preoperative planning is presented. In particular, the surgeon can navigate in the 3D models of the patient's organs in order to have the possibility to perfectly understand the anatomy and plan in the best way the surgical procedure. In addition, a touchless interaction with the virtual organs is available thanks to the use of an armband provided of electromiographic muscle sensors. To validate the system, we focused in a navigation through aorta artery for mitral valve repair surgery