Exploiting Temporal Image Information in Minimally Invasive Surgery

Minimally invasive procedures rely on medical imaging instead of the surgeons direct vision. While preoperative images can be used for surgical planning and navigation, once the surgeon arrives at the target site real-time intraoperative imaging is needed. However, acquiring and interpreting these images can be challenging and much of the rich temporal information present in these images is not visible. The goal of this thesis is to improve image guidance for minimally invasive surgery in two main areas. First, by showing how high-quality ultrasound video can be obtained by integrating an ultrasound transducer directly into delivery devices for beating heart valve surgery. Secondly, by extracting hidden temporal information through video processing methods to help the surgeon localize important anatomical structures. Prototypes of delivery tools, with integrated ultrasound imaging, were developed for both transcatheter aortic valve implantation and mitral valve repair. These tools provided an on-site view that shows the tool-tissue interactions during valve repair. Additionally, augmented reality environments were used to add more anatomical context that aids in navigation and in interpreting the on-site video. Other procedures can be improved by extracting hidden temporal information from the intraoperative video. In ultrasound guided epidural injections, dural pulsation provides a cue in finding a clear trajectory to the epidural space. By processing the video using extended Kalman filtering, subtle pulsations were automatically detected and visualized in real-time. A statistical framework for analyzing periodicity was developed based on dynamic linear modelling. In addition to detecting dural pulsation in lumbar spine ultrasound, this approach was used to image tissue perfusion in natural video and generate ventilation maps from free-breathing magnetic resonance imaging. A second statistical method, based on spectral analysis of pixel intensity values, allowed blood flow to be detected directly from high-frequency B-mode ultrasound video. Finally, pulsatile cues in endoscopic video were enhanced through Eulerian video magnification to help localize critical vasculature. This approach shows particular promise in identifying the basilar artery in endoscopic third ventriculostomy and the prostatic artery in nerve-sparing prostatectomy. A real-time implementation was developed which processed full-resolution stereoscopic video on the da Vinci Surgical System

Surgical Video Motion Magnification with Suppression of Instrument Artefacts

Video motion magnification can make blood vessels in surgical video more apparent by exaggerating their pulsatile motion and could prevent inadvertent damage and bleeding due to their increased prominence. It could also indicate the success of restricting blood supply to an organ when using a vessel clamp. However, the direct application to surgical video could result in aberration artefacts caused by its sensitivity to residual motion from the surgical instruments and would impede its practical usage in the operating theatre. By storing the previously obtained jerk filter response of each spatial component of each image frame - both prior to surgical instrument introduction and adhering to a Eulerian frame of reference - it is possible to prevent such aberrations from occurring. The comparison of the current readings to the prior readings of a single cardiac cycle at the corresponding cycle point, are used to determine if motion magnification should be active for each spatial component of the surgical video at that given point in time. In this paper, we demonstrate this technique and incorporate a scaling variable to loosen the effect which accounts for variabilities and misalignments in the temporal domain. We present promising results on endoscopic transnasal transsphenoidal pituitary surgery with a quantitative comparison to recent methods using Structural Similarity (SSIM), as well as qualitative analysis by comparing spatio-temporal cross sections of the videos and individual frames

Surgical Video Motion Magnification with Suppression of Instrument Artefacts

Video motion magnification could directly highlight subsurface blood vessels in endoscopic video in order to prevent inadvertent damage and bleeding. Applying motion filters to the full surgical image is however sensitive to residual motion from the surgical instruments and can impede practical application due to aberration motion artefacts. By storing the temporal filter response from local spatial frequency information for a single cardiovascular cycle prior to tool introduction to the scene, a filter can be used to determine if motion magnification should be active for a spatial region of the surgical image. In this paper, we propose a strategy to reduce aberration due to non-physiological motion for surgical video motion magnification. We present promising results on endoscopic transnasal transsphenoidal pituitary surgery with a quantitative comparison to recent methods using Structural Similarity (SSIM), as well as qualitative analysis by comparing spatio-temporal cross sections of the videos and individual frames.Comment: Early accept to the Internation Conference on Medical Imaging Computing and Computer Assisted Intervention (MICCAI) 2020 Presentation available here: https://www.youtube.com/watch?v=kKI_Ygny76Q Supplementary video available here: https://www.youtube.com/watch?v=8DUkcHI149

STB-VMM: Swin Transformer Based Video Motion Magnification

The goal of video motion magnification techniques is to magnify small motions in a video to reveal previously invisible or unseen movement. Its uses extend from bio-medical applications and deepfake detection to structural modal analysis and predictive maintenance. However, discerning small motion from noise is a complex task, especially when attempting to magnify very subtle, often sub-pixel movement. As a result, motion magnification techniques generally suffer from noisy and blurry outputs. This work presents a new state-of-the-art model based on the Swin Transformer, which offers better tolerance to noisy inputs as well as higher-quality outputs that exhibit less noise, blurriness, and artifacts than prior-art. Improvements in output image quality will enable more precise measurements for any application reliant on magnified video sequences, and may enable further development of video motion magnification techniques in new technical fields.Comment: Code available at: https://github.com/RLado/STB-VM

Higher Order of Motion Magnification for Vessel Identification in Surgical Video

Locating vessels during surgery is critical for avoiding inadvertent damage, yet vasculature can be difficult to identify. Video motion magnification can potentially highlight vessels by exaggerating subtle motion embedded within the video to become perceivable to the surgeon. In this paper, we explore a physiological model of artery distension to extend motion magnification to incorporate higher orders of motion, leveraging the difference in acceleration over time (jerk) in pulsatile motion to highlight the vascular pulse wave. Our method is compared to first and second order motion based Eulerian video magnification algorithms. Using data from a surgical video retrieved during a robotic prostatectomy, we show that our method can accentuate cardio-physiological features and produce a more succinct and clearer video for motion magnification, with more similarities in areas without motion to the source video at large magnifications. We validate the approach with a Structure Similarity (SSIM) and Peak Signal to Noise Ratio (PSNR) assessment of three videos at an increasing working distance, using three different levels of optical magnification. Spatio-temporal cross sections are presented to show the effectiveness of our proposal and video samples are provided to demonstrates qualitatively our results

Advanced Endoscopic Navigation:Surgical Big Data,Methodology,and Applications

随着科学技术的飞速发展，健康与环境问题日益成为人类面临的最重大问题之一。信息科学、计算机技术、电子工程与生物医学工程等学科的综合应用交叉前沿课题，研究现代工程技术方法，探索肿瘤癌症等疾病早期诊断、治疗和康复手段。本论文综述了计算机辅助微创外科手术导航、多模态医疗大数据、方法论及其临床应用：从引入微创外科手术导航概念出发，介绍了医疗大数据的术前与术中多模态医学成像方法、阐述了先进微创外科手术导航的核心流程包括计算解剖模型、术中实时导航方案、三维可视化方法及交互式软件技术，归纳了各类微创外科手术方法的临床应用。同时，重点讨论了全球各种手术导航技术在临床应用中的优缺点，分析了目前手术导航领域内的最新技术方法。在此基础上，提出了微创外科手术方法正向数字化、个性化、精准化、诊疗一体化、机器人化以及高度智能化的发展趋势。【Abstract】Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.X.L. acknowledges funding from the Fundamental Research Funds for the Central Universities. T.M.P. acknowledges funding from the Canadian Foundation for Innovation, the Canadian Institutes for Health Research, the National Sciences and Engineering Research Council of Canada, and a grant from Intuitive Surgical Inc

Development of an image guidance system for laparoscopic liver surgery and evaluation of optical and computer vision techniques for the assessment of liver tissue

Introduction: Liver resection is increasingly being carried out via the laparoscopic approach (keyhole surgery) because there is mounting evidence that it benefits patients by reducing pain and length of hospitalisation. There are however ongoing concerns about oncological radicality (i.e. ability to completely remove cancer) and an inability to control massive haemorrhage. These issues can partially be attributed to a loss of sensation such as depth perception, tactile feedback and a reduced field of view. Utilisation of optical imaging and computer vision may be able to compensate for some of the lost sensory input because these modalities can facilitate visualisation of liver tissue and structural anatomy. Their use in laparoscopy is attractive because it is easy to adapt or integrate with existing technology. The aim of this thesis is to explore to what extent this technology can aid in the detection of normal and abnormal liver tissue and structures. / Methods: The current state of the art for optical imaging and computer vision in laparoscopic liver surgery is assessed in a systematic review. Evaluation of confocal laser endomicroscopy is carried out on a murine and porcine model of liver disease. Multispectral near infrared imaging is evaluated on ex-vivo liver specimen. Video magnification is assessed on a mechanical flow phantom and a porcine model of liver disease. The latter model was also employed to develop a computer vision based image guidance system for laparoscopic liver surgery. This image guidance system is further evaluated in a clinical feasibility study. Where appropriate, experimental findings are substantiated with statistical analysis. / Results: Use of confocal laser endomicroscopy enabled discrimination between cancer and normal liver tissue with a sub-millimetre precision. This technology also made it possible to verify the adequacy of thermal liver ablation. Multispectral imaging, at specific wavelengths was shown to have the potential to highlight the presence of colorectal and hepatocellular cancer. An image reprocessing algorithm is proposed to simplify visual interpretation of the resulting images. It is shown that video magnification can determine the presence of pulsatile motion but that it cannot reliably determine the extent of motion. Development and performance metrics of an image guidance system for laparoscopic liver surgery are outlined. The system was found to improve intraoperative orientation more development work is however required to enable reliable prediction of oncological margins. / Discussion: The results in this thesis indicate that confocal laser endomicroscopy and image guidance systems have reached a development stage where their intraoperative use may benefit surgeons by visualising features of liver anatomy and tissue characteristics. Video magnification and multispectral imaging require more development and suggestions are made to direct this work. It is also highlighted that it is crucial to standardise assessment methods for these technologies which will allow a more direct comparison between the outcomes of different groups. Limited imaging depth is a major restriction of these technologies but this may be overcome by combining them with preoperatively obtained imaging data. Just like laparoscopy, optical imaging and computer vision use functions of light, a shared characteristic that makes their combined use complementary