Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions

Probe-based Rapid Hybrid Hyperspectral and Tissue Surface Imaging Aided by Fully Convolutional Networks

Tissue surface shape and reflectance spectra provide rich intra-operative information useful in surgical guidance. We propose a hybrid system which displays an endoscopic image with a fast joint inspection of tissue surface shape using structured light (SL) and hyperspectral imaging (HSI). For SL a miniature fibre probe is used to project a coloured spot pattern onto the tissue surface. In HSI mode standard endoscopic illumination is used, with the fibre probe collecting reflected light and encoding the spatial information into a linear format that can be imaged onto the slit of a spectrograph. Correspondence between the arrangement of fibres at the distal and proximal ends of the bundle was found using spectral encoding. Then during pattern decoding, a fully convolutional network (FCN) was used for spot detection, followed by a matching propagation algorithm for spot identification. This method enabled fast reconstruction (12 frames per second) using a GPU. The hyperspectral image was combined with the white light image and the reconstructed surface, showing the spectral information of different areas. Validation of this system using phantom and ex vivo experiments has been demonstrated.Comment: This paper has been submitted to MICCAI2016 on 17 March, 2016, and conditionally accepted on 2 June, 201

Surgical polarimetric endoscopy for the detection of laryngeal cancer

The standard-of-care for the detection of laryngeal pathologies involves distinguishing suspicious lesions from surrounding healthy tissue via contrasts in colour and texture captured by white-light endoscopy. However, the technique is insufficiently sensitive and thus leads to unsatisfactory rates of false negatives. Here we show that laryngeal lesions can be better detected in real time by taking advantage of differences in the light-polarization properties of cancer and healthy tissues. By measuring differences in polarized-light retardance and depolarization, the technique, which we named 'surgical polarimetric endoscopy' (SPE), generates about one-order-of-magnitude greater contrast than white-light endoscopy, and hence allows for the better discrimination of cancerous lesions, as we show with patients diagnosed with squamous cell carcinoma. Polarimetric imaging of excised and stained slices of laryngeal tissue indicated that changes in the retardance of polarized light can be largely attributed to architectural features of the tissue. We also assessed SPE to aid routine transoral laser surgery for the removal of a cancerous lesion, indicating that SPE can complement white-light endoscopy for the detection of laryngeal cancer

Intraoperative Endoscopic Augmented Reality in Third Ventriculostomy

In neurosurgery, as a result of the brain-shift, the preoperative patient models used as a intraoperative reference change. A meaningful use of the preoperative virtual models during the operation requires for a model update. The NEAR project, Neuroendoscopy towards Augmented Reality, describes a new camera calibration model for high distorted lenses and introduces the concept of active endoscopes endowed with with navigation, camera calibration, augmented reality and triangulation modules

EndoAbS dataset: Endoscopic abdominal stereo image dataset for benchmarking 3D stereo reconstruction algorithms

none5siembargoed_20190801Penza, Veronica; Ciullo, Andrea S.; Moccia, Sara; Mattos, Leonardo S.; De Momi, ElenaPenza, Veronica; Ciullo, Andrea S.; Moccia, Sara; Mattos, Leonardo S.; De Momi, Elen

EndoAbS dataset: Endoscopic abdominal stereo image dataset for benchmarking 3D stereo reconstruction algorithms

Background: 3D reconstruction algorithms are of fundamental importance for augmented reality applications in computer-assisted surgery. However, few datasets of endoscopic stereo images with associated 3D surface references are currently openly available, preventing the proper validation of such algorithms. This work presents a new and rich dataset of endoscopic stereo images (EndoAbS dataset). Methods: The dataset includes (i) endoscopic stereo images of phantom abdominal organs, (ii) a 3D organ surface reference (RF) generated with a laser scanner and (iii) camera calibration parameters. A detailed description of the generation of the phantom and the camera–laser calibration method is also provided. Results: An estimation of the overall error in creation of the dataset is reported (camera–laser calibration error 0.43 mm) and the performance of a 3D reconstruction algorithm is evaluated using EndoAbS, resulting in an accuracy error in accordance with state-of-the-art results (<2 mm). Conclusions: The EndoAbS dataset contributes to an increase the number and variety of openly available datasets of surgical stereo images, including a highly accurate RF and different surgical conditions

Reconstructing 3d lung shape from a single 2d image during the deaeration deformation process using model-based data augmentation

Three-dimensional (3D) shape reconstruction is particularly important for computer assisted medical systems, especially in the case of lung surgeries, where large deaeration deformation occurs. Recently, 3D reconstruction methods based on machine learning techniques have achieved considerable success in computer vision. However, it is difficult to apply these approaches to the medical field, because the collection of a massive amount of clinic data for training is impractical. To solve this problem, this paper proposes a novel 3D shape reconstruction method that adopts both data augmentation techniques and convolutional neural networks. In the proposed method, a deformable statistical model of the 3D lungs is designed to augment various training data. As the experimental results demonstrate, even with a small database, the proposed method can realize 3D shape reconstruction for lungs during a deaeration deformation process from only one captured 2D image. Moreover, the proposed data augmentation technique can also be used in other fields where the training data are insufficient

Appearance Modelling and Reconstruction for Navigation in Minimally Invasive Surgery

Minimally invasive surgery is playing an increasingly important role for patient care. Whilst its direct patient benefit in terms of reduced trauma, improved recovery and shortened hospitalisation has been well established, there is a sustained need for improved training of the existing procedures and the development of new smart instruments to tackle the issue of visualisation, ergonomic control, haptic and tactile feedback. For endoscopic intervention, the small field of view in the presence of a complex anatomy can easily introduce disorientation to the operator as the tortuous access pathway is not always easy to predict and control with standard endoscopes. Effective training through simulation devices, based on either virtual reality or mixed-reality simulators, can help to improve the spatial awareness, consistency and safety of these procedures. This thesis examines the use of endoscopic videos for both simulation and navigation purposes. More specifically, it addresses the challenging problem of how to build high-fidelity subject-specific simulation environments for improved training and skills assessment. Issues related to mesh parameterisation and texture blending are investigated. With the maturity of computer vision in terms of both 3D shape reconstruction and localisation and mapping, vision-based techniques have enjoyed significant interest in recent years for surgical navigation. The thesis also tackles the problem of how to use vision-based techniques for providing a detailed 3D map and dynamically expanded field of view to improve spatial awareness and avoid operator disorientation. The key advantage of this approach is that it does not require additional hardware, and thus introduces minimal interference to the existing surgical workflow. The derived 3D map can be effectively integrated with pre-operative data, allowing both global and local 3D navigation by taking into account tissue structural and appearance changes. Both simulation and laboratory-based experiments are conducted throughout this research to assess the practical value of the method proposed

Computational Multispectral Endoscopy

Minimal Access Surgery (MAS) is increasingly regarded as the de-facto approach in interventional medicine for conducting many procedures this is due to the reduced patient trauma and consequently reduced recovery times, complications and costs. However, there are many challenges in MAS that come as a result of viewing the surgical site through an endoscope and interacting with tissue remotely via tools, such as lack of haptic feedback; limited field of view; and variation in imaging hardware. As such, it is important best utilise the imaging data available to provide a clinician with rich data corresponding to the surgical site. Measuring tissue haemoglobin concentrations can give vital information, such as perfusion assessment after transplantation; visualisation of the health of blood supply to organ; and to detect ischaemia. In the area of transplant and bypass procedures measurements of the tissue tissue perfusion/total haemoglobin (THb) and oxygen saturation (SO2) are used as indicators of organ viability, these measurements are often acquired at multiple discrete points across the tissue using with a specialist probe. To acquire measurements across the whole surface of an organ one can use a specialist camera to perform multispectral imaging (MSI), which optically acquires sequential spectrally band limited images of the same scene. This data can be processed to provide maps of the THb and SO2 variation across the tissue surface which could be useful for intra operative evaluation. When capturing MSI data, a trade off often has to be made between spectral sensitivity and capture speed. The work in thesis first explores post processing blurry MSI data from long exposure imaging devices. It is of interest to be able to use these MSI data because the large number of spectral bands that can be captured, the long capture times, however, limit the potential real time uses for clinicians. Recognising the importance to clinicians of real-time data, the main body of this thesis develops methods around estimating oxy- and deoxy-haemoglobin concentrations in tissue using only monocular and stereo RGB imaging data