Self-supervised monocular depth estimation with 3-D displacement module for laparoscopic images

We present a novel self-supervised training framework with 3D displacement (3DD) module for accurately estimating per-pixel depth maps from single laparoscopic images. Recently, several self-supervised learning based monocular depth estimation models have achieved good results on the KITTI dataset, under the hypothesis that the camera is dynamic and the objects are stationary, however this hypothesis is often reversed in the surgical setting (laparoscope is stationary, the surgical instruments and tissues are dynamic). Therefore, a 3DD module is proposed to establish the relation between frames instead of ego-motion estimation. In the 3DD module, a convolutional neural network (CNN) analyses source and target frames to predict the 3D displacement of a 3D point cloud from a target frame to a source frame in the coordinates of the camera. Since it is difficult to constrain the depth displacement from two 2D images, a novel depth consistency module is proposed to maintain depth consistency between displacement-updated depth and model-estimated depth to constrain 3D displacement effectively. Our proposed method achieves remarkable performance for monocular depth estimation on the Hamlyn surgical dataset and acquired ground truth depth maps, outperforming monodepth, monodepth2 and packnet models

Flexible multimode endoscope for tissue reflectance and autofluorescence hyperspectral imaging

A dual reflectance and autofluorescence spectral imaging probe compatible with the biopsy channels of standard flexible endoscopes is demonstrated. Spatially-resolved haemoglobin and autofluorescent signals from porcine bowel were obtained in vivo

Cable-driven parallel robot assisted confocal imaging of the larynx

LaryngoTORS, a transoral laryngeal surgery robot, can manipulate instruments accurately. Confocal imaging has potentials in laryngeal cancer diagnosis but suffer from high scanning requirement. This work studies using LaryngoTORS to assist confocal imaging of larynx

H-Net: unsupervised attention-based stereo depth estimation leveraging epipolar geometry

Depth estimation from a stereo image pair has become one of the most explored applications in computer vision, with most previous methods relying on fully supervised learning settings. However, due to the difficulty in acquiring accurate and scalable ground truth data, the training of fully supervised methods is challenging. As an alternative, self-supervised methods are becoming more popular to mitigate this challenge. In this paper, we introduce the H-Net, a deep-learning framework for unsupervised stereo depth estimation that leverages epipolar geometry to refine stereo matching. For the first time, a Siamese autoencoder architecture is used for depth estimation which allows mutual information between rectified stereo images to be extracted. To enforce the epipolar constraint, the mutual epipolar attention mechanism has been designed which gives more emphasis to correspondences of features that lie on the same epipolar line while learning mutual information between the input stereo pair. Stereo correspondences are further enhanced by incorporating semantic information to the proposed attention mechanism. More specifically, the optimal transport algorithm is used to suppress attention and eliminate outliers in areas not visible in both cameras. Extensive experiments on KITTI2015 and Cityscapes show that the proposed modules are able to improve the performance of the unsupervised stereo depth estimation methods while closing the gap with the fully supervised approaches

A high definition Mueller polarimetric endoscope for tissue characterisation

The contrast mechanism of medical endoscopy is mainly based on metrics of optical intensity and wavelength. As another fundamental property of light, polarization can not only reveal tissue scattering and absorption information from a different perspective, but can also provide insight into directional tissue birefringence properties to monitor pathological changes in collagen and elastin. Here we demonstrate a low cost wide field high definition Mueller polarimetric endoscope with minimal alterations to a rigid endoscope. We show that this novel endoscopic imaging modality is able to provide a number of image contrast mechanisms besides traditional unpolarized radiation intensity, including linear depolarization, circular depolarization, cross-polarization, directional birefringence and dichroism. This enhances tissue features of interest, and additionally reveals tissue micro-structure and composition, which is of central importance for tissue diagnosis and image guidance for surgery. The potential applications of the Mueller polarimetric endoscope include wide field early epithelial cancer diagnosis, surgical margin detection and energy-based tissue fusion monitoring, and could further benefit a wide range of endoscopic investigations through intra-operative guidance