Tissue classification for laparoscopic image understanding based on multispectral texture analysis.

Intraoperative tissue classification is one of the prerequisites for providing context-aware visualization in computer-assisted minimally invasive surgeries. As many anatomical structures are difficult to differentiate in conventional RGB medical images, we propose a classification method based on multispectral image patches. In a comprehensive ex vivo study through statistical analysis, we show that (1) multispectral imaging data are superior to RGB data for organ tissue classification when used in conjunction with widely applied feature descriptors and (2) combining the tissue texture with the reflectance spectrum improves the classification performance. The classifier reaches an accuracy of 98.4% on our dataset. Multispectral tissue analysis could thus evolve as a key enabling technique in computer-assisted laparoscopy

Intraoperative colon perfusion assessment using multispectral imaging

In colorectal surgery an anastomosis performed using poorly-perfused, ischaemic bowel segments may result in a leak and consequent morbidity. Traditional measures of perfusion assessment rely on clinical judgement and are mainly subjective, based on tissue appearance, leading to variability between clinicians. This paper describes a multispectral imaging (MSI) laparoscope that can derive quantitative measures of tissue oxygen saturation (SO2). The system uses a xenon surgical light source and fast filter wheel camera to capture eight narrow waveband images across the visible range in approximately 0.3 s. Spectral validation measurements were performed by imaging standardised colour tiles and comparing reflectance with ground truth spectrometer data. Tissue spectra were decomposed into individual contributions from haemoglobin, adipose tissue and scattering, using a previously-developed regression approach. Initial clinical results from seven patients undergoing colorectal surgery are presented and used to characterise measurement stability and reproducibility in vivo. Strategies to improve signal-To-noise ratio and correct for motion are described. Images of healthy bowel tissue (in vivo) indicate that baseline SO2 is approximately 75 } 6%. The SO2 profile along a bowel segment following ligation of the inferior mesenteric artery (IMA) shows a decrease from the proximal to distal end. In the clinical cases shown, imaging results concurred with clinical judgements of the location of well-perfused tissue. Adipose tissue, visibly yellow in the RGB images, is shown to surround the mesentery and cover some of the serosa. SO2 in this tissue is consistently high, with mean value of 90%. These results show that MSI is a potential intraoperative guidance tool for assessment of perfusion. Mapping of SO2 in the colon could be used by surgeons to guide choice of transection points and ensure that well-perfused tissue is used to form an anastomosis. The observation of high mesenteric SO2 agrees with work in the literature and warrants further exploration. Larger studies incorporating with a wider cohort of clinicians will help to provide retrospective evidence of how this imaging technique may be able to reduce inter-operator variability

Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy

Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical structure classification and image tagging that features an intrinsic measure of confidence to estimate its own performance with high reliability and which can be applied to both RGB and multispectral imaging (MI) data. Methods: Organ recognition is performed using a superpixel classification strategy based on textural and reflectance information. Classification confidence is estimated by analyzing the dispersion of class probabilities. Assessment of the proposed technology is performed through a comprehensive in vivo study with seven pigs. Results: When applied to image tagging, mean accuracy in our experiments increased from 65% (RGB) and 80% (MI) to 90% (RGB) and 96% (MI) with the confidence measure. Conclusion: Results showed that the confidence measure had a significant influence on the classification accuracy, and MI data are better suited for anatomical structure labeling than RGB data. Significance: This work significantly enhances the state of art in automatic labeling of endoscopic videos by introducing the use of the confidence metric, and by being the first study to use MI data for in vivo laparoscopic tissue classification. The data of our experiments will be released as the first in vivo MI dataset upon publication of this paper.Comment: 7 pages, 6 images, 2 table

Optimized spectral filter design enables more accurate estimation of oxygen saturation in spectral imaging

Oxygen saturation (SO2) in tissue is a crucially important physiological parameter with ubiquitous clinical utility in diagnosis, treatment, and monitoring, as well as widespread use as an invaluable preclinical research tool. Multispectral imaging can be used to visualize SO2 non-invasively, non-destructively and without contact in real-time using narrow spectral filter sets, but typically, these spectral filter sets are poorly suited to a specific clinical task, application, or tissue type. In this work, we demonstrate the merit of optimizing spectral filter sets for more accurate estimation of SO2. Using tissue modelling and simulated multispectral imaging, we demonstrate filter optimization reduces the root-mean-square-error (RMSE) in estimating SO2 by up to 37% compared with evenly spaced filters. Moreover, we demonstrate up to a 79% decrease in RMSE for optimized filter sets compared with filter sets chosen to minimize mutual information. Wider adoption of this approach will result in more effective multispectral imaging systems that can address specific clinical needs and consequently, more widespread adoption of multispectral imaging technologies in disease diagnosis and treatment

Bayesian Estimation of Intrinsic Tissue Oxygenation and Perfusion from RGB Images

Multispectral imaging (MSI) can potentially assist the intra-operative assessment of tissue structure, function and viability, by providing information about oxygenation. In this paper, we present a novel technique for recovering intrinsic MSI measurements from endoscopic RGB images without custom hardware adaptations. The advantage of this approach is that it requires no modification to existing surgical and diagnostic endoscopic imaging systems. Our method uses a radiometric colour calibration of the endoscopic camera's sensor in conjunction with a Bayesian framework to recover a per-pixel measurement of the total blood volume (THb) and oxygen saturation (SO2) in the observed tissue. The sensor's pixel measurements are modelled as weighted sums over a mixture of Poisson distributions and we optimise the variables SO2 and THb to maximise the likelihood of the observations. To validate our technique, we use synthetic images generated from Monte Carlo (MC) physics simulation of light transport through soft tissue containing sub-surface blood vessels. We also validate our method on in vivo data by comparing it to a MSI dataset acquired with a hardware system that sequentially images multiple spectral bands without overlap. Our results are promising and show that we are able to provide surgeons with additional relevant information by processing endoscopic images with our modelling and inference framework

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013–2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation

Multispectral image analysis in laparoscopy – A machine learning approach to live perfusion monitoring

Modern visceral surgery is often performed through small incisions. Compared to open surgery, these minimally invasive interventions result in smaller scars, fewer complications and a quicker recovery. While to the patients benefit, it has the drawback of limiting the physician’s perception largely to that of visual feedback through a camera mounted on a rod lens: the laparoscope. Conventional laparoscopes are limited by “imitating” the human eye. Multispectral cameras remove this arbitrary restriction of recording only red, green and blue colors. Instead, they capture many specific bands of light. Although these could help characterize important indications such as ischemia and early stage adenoma, the lack of powerful digital image processing prevents realizing the technique’s full potential. The primary objective of this thesis was to pioneer fluent functional multispectral imaging (MSI) in laparoscopy. The main technical obstacles were: (1) The lack of image analysis concepts that provide both high accuracy and speed. (2) Multispectral image recording is slow, typically ranging from seconds to minutes. (3) Obtaining a quantitative ground truth for the measurements is hard or even impossible. To overcome these hurdles and enable functional laparoscopy, for the first time in this field physical models are combined with powerful machine learning techniques. The physical model is employed to create highly accurate simulations, which in turn teach the algorithm to rapidly relate multispectral pixels to underlying functional changes. To reduce the domain shift introduced by learning from simulations, a novel transfer learning approach automatically adapts generic simulations to match almost arbitrary recordings of visceral tissue. In combination with the only available video-rate capable multispectral sensor, the method pioneers fluent perfusion monitoring with MSI. This system was carefully tested in a multistage process, involving in silico quantitative evaluations, tissue phantoms and a porcine study. Clinical applicability was ensured through in-patient recordings in the context of partial nephrectomy; in these, the novel system characterized ischemia live during the intervention. Verified against a fluorescence reference, the results indicate that fluent, non-invasive ischemia detection and monitoring is now possible. In conclusion, this thesis presents the first multispectral laparoscope capable of videorate functional analysis. The system was successfully evaluated in in-patient trials, and future work should be directed towards evaluation of the system in a larger study. Due to the broad applicability and the large potential clinical benefit of the presented functional estimation approach, I am confident the descendants of this system are an integral part of the next generation OR