55 research outputs found
Prospects for Theranostics in Neurosurgical Imaging: Empowering Confocal Laser Endomicroscopy Diagnostics via Deep Learning
Confocal laser endomicroscopy (CLE) is an advanced optical fluorescence
imaging technology that has the potential to increase intraoperative precision,
extend resection, and tailor surgery for malignant invasive brain tumors
because of its subcellular dimension resolution. Despite its promising
diagnostic potential, interpreting the gray tone fluorescence images can be
difficult for untrained users. In this review, we provide a detailed
description of bioinformatical analysis methodology of CLE images that begins
to assist the neurosurgeon and pathologist to rapidly connect on-the-fly
intraoperative imaging, pathology, and surgical observation into a
conclusionary system within the concept of theranostics. We present an overview
and discuss deep learning models for automatic detection of the diagnostic CLE
images and discuss various training regimes and ensemble modeling effect on the
power of deep learning predictive models. Two major approaches reviewed in this
paper include the models that can automatically classify CLE images into
diagnostic/nondiagnostic, glioma/nonglioma, tumor/injury/normal categories and
models that can localize histological features on the CLE images using weakly
supervised methods. We also briefly review advances in the deep learning
approaches used for CLE image analysis in other organs. Significant advances in
speed and precision of automated diagnostic frame selection would augment the
diagnostic potential of CLE, improve operative workflow and integration into
brain tumor surgery. Such technology and bioinformatics analytics lend
themselves to improved precision, personalization, and theranostics in brain
tumor treatment.Comment: See the final version published in Frontiers in Oncology here:
https://www.frontiersin.org/articles/10.3389/fonc.2018.00240/ful
Confocal Laser Endomicroscopy Image Analysis with Deep Convolutional Neural Networks
abstract: Rapid intraoperative diagnosis of brain tumors is of great importance for planning treatment and guiding the surgeon about the extent of resection. Currently, the standard for the preliminary intraoperative tissue analysis is frozen section biopsy that has major limitations such as tissue freezing and cutting artifacts, sampling errors, lack of immediate interaction between the pathologist and the surgeon, and time consuming.
Handheld, portable confocal laser endomicroscopy (CLE) is being explored in neurosurgery for its ability to image histopathological features of tissue at cellular resolution in real time during brain tumor surgery. Over the course of examination of the surgical tumor resection, hundreds to thousands of images may be collected. The high number of images requires significant time and storage load for subsequent reviewing, which motivated several research groups to employ deep convolutional neural networks (DCNNs) to improve its utility during surgery. DCNNs have proven to be useful in natural and medical image analysis tasks such as classification, object detection, and image segmentation.
This thesis proposes using DCNNs for analyzing CLE images of brain tumors. Particularly, it explores the practicality of DCNNs in three main tasks. First, off-the shelf DCNNs were used to classify images into diagnostic and non-diagnostic. Further experiments showed that both ensemble modeling and transfer learning improved the classifier’s accuracy in evaluating the diagnostic quality of new images at test stage. Second, a weakly-supervised learning pipeline was developed for localizing key features of diagnostic CLE images from gliomas. Third, image style transfer was used to improve the diagnostic quality of CLE images from glioma tumors by transforming the histology patterns in CLE images of fluorescein sodium-stained tissue into the ones in conventional hematoxylin and eosin-stained tissue slides.
These studies suggest that DCNNs are opted for analysis of CLE images. They may assist surgeons in sorting out the non-diagnostic images, highlighting the key regions and enhancing their appearance through pattern transformation in real time. With recent advances in deep learning such as generative adversarial networks and semi-supervised learning, new research directions need to be followed to discover more promises of DCNNs in CLE image analysis.Dissertation/ThesisDoctoral Dissertation Neuroscience 201
Transferability of Deep Learning Algorithms for Malignancy Detection in Confocal Laser Endomicroscopy Images from Different Anatomical Locations of the Upper Gastrointestinal Tract
Squamous Cell Carcinoma (SCC) is the most common cancer type of the
epithelium and is often detected at a late stage. Besides invasive diagnosis of
SCC by means of biopsy and histo-pathologic assessment, Confocal Laser
Endomicroscopy (CLE) has emerged as noninvasive method that was successfully
used to diagnose SCC in vivo. For interpretation of CLE images, however,
extensive training is required, which limits its applicability and use in
clinical practice of the method. To aid diagnosis of SCC in a broader scope,
automatic detection methods have been proposed. This work compares two methods
with regard to their applicability in a transfer learning sense, i.e. training
on one tissue type (from one clinical team) and applying the learnt
classification system to another entity (different anatomy, different clinical
team). Besides a previously proposed, patch-based method based on convolutional
neural networks, a novel classification method on image level (based on a
pre-trained Inception V.3 network with dedicated preprocessing and
interpretation of class activation maps) is proposed and evaluated. The newly
presented approach improves recognition performance, yielding accuracies of
91.63% on the first data set (oral cavity) and 92.63% on a joint data set. The
generalization from oral cavity to the second data set (vocal folds) lead to
similar area-under-the-ROC curve values than a direct training on the vocal
folds data set, indicating good generalization.Comment: Erratum for version 1, correcting the number of CLE image sequences
used in one data se
Bayesian image restoration and bacteria detection in optical endomicroscopy
Optical microscopy systems can be used to obtain high-resolution microscopic images of tissue cultures and ex vivo tissue samples. This imaging technique can be translated for in vivo, in situ applications by using optical fibres and miniature optics. Fibred optical endomicroscopy (OEM) can enable optical biopsy in organs inaccessible by any other imaging systems, and hence can provide rapid and accurate diagnosis in a short time. The raw data the system produce is difficult to interpret as it is modulated by a fibre bundle pattern, producing what is called the “honeycomb effect”. Moreover, the data is further degraded due to the fibre core cross coupling problem. On the other hand, there is an unmet clinical need for automatic tools that can help the clinicians to detect fluorescently labelled bacteria in distal lung images. The aim of this thesis is to develop advanced image processing algorithms that can address the above mentioned problems. First, we provide a statistical model for the fibre core cross coupling problem and the sparse sampling by imaging fibre bundles (honeycomb artefact), which are formulated here as a restoration problem for the first time in the literature. We then introduce a non-linear interpolation method, based on Gaussian processes regression, in order to recover an interpretable scene from the deconvolved data. Second, we develop two bacteria detection algorithms, each of which provides different characteristics. The first approach considers joint formulation to the sparse coding and anomaly detection problems. The anomalies here are considered as candidate bacteria, which are annotated with the help of a trained clinician. Although this approach provides good detection performance and outperforms existing methods in the literature, the user has to carefully tune some crucial model parameters. Hence, we propose a more adaptive approach, for which a Bayesian framework is adopted. This approach not only outperforms the proposed supervised approach and existing methods in the literature but also provides computation time that competes with optimization-based methods
Online Super-Resolution For Fibre-Bundle-Based Confocal Laser Endomicroscopy
Probe-based Confocal Laser Endomicroscopy (pCLE) produces microscopic images enabling real-time in vivo optical biopsy. However, the miniaturisation of the optical hardware, specifically the reliance on an optical fibre bundle as an imaging guide, fundamentally limits image quality by producing artefacts, noise, and relatively low contrast and resolution. The reconstruction approaches in clinical pCLE products do not fully alleviate these problems. Consequently, image quality remains a barrier that curbs the full potential of pCLE. Enhancing the image quality of pCLE in real-time remains a challenge. The research in this thesis is a response to this need. I have developed dedicated online super-resolution methods that account for the physics of the image acquisition process. These methods have the potential to replace existing reconstruction algorithms without interfering with the fibre design or the hardware of the device. In this thesis, novel processing pipelines are proposed for enhancing the image quality of pCLE. First, I explored a learning-based super-resolution method that relies on mapping from the low to the high-resolution space. Due to the lack of high-resolution pCLE, I proposed to simulate high-resolution data and use it as a ground truth model that is based on the pCLE acquisition physics. However, pCLE images are reconstructed from irregularly distributed fibre signals, and grid-based Convolutional Neural Networks are not designed to take irregular data as input. To alleviate this problem, I designed a new trainable layer that embeds Nadaraya- Watson regression. Finally, I proposed a novel blind super-resolution approach by deploying unsupervised zero-shot learning accompanied by a down-sampling kernel crafted for pCLE. I evaluated these new methods in two ways: a robust image quality assessment and a perceptual quality test assessed by clinical experts. The results demonstrate that the proposed super-resolution pipelines are superior to the current reconstruction algorithm in terms of image quality and clinician preference
Deconvolution and Restoration of Optical Endomicroscopy Images
Optical endomicroscopy (OEM) is an emerging technology platform with
preclinical and clinical imaging applications. Pulmonary OEM via fibre bundles
has the potential to provide in vivo, in situ molecular signatures of disease
such as infection and inflammation. However, enhancing the quality of data
acquired by this technique for better visualization and subsequent analysis
remains a challenging problem. Cross coupling between fiber cores and sparse
sampling by imaging fiber bundles are the main reasons for image degradation,
and poor detection performance (i.e., inflammation, bacteria, etc.). In this
work, we address the problem of deconvolution and restoration of OEM data. We
propose a hierarchical Bayesian model to solve this problem and compare three
estimation algorithms to exploit the resulting joint posterior distribution.
The first method is based on Markov chain Monte Carlo (MCMC) methods, however,
it exhibits a relatively long computational time. The second and third
algorithms deal with this issue and are based on a variational Bayes (VB)
approach and an alternating direction method of multipliers (ADMM) algorithm
respectively. Results on both synthetic and real datasets illustrate the
effectiveness of the proposed methods for restoration of OEM images
The intersection of video capsule endoscopy and artificial intelligence: addressing unique challenges using machine learning
Introduction: Technical burdens and time-intensive review processes limit the
practical utility of video capsule endoscopy (VCE). Artificial intelligence
(AI) is poised to address these limitations, but the intersection of AI and VCE
reveals challenges that must first be overcome. We identified five challenges
to address. Challenge #1: VCE data are stochastic and contains significant
artifact. Challenge #2: VCE interpretation is cost-intensive. Challenge #3: VCE
data are inherently imbalanced. Challenge #4: Existing VCE AIMLT are
computationally cumbersome. Challenge #5: Clinicians are hesitant to accept
AIMLT that cannot explain their process.
Methods: An anatomic landmark detection model was used to test the
application of convolutional neural networks (CNNs) to the task of classifying
VCE data. We also created a tool that assists in expert annotation of VCE data.
We then created more elaborate models using different approaches including a
multi-frame approach, a CNN based on graph representation, and a few-shot
approach based on meta-learning.
Results: When used on full-length VCE footage, CNNs accurately identified
anatomic landmarks (99.1%), with gradient weighted-class activation mapping
showing the parts of each frame that the CNN used to make its decision. The
graph CNN with weakly supervised learning (accuracy 89.9%, sensitivity of
91.1%), the few-shot model (accuracy 90.8%, precision 91.4%, sensitivity
90.9%), and the multi-frame model (accuracy 97.5%, precision 91.5%, sensitivity
94.8%) performed well. Discussion: Each of these five challenges is addressed,
in part, by one of our AI-based models. Our goal of producing high performance
using lightweight models that aim to improve clinician confidence was achieved
- …