35 research outputs found
Trustworthy Deep Learning for Medical Image Segmentation
Despite the recent success of deep learning methods at achieving new
state-of-the-art accuracy for medical image segmentation, some major
limitations are still restricting their deployment into clinics. One major
limitation of deep learning-based segmentation methods is their lack of
robustness to variability in the image acquisition protocol and in the imaged
anatomy that were not represented or were underrepresented in the training
dataset. This suggests adding new manually segmented images to the training
dataset to better cover the image variability. However, in most cases, the
manual segmentation of medical images requires highly skilled raters and is
time-consuming, making this solution prohibitively expensive. Even when
manually segmented images from different sources are available, they are rarely
annotated for exactly the same regions of interest. This poses an additional
challenge for current state-of-the-art deep learning segmentation methods that
rely on supervised learning and therefore require all the regions of interest
to be segmented for all the images to be used for training. This thesis
introduces new mathematical and optimization methods to mitigate those
limitations.Comment: PhD thesis successfully defended on 1st July 2022. Examiners: Prof
Sotirios Tsaftaris and Dr Wenjia Ba
Generalized Wasserstein Dice Score, Distributionally Robust Deep Learning, and Ranger for brain tumor segmentation: BraTS 2020 challenge
Training a deep neural network is an optimization problem with four main
ingredients: the design of the deep neural network, the per-sample loss
function, the population loss function, and the optimizer. However, methods
developed to compete in recent BraTS challenges tend to focus only on the
design of deep neural network architectures, while paying less attention to the
three other aspects. In this paper, we experimented with adopting the opposite
approach. We stuck to a generic and state-of-the-art 3D U-Net architecture and
experimented with a non-standard per-sample loss function, the generalized
Wasserstein Dice loss, a non-standard population loss function, corresponding
to distributionally robust optimization, and a non-standard optimizer, Ranger.
Those variations were selected specifically for the problem of multi-class
brain tumor segmentation. The generalized Wasserstein Dice loss is a per-sample
loss function that allows taking advantage of the hierarchical structure of the
tumor regions labeled in BraTS. Distributionally robust optimization is a
generalization of empirical risk minimization that accounts for the presence of
underrepresented subdomains in the training dataset. Ranger is a generalization
of the widely used Adam optimizer that is more stable with small batch size and
noisy labels. We found that each of those variations of the optimization of
deep neural networks for brain tumor segmentation leads to improvements in
terms of Dice scores and Hausdorff distances. With an ensemble of three deep
neural networks trained with various optimization procedures, we achieved
promising results on the validation dataset of the BraTS 2020 challenge. Our
ensemble ranked fourth out of the 693 registered teams for the segmentation
task of the BraTS 2020 challenge.Comment: MICCAI 2020 BrainLes Workshop. Our method ranked fourth out of the
693 registered teams for the segmentation task of the BraTS 2020 challenge.
v2: Added some clarifications following reviewers' feedback (camera-ready
version
ECONet: Efficient Convolutional Online Likelihood Network for Scribble-based Interactive Segmentation
Automatic segmentation of lung lesions associated with COVID-19 in CT images
requires large amount of annotated volumes. Annotations mandate expert
knowledge and are time-intensive to obtain through fully manual segmentation
methods. Additionally, lung lesions have large inter-patient variations, with
some pathologies having similar visual appearance as healthy lung tissues. This
poses a challenge when applying existing semi-automatic interactive
segmentation techniques for data labelling. To address these challenges, we
propose an efficient convolutional neural networks (CNNs) that can be learned
online while the annotator provides scribble-based interaction. To accelerate
learning from only the samples labelled through user-interactions, a
patch-based approach is used for training the network. Moreover, we use
weighted cross-entropy loss to address the class imbalance that may result from
user-interactions. During online inference, the learned network is applied to
the whole input volume using a fully convolutional approach. We compare our
proposed method with state-of-the-art using synthetic scribbles and show that
it outperforms existing methods on the task of annotating lung lesions
associated with COVID-19, achieving 16% higher Dice score while reducing
execution time by 3 and requiring 9000 lesser scribbles-based labelled
voxels. Due to the online learning aspect, our approach adapts quickly to user
input, resulting in high quality segmentation labels. Source code for ECONet is
available at: https://github.com/masadcv/ECONet-MONAILabel.Comment: Accepted at MIDL 202
A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentation
Deep learning models for medical image segmentation can fail unexpectedly and spectacularly for pathological cases and images acquired at different centers than training images, with labeling errors that violate expert knowledge. Such errors undermine the trustworthiness of deep learning models for medical image segmentation. Mechanisms for detecting and correcting such failures are essential for safely translating this technology into clinics and are likely to be a requirement of future regulations on artificial intelligence (AI). In this work, we propose a trustworthy AI theoretical framework and a practical system that can augment any backbone AI system using a fallback method and a fail-safe mechanism based on Dempster-Shafer theory. Our approach relies on an actionable definition of trustworthy AI. Our method automatically discards the voxel-level labeling predicted by the backbone AI that violate expert knowledge and relies on a fallback for those voxels. We demonstrate the effectiveness of the proposed trustworthy AI approach on the largest reported annotated dataset of fetal MRI consisting of 540 manually annotated fetal brain 3D T2w MRIs from 13 centers. Our trustworthy AI method improves the robustness of a state-of-the-art backbone AI for fetal brain MRIs acquired across various centers and for fetuses with various brain abnormalities
Scalable multimodal convolutional networks for brain tumour segmentation
Brain tumour segmentation plays a key role in computer-assisted surgery. Deep
neural networks have increased the accuracy of automatic segmentation
significantly, however these models tend to generalise poorly to different
imaging modalities than those for which they have been designed, thereby
limiting their applications. For example, a network architecture initially
designed for brain parcellation of monomodal T1 MRI can not be easily
translated into an efficient tumour segmentation network that jointly utilises
T1, T1c, Flair and T2 MRI. To tackle this, we propose a novel scalable
multimodal deep learning architecture using new nested structures that
explicitly leverage deep features within or across modalities. This aims at
making the early layers of the architecture structured and sparse so that the
final architecture becomes scalable to the number of modalities. We evaluate
the scalable architecture for brain tumour segmentation and give evidence of
its regularisation effect compared to the conventional concatenation approach.Comment: Paper accepted at MICCAI 201
Image Compositing for Segmentation of Surgical Tools Without Manual Annotations
Producing manual, pixel-accurate, image segmentation labels is tedious and time-consuming. This is often a rate-limiting factor when large amounts of labeled images are required, such as for training deep convolutional networks for instrument-background segmentation in surgical scenes. No large datasets comparable to industry standards in the computer vision community are available for this task. To circumvent this problem, we propose to automate the creation of a realistic training dataset by exploiting techniques stemming from special effects and harnessing them to target training performance rather than visual appeal. Foreground data is captured by placing sample surgical instruments over a chroma key (a.k.a. green screen) in a controlled environment, thereby making extraction of the relevant image segment straightforward. Multiple lighting conditions and viewpoints can be captured and introduced in the simulation by moving the instruments and camera and modulating the light source. Background data is captured by collecting videos that do not contain instruments. In the absence of pre-existing instrument-free background videos, minimal labeling effort is required, just to select frames that do not contain surgical instruments from videos of surgical interventions freely available online. We compare different methods to blend instruments over tissue and propose a novel data augmentation approach that takes advantage of the plurality of options. We show that by training a vanilla U-Net on semi-synthetic data only and applying a simple post-processing, we are able to match the results of the same network trained on a publicly available manually labeled real dataset
ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
Real-time tool segmentation from endoscopic videos is an essential part of
many computer-assisted robotic surgical systems and of critical importance in
robotic surgical data science. We propose two novel deep learning architectures
for automatic segmentation of non-rigid surgical instruments. Both methods take
advantage of automated deep-learning-based multi-scale feature extraction while
trying to maintain an accurate segmentation quality at all resolutions. The two
proposed methods encode the multi-scale constraint inside the network
architecture. The first proposed architecture enforces it by cascaded
aggregation of predictions and the second proposed network does it by means of
a holistically-nested architecture where the loss at each scale is taken into
account for the optimization process. As the proposed methods are for real-time
semantic labeling, both present a reduced number of parameters. We propose the
use of parametric rectified linear units for semantic labeling in these small
architectures to increase the regularization ability of the design and maintain
the segmentation accuracy without overfitting the training sets. We compare the
proposed architectures against state-of-the-art fully convolutional networks.
We validate our methods using existing benchmark datasets, including ex vivo
cases with phantom tissue and different robotic surgical instruments present in
the scene. Our results show a statistically significant improved Dice
Similarity Coefficient over previous instrument segmentation methods. We
analyze our design choices and discuss the key drivers for improving accuracy.Comment: Paper accepted at IROS 201
A spatio-temporal atlas of the developing fetal brain with spina bifida aperta [version 2; peer review: 2 approved]
Background: Spina bifida aperta (SBA) is a birth defect associated with severe anatomical changes in the developing fetal brain. Brain magnetic resonance imaging (MRI) atlases are popular tools for studying neuropathology in the brain anatomy, but previous fetal brain MRI atlases have focused on the normal fetal brain. We aimed to develop a spatio-temporal fetal brain MRI atlas for SBA.
Methods: We developed a semi-automatic computational method to compute the first spatio-temporal fetal brain MRI atlas for SBA. We used 90 MRIs of fetuses with SBA with gestational ages ranging from 21 to 35 weeks. Isotropic and motion-free 3D reconstructed MRIs were obtained for all the examinations. We propose a protocol for the annotation of anatomical landmarks in brain 3D MRI of fetuses with SBA with the aim of making spatial alignment of abnormal fetal brain MRIs more robust. In addition, we propose a weighted generalized Procrustes method based on the anatomical landmarks for the initialization of the atlas. The proposed weighted generalized Procrustes can handle temporal regularization and missing annotations. After initialization, the atlas is refined iteratively using non-linear image registration based on the image intensity and the anatomical land-marks. A semi-automatic method is used to obtain a parcellation of our fetal brain atlas into eight tissue types: white matter, ventricular system, cerebellum, extra-axial cerebrospinal fluid, cortical gray matter, deep gray matter, brainstem, and corpus callosum.
Results: An intra-rater variability analysis suggests that the seven anatomical land-marks are sufficiently reliable. We find that the proposed atlas outperforms a normal fetal brain atlas for the automatic segmentation of brain 3D MRI of fetuses with SBA.
Conclusions: We make publicly available a spatio-temporal fetal brain MRI atlas for SBA, available here: https://doi.org/10.7303/syn25887675. This atlas can support future research on automatic segmentation methods for brain 3D MRI of fetuses with SBA
NiftyNet: a deep-learning platform for medical imaging
Medical image analysis and computer-assisted intervention problems are
increasingly being addressed with deep-learning-based solutions. Established
deep-learning platforms are flexible but do not provide specific functionality
for medical image analysis and adapting them for this application requires
substantial implementation effort. Thus, there has been substantial duplication
of effort and incompatible infrastructure developed across many research
groups. This work presents the open-source NiftyNet platform for deep learning
in medical imaging. The ambition of NiftyNet is to accelerate and simplify the
development of these solutions, and to provide a common mechanism for
disseminating research outputs for the community to use, adapt and build upon.
NiftyNet provides a modular deep-learning pipeline for a range of medical
imaging applications including segmentation, regression, image generation and
representation learning applications. Components of the NiftyNet pipeline
including data loading, data augmentation, network architectures, loss
functions and evaluation metrics are tailored to, and take advantage of, the
idiosyncracies of medical image analysis and computer-assisted intervention.
NiftyNet is built on TensorFlow and supports TensorBoard visualization of 2D
and 3D images and computational graphs by default.
We present 3 illustrative medical image analysis applications built using
NiftyNet: (1) segmentation of multiple abdominal organs from computed
tomography; (2) image regression to predict computed tomography attenuation
maps from brain magnetic resonance images; and (3) generation of simulated
ultrasound images for specified anatomical poses.
NiftyNet enables researchers to rapidly develop and distribute deep learning
solutions for segmentation, regression, image generation and representation
learning applications, or extend the platform to new applications.Comment: Wenqi Li and Eli Gibson contributed equally to this work. M. Jorge
Cardoso and Tom Vercauteren contributed equally to this work. 26 pages, 6
figures; Update includes additional applications, updated author list and
formatting for journal submissio