20 research outputs found
Analyzing the Quality and Challenges of Uncertainty Estimations for Brain Tumor Segmentation
Automatic segmentation of brain tumors has the potential to enable volumetric measures and high-throughput analysis in the clinical setting. Reaching this potential seems almost achieved, considering the steady increase in segmentation accuracy. However, despite segmentation accuracy, the current methods still do not meet the robustness levels required for patient-centered clinical use. In this regard, uncertainty estimates are a promising direction to improve the robustness of automated segmentation systems. Different uncertainty estimation methods have been proposed, but little is known about their usefulness and limitations for brain tumor segmentation. In this study, we present an analysis of the most commonly used uncertainty estimation methods in regards to benefits and challenges for brain tumor segmentation. We evaluated their quality in terms of calibration, segmentation error localization, and segmentation failure detection. Our results show that the uncertainty methods are typically well-calibrated when evaluated at the dataset level. Evaluated at the subject level, we found notable miscalibrations and limited segmentation error localization (e.g., for correcting segmentations), which hinder the direct use of the voxel-wise uncertainties. Nevertheless, voxel-wise uncertainty showed value to detect failed segmentations when uncertainty estimates are aggregated at the subject level. Therefore, we suggest a careful usage of voxel-wise uncertainty measures and highlight the importance of developing solutions that address the subject-level requirements on calibration and segmentation error localization
On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation
Uncertainty estimation methods are expected to improve the understanding and
quality of computer-assisted methods used in medical applications (e.g.,
neurosurgical interventions, radiotherapy planning), where automated medical
image segmentation is crucial. In supervised machine learning, a common
practice to generate ground truth label data is to merge observer annotations.
However, as many medical image tasks show a high inter-observer variability
resulting from factors such as image quality, different levels of user
expertise and domain knowledge, little is known as to how inter-observer
variability and commonly used fusion methods affect the estimation of
uncertainty of automated image segmentation. In this paper we analyze the
effect of common image label fusion techniques on uncertainty estimation, and
propose to learn the uncertainty among observers. The results highlight the
negative effect of fusion methods applied in deep learning, to obtain reliable
estimates of segmentation uncertainty. Additionally, we show that the learned
observers' uncertainty can be combined with current standard Monte Carlo
dropout Bayesian neural networks to characterize uncertainty of model's
parameters.Comment: Appears in Medical Image Computing and Computer Assisted
Interventions (MICCAI), 201
pymia: A Python package for data handling and evaluation in deep learning-based medical image analysis
Background and Objective: Deep learning enables tremendous progress in
medical image analysis. One driving force of this progress are open-source
frameworks like TensorFlow and PyTorch. However, these frameworks rarely
address issues specific to the domain of medical image analysis, such as 3-D
data handling and distance metrics for evaluation. pymia, an open-source Python
package, tries to address these issues by providing flexible data handling and
evaluation independent of the deep learning framework.
Methods: The pymia package provides data handling and evaluation
functionalities. The data handling allows flexible medical image handling in
every commonly used format (e.g., 2-D, 2.5-D, and 3-D; full- or patch-wise).
Even data beyond images like demographics or clinical reports can easily be
integrated into deep learning pipelines. The evaluation allows stand-alone
result calculation and reporting, as well as performance monitoring during
training using a vast amount of domain-specific metrics for segmentation,
reconstruction, and regression.
Results: The pymia package is highly flexible, allows for fast prototyping,
and reduces the burden of implementing data handling routines and evaluation
methods. While data handling and evaluation are independent of the deep
learning framework used, they can easily be integrated into TensorFlow and
PyTorch pipelines. The developed package was successfully used in a variety of
research projects for segmentation, reconstruction, and regression.
Conclusions: The pymia package fills the gap of current deep learning
frameworks regarding data handling and evaluation in medical image analysis. It
is available at https://github.com/rundherum/pymia and can directly be
installed from the Python Package Index using pip install pymia.Comment: first and last author contributed equall
Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation
Uncertainty estimates of modern neuronal networks provide additional
information next to the computed predictions and are thus expected to improve
the understanding of the underlying model. Reliable uncertainties are
particularly interesting for safety-critical computer-assisted applications in
medicine, e.g., neurosurgical interventions and radiotherapy planning. We
propose an uncertainty-driven sanity check for the identification of
segmentation results that need particular expert review. Our method uses a
fully-convolutional neural network and computes uncertainty estimates by the
principle of Monte Carlo dropout. We evaluate the performance of the proposed
method on a clinical dataset with 30 postoperative brain tumor images. The
method can segment the highly inhomogeneous resection cavities accurately (Dice
coefficients 0.792 0.154). Furthermore, the proposed sanity check is able
to detect the worst segmentation and three out of the four outliers. The
results highlight the potential of using the additional information from the
model's parameter uncertainty to validate the segmentation performance of a
deep learning model.Comment: Appears in Medical Imaging with Deep Learning (MIDL), 201
Learning Bloch Simulations for MR Fingerprinting by Invertible Neural Networks
Magnetic resonance fingerprinting (MRF) enables fast and multiparametric MR
imaging. Despite fast acquisition, the state-of-the-art reconstruction of MRF
based on dictionary matching is slow and lacks scalability. To overcome these
limitations, neural network (NN) approaches estimating MR parameters from
fingerprints have been proposed recently. Here, we revisit NN-based MRF
reconstruction to jointly learn the forward process from MR parameters to
fingerprints and the backward process from fingerprints to MR parameters by
leveraging invertible neural networks (INNs). As a proof-of-concept, we perform
various experiments showing the benefit of learning the forward process, i.e.,
the Bloch simulations, for improved MR parameter estimation. The benefit
especially accentuates when MR parameter estimation is difficult due to MR
physical restrictions. Therefore, INNs might be a feasible alternative to the
current solely backward-based NNs for MRF reconstruction.Comment: Accepted at MICCAI MLMIR 202
Unsupervised out-of-distribution detection for safer robotically-guided retinal microsurgery
Purpose: A fundamental problem in designing safe machine learning systems is
identifying when samples presented to a deployed model differ from those
observed at training time. Detecting so-called out-of-distribution (OoD)
samples is crucial in safety-critical applications such as robotically-guided
retinal microsurgery, where distances between the instrument and the retina are
derived from sequences of 1D images that are acquired by an
instrument-integrated optical coherence tomography (iiOCT) probe.
Methods: This work investigates the feasibility of using an OoD detector to
identify when images from the iiOCT probe are inappropriate for subsequent
machine learning-based distance estimation. We show how a simple OoD detector
based on the Mahalanobis distance can successfully reject corrupted samples
coming from real-world ex-vivo porcine eyes.
Results: Our results demonstrate that the proposed approach can successfully
detect OoD samples and help maintain the performance of the downstream task
within reasonable levels. MahaAD outperformed a supervised approach trained on
the same kind of corruptions and achieved the best performance in detecting OoD
cases from a collection of iiOCT samples with real-world corruptions.
Conclusion: The results indicate that detecting corrupted iiOCT data through
OoD detection is feasible and does not need prior knowledge of possible
corruptions. Consequently, MahaAD could aid in ensuring patient safety during
robotically-guided microsurgery by preventing deployed prediction models from
estimating distances that put the patient at risk.Comment: Accepted at IPCAI 202
Tournesol: Permissionless Collaborative Algorithmic Governance with Security Guarantees
Recommendation algorithms play an increasingly central role in our societies.
However, thus far, these algorithms are mostly designed and parameterized
unilaterally by private groups or governmental authorities. In this paper, we
present an end-to-end permissionless collaborative algorithmic governance
method with security guarantees. Our proposed method is deployed as part of an
open-source content recommendation platform https://tournesol.app, whose
recommender is collaboratively parameterized by a community of (non-technical)
contributors. This algorithmic governance is achieved through three main steps.
First, the platform contains a mechanism to assign voting rights to the
contributors. Second, the platform uses a comparison-based model to evaluate
the individual preferences of contributors. Third, the platform aggregates the
judgements of all contributors into collective scores for content
recommendations. We stress that the first and third steps are vulnerable to
attacks from malicious contributors. To guarantee the resilience against fake
accounts, the first step combines email authentication, a vouching mechanism, a
novel variant of the reputation-based EigenTrust algorithm and an adaptive
voting rights assignment for alternatives that are scored by too many untrusted
accounts. To provide resilience against malicious authenticated contributors,
we adapt Mehestan, an algorithm previously proposed for robust sparse voting.
We believe that these algorithms provide an appealing foundation for a
collaborative, effective, scalable, fair, contributor-friendly, interpretable
and secure governance. We conclude by highlighting key challenges to make our
solution applicable to larger-scale settings.Comment: 31 pages, 5 figure
The SIB Swiss Institute of Bioinformatics' resources: focus on curated databases
The SIB Swiss Institute of Bioinformatics (www.isb-sib.ch) provides world-class bioinformatics databases, software tools, services and training to the international life science community in academia and industry. These solutions allow life scientists to turn the exponentially growing amount of data into knowledge. Here, we provide an overview of SIB's resources and competence areas, with a strong focus on curated databases and SIB's most popular and widely used resources. In particular, SIB's Bioinformatics resource portal ExPASy features over 150 resources, including UniProtKB/Swiss-Prot, ENZYME, PROSITE, neXtProt, STRING, UniCarbKB, SugarBindDB, SwissRegulon, EPD, arrayMap, Bgee, SWISS-MODEL Repository, OMA, OrthoDB and other databases, which are briefly described in this article