5 research outputs found
How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation
Recent developments in deep learning (DL) techniques have led to great
performance improvement in medical image segmentation tasks, especially with
the latest Transformer model and its variants. While labels from fusing
multi-rater manual segmentations are often employed as ideal ground truths in
DL model training, inter-rater variability due to factors such as training
bias, image noise, and extreme anatomical variability can still affect the
performance and uncertainty of the resulting algorithms. Knowledge regarding
how inter-rater variability affects the reliability of the resulting DL
algorithms, a key element in clinical deployment, can help inform better
training data construction and DL models, but has not been explored
extensively. In this paper, we measure aleatoric and epistemic uncertainties
using test-time augmentation (TTA), test-time dropout (TTD), and deep ensemble
to explore their relationship with inter-rater variability. Furthermore, we
compare UNet and TransUNet to study the impacts of Transformers on model
uncertainty with two label fusion strategies. We conduct a case study using
multi-class paraspinal muscle segmentation from T2w MRIs. Our study reveals the
interplay between inter-rater variability and uncertainties, affected by
choices of label fusion strategies and DL models.Comment: Accepted in UNSURE MICCAI 202
Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias
Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias
A survey of uncertainty in deep neural networks
Over the last decade, neural networks have reached almost every field of science and become a crucial part of various real world applications. Due to the increasing spread, confidence in neural network predictions has become more and more important. However, basic neural networks do not deliver certainty estimates or suffer from over- or under-confidence, i.e. are badly calibrated. To overcome this, many researchers have been working on understanding and quantifying uncertainty in a neural network's prediction. As a result, different types and sources of uncertainty have been identified and various approaches to measure and quantify uncertainty in neural networks have been proposed. This work gives a comprehensive overview of uncertainty estimation in neural networks, reviews recent advances in the field, highlights current challenges, and identifies potential research opportunities. It is intended to give anyone interested in uncertainty estimation in neural networks a broad overview and introduction, without presupposing prior knowledge in this field. For that, a comprehensive introduction to the most crucial sources of uncertainty is given and their separation into reducible model uncertainty and irreducible data uncertainty is presented. The modeling of these uncertainties based on deterministic neural networks, Bayesian neural networks (BNNs), ensemble of neural networks, and test-time data augmentation approaches is introduced and different branches of these fields as well as the latest developments are discussed. For a practical application, we discuss different measures of uncertainty, approaches for calibrating neural networks, and give an overview of existing baselines and available implementations. Different examples from the wide spectrum of challenges in the fields of medical image analysis, robotics, and earth observation give an idea of the needs and challenges regarding uncertainties in the practical applications of neural networks. Additionally, the practical limitations of uncertainty quantification methods in neural networks for mission- and safety-critical real world applications are discussed and an outlook on the next steps towards a broader usage of such methods is given