22 research outputs found
BIAS: Transparent reporting of biomedical image analysis challenges
The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit
A large annotated medical image dataset for the development and evaluation of segmentation algorithms
Semantic segmentation of medical images aims to associate a pixel with a
label in a medical image without human initialization. The success of semantic
segmentation algorithms is contingent on the availability of high-quality
imaging data with corresponding labels provided by experts. We sought to create
a large collection of annotated medical image datasets of various clinically
relevant anatomies available under open source license to facilitate the
development of semantic segmentation algorithms. Such a resource would allow:
1) objective assessment of general-purpose segmentation methods through
comprehensive benchmarking and 2) open and free access to medical image data
for any researcher interested in the problem domain. Through a
multi-institutional effort, we generated a large, curated dataset
representative of several highly variable segmentation tasks that was used in a
crowd-sourced challenge - the Medical Segmentation Decathlon held during the
2018 Medical Image Computing and Computer Aided Interventions Conference in
Granada, Spain. Here, we describe these ten labeled image datasets so that
these data may be effectively reused by the research community
Common Limitations of Image Processing Metrics:A Picture Story
While the importance of automatic image analysis is continuously increasing,
recent meta-research revealed major flaws with respect to algorithm validation.
Performance metrics are particularly key for meaningful, objective, and
transparent performance assessment and validation of the used automatic
algorithms, but relatively little attention has been given to the practical
pitfalls when using specific metrics for a given image analysis task. These are
typically related to (1) the disregard of inherent metric properties, such as
the behaviour in the presence of class imbalance or small target structures,
(2) the disregard of inherent data set properties, such as the non-independence
of the test cases, and (3) the disregard of the actual biomedical domain
interest that the metrics should reflect. This living dynamically document has
the purpose to illustrate important limitations of performance metrics commonly
applied in the field of image analysis. In this context, it focuses on
biomedical image analysis problems that can be phrased as image-level
classification, semantic segmentation, instance segmentation, or object
detection task. The current version is based on a Delphi process on metrics
conducted by an international consortium of image analysis experts from more
than 60 institutions worldwide.Comment: This is a dynamic paper on limitations of commonly used metrics. The
current version discusses metrics for image-level classification, semantic
segmentation, object detection and instance segmentation. For missing use
cases, comments or questions, please contact [email protected] or
[email protected]. Substantial contributions to this document will be
acknowledged with a co-authorshi
Understanding metric-related pitfalls in image analysis validation
Validation metrics are key for the reliable tracking of scientific progress
and for bridging the current chasm between artificial intelligence (AI)
research and its translation into practice. However, increasing evidence shows
that particularly in image analysis, metrics are often chosen inadequately in
relation to the underlying research problem. This could be attributed to a lack
of accessibility of metric-related knowledge: While taking into account the
individual strengths, weaknesses, and limitations of validation metrics is a
critical prerequisite to making educated choices, the relevant knowledge is
currently scattered and poorly accessible to individual researchers. Based on a
multi-stage Delphi process conducted by a multidisciplinary expert consortium
as well as extensive community feedback, the present work provides the first
reliable and comprehensive common point of access to information on pitfalls
related to validation metrics in image analysis. Focusing on biomedical image
analysis but with the potential of transfer to other fields, the addressed
pitfalls generalize across application domains and are categorized according to
a newly created, domain-agnostic taxonomy. To facilitate comprehension,
illustrations and specific examples accompany each pitfall. As a structured
body of information accessible to researchers of all levels of expertise, this
work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior
authors: Paul F. J\"ager, Lena Maier-Hei
The Medical Segmentation Decathlon
International challenges have become the de facto standard for comparative
assessment of image analysis algorithms given a specific task. Segmentation is
so far the most widely investigated medical image processing task, but the
various segmentation challenges have typically been organized in isolation,
such that algorithm development was driven by the need to tackle a single
specific clinical problem. We hypothesized that a method capable of performing
well on multiple tasks will generalize well to a previously unseen task and
potentially outperform a custom-designed solution. To investigate the
hypothesis, we organized the Medical Segmentation Decathlon (MSD) - a
biomedical image analysis challenge, in which algorithms compete in a multitude
of both tasks and modalities. The underlying data set was designed to explore
the axis of difficulties typically encountered when dealing with medical
images, such as small data sets, unbalanced labels, multi-site data and small
objects. The MSD challenge confirmed that algorithms with a consistent good
performance on a set of tasks preserved their good average performance on a
different set of previously unseen tasks. Moreover, by monitoring the MSD
winner for two years, we found that this algorithm continued generalizing well
to a wide range of other clinical problems, further confirming our hypothesis.
Three main conclusions can be drawn from this study: (1) state-of-the-art image
segmentation algorithms are mature, accurate, and generalize well when
retrained on unseen tasks; (2) consistent algorithmic performance across
multiple tasks is a strong surrogate of algorithmic generalizability; (3) the
training of accurate AI segmentation models is now commoditized to non AI
experts
Why rankings of biomedical image analysis competitions should be interpreted with care
International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future
Exploiting the potential of unlabeled endoscopic video data with self-supervised learning
Projective biomechanical depth matching for soft tissue registration in laparoscopic surgery
Robust near real-time estimation of physiological parameters from megapixel multispectral images with inverse Monte Carlo and random forest regression
PURPOSE: Multispectral imaging can provide reflectance measurements at multiple spectral bands for each image pixel. These measurements can be used for estimation of important physiological parameters, such as oxygenation, which can provide indicators for the success of surgical treatment or the presence of abnormal tissue. The goal of this work was to develop a method to estimate physiological parameters in an accurate and rapid manner suited for modern high-resolution laparoscopic images. METHODS: While previous methods for oxygenation estimation are based on either simple linear methods or complex model-based approaches exclusively suited for off-line processing, we propose a new approach that combines the high accuracy of model-based approaches with the speed and robustness of modern machine learning methods. Our concept is based on training random forest regressors using reflectance spectra generated with Monte Carlo simulations. RESULTS: According to extensive in silico and in vivo experiments, the method features higher accuracy and robustness than state-of-the-art online methods and is orders of magnitude faster than other nonlinear regression based methods. CONCLUSION: Our current implementation allows for near real-time oxygenation estimation from megapixel multispectral images and is thus well suited for online tissue analysis