135 research outputs found
Metrics reloaded: Pitfalls and recommendations for image analysis validation
The field of automatic biomedical image analysis crucially depends on robust and meaningful performance metrics for algorithm validation. Current metric usage, however, is often ill-informed and does not reflect the underlying domain interest. Here, we present a comprehensive framework that guides researchers towards choosing performance metrics in a problem-aware manner. Specifically, we focus on biomedical image analysis problems that can be interpreted as a classification task at image, object or pixel level. The framework first compiles domain interest-, target structure-, data set- and algorithm output-related properties of a given problem into a problem fingerprint, while also mapping it to the appropriate problem category, namely image-level classification, semantic segmentation, instance segmentation, or object detection. It then guides users through the process of selecting and applying a set of appropriate validation metrics while making them aware of potential pitfalls related to individual choices. In this paper, we describe the current status of the Metrics Reloaded recommendation framework, with the goal of obtaining constructive feedback from the image analysis community. The current version has been developed within an international consortium of more than 60 image analysis experts and will be made openly available as a user-friendly toolkit after community-driven optimization
Challenge Results Are Not Reproducible
While clinical trials are the state-of-the-art methods to assess the effect
of new medication in a comparative manner, benchmarking in the field of medical
image analysis is performed by so-called challenges. Recently, comprehensive
analysis of multiple biomedical image analysis challenges revealed large
discrepancies between the impact of challenges and quality control of the
design and reporting standard. This work aims to follow up on these results and
attempts to address the specific question of the reproducibility of the
participants methods. In an effort to determine whether alternative
interpretations of the method description may change the challenge ranking, we
reproduced the algorithms submitted to the 2019 Robust Medical Image
Segmentation Challenge (ROBUST-MIS). The leaderboard differed substantially
between the original challenge and reimplementation, indicating that challenge
rankings may not be sufficiently reproducible.Comment: Accepted at BVM 202
- …