11 research outputs found
Common Limitations of Image Processing Metrics:A Picture Story
While the importance of automatic image analysis is continuously increasing,
recent meta-research revealed major flaws with respect to algorithm validation.
Performance metrics are particularly key for meaningful, objective, and
transparent performance assessment and validation of the used automatic
algorithms, but relatively little attention has been given to the practical
pitfalls when using specific metrics for a given image analysis task. These are
typically related to (1) the disregard of inherent metric properties, such as
the behaviour in the presence of class imbalance or small target structures,
(2) the disregard of inherent data set properties, such as the non-independence
of the test cases, and (3) the disregard of the actual biomedical domain
interest that the metrics should reflect. This living dynamically document has
the purpose to illustrate important limitations of performance metrics commonly
applied in the field of image analysis. In this context, it focuses on
biomedical image analysis problems that can be phrased as image-level
classification, semantic segmentation, instance segmentation, or object
detection task. The current version is based on a Delphi process on metrics
conducted by an international consortium of image analysis experts from more
than 60 institutions worldwide.Comment: This is a dynamic paper on limitations of commonly used metrics. The
current version discusses metrics for image-level classification, semantic
segmentation, object detection and instance segmentation. For missing use
cases, comments or questions, please contact [email protected] or
[email protected]. Substantial contributions to this document will be
acknowledged with a co-authorshi
Understanding metric-related pitfalls in image analysis validation
Validation metrics are key for the reliable tracking of scientific progress
and for bridging the current chasm between artificial intelligence (AI)
research and its translation into practice. However, increasing evidence shows
that particularly in image analysis, metrics are often chosen inadequately in
relation to the underlying research problem. This could be attributed to a lack
of accessibility of metric-related knowledge: While taking into account the
individual strengths, weaknesses, and limitations of validation metrics is a
critical prerequisite to making educated choices, the relevant knowledge is
currently scattered and poorly accessible to individual researchers. Based on a
multi-stage Delphi process conducted by a multidisciplinary expert consortium
as well as extensive community feedback, the present work provides the first
reliable and comprehensive common point of access to information on pitfalls
related to validation metrics in image analysis. Focusing on biomedical image
analysis but with the potential of transfer to other fields, the addressed
pitfalls generalize across application domains and are categorized according to
a newly created, domain-agnostic taxonomy. To facilitate comprehension,
illustrations and specific examples accompany each pitfall. As a structured
body of information accessible to researchers of all levels of expertise, this
work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior
authors: Paul F. J\"ager, Lena Maier-Hei
Tattoo tomography: Freehand 3D photoacoustic image reconstruction with an optical pattern
Purpose!#!Photoacoustic tomography (PAT) is a novel imaging technique that can spatially resolve both morphological and functional tissue properties, such as vessel topology and tissue oxygenation. While this capacity makes PAT a promising modality for the diagnosis, treatment, and follow-up of various diseases, a current drawback is the limited field of view provided by the conventionally applied 2D probes.!##!Methods!#!In this paper, we present a novel approach to 3D reconstruction of PAT data (Tattoo tomography) that does not require an external tracking system and can smoothly be integrated into clinical workflows. It is based on an optical pattern placed on the region of interest prior to image acquisition. This pattern is designed in a way that a single tomographic image of it enables the recovery of the probe pose relative to the coordinate system of the pattern, which serves as a global coordinate system for image compounding.!##!Results!#!To investigate the feasibility of Tattoo tomography, we assessed the quality of 3D image reconstruction with experimental phantom data and in vivo forearm data. The results obtained with our prototype indicate that the Tattoo method enables the accurate and precise 3D reconstruction of PAT data and may be better suited for this task than the baseline method using optical tracking.!##!Conclusions!#!In contrast to previous approaches to 3D ultrasound (US) or PAT reconstruction, the Tattoo approach neither requires complex external hardware nor training data acquired for a specific application. It could thus become a valuable tool for clinical freehand PAT
SMARCE1 is required for the invasive progression of in situ cancers
Advances in mammography have sparked an exponential increase in the detection of early-stage breast lesions, most commonly ductal carcinoma in situ (DCIS). More than 50% of DCIS lesions are benign and will remain indolent, never progressing to invasive cancers. However, the factors that promote DCIS invasion remain poorly understood. Here, we show that SMARCE1 is required for the invasive progression of DCIS and other early-stage tumors. We show that SMARCE1 drives invasion by regulating the expression of secreted proteases that degrade basement membrane, an ECM barrier surrounding all epithelial tissues. In functional studies, SMARCE1 promotes invasion of in situ cancers growing within primary human mammary tissues and is also required for metastasis in vivo. Mechanistically, SMARCE1 drives invasion by forming a SWI/SNF-independent complex with the transcription factor ILF3. In patients diagnosed with early-stage cancers, SMARCE1 expression is a strong predictor of eventual relapse and metastasis. Collectively, these findings establish SMARCE1 as a key driver of invasive progression in early-stage tumors. Keywords: DCIS; invasive progression; biomarker; SMARCE1National Science Foundation (U.S.) (Grant 1122374
Surgical data science â from concepts toward clinical translation
International audienceRecent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process
Metrics reloaded: Pitfalls and recommendations for image analysis validation
The field of automatic biomedical image analysis crucially depends on robust
and meaningful performance metrics for algorithm validation. Current metric
usage, however, is often ill-informed and does not reflect the underlying
domain interest. Here, we present a comprehensive framework that guides
researchers towards choosing performance metrics in a problem-aware manner.
Specifically, we focus on biomedical image analysis problems that can be
interpreted as a classification task at image, object or pixel level. The
framework first compiles domain interest-, target structure-, data set- and
algorithm output-related properties of a given problem into a problem
fingerprint, while also mapping it to the appropriate problem category, namely
image-level classification, semantic segmentation, instance segmentation, or
object detection. It then guides users through the process of selecting and
applying a set of appropriate validation metrics while making them aware of
potential pitfalls related to individual choices. In this paper, we describe
the current status of the Metrics Reloaded recommendation framework, with the
goal of obtaining constructive feedback from the image analysis community. The
current version has been developed within an international consortium of more
than 60 image analysis experts and will be made openly available as a
user-friendly toolkit after community-driven optimization.Comment: Shared first authors: Lena Maier-Hein, Annika Reinke. arXiv admin
note: substantial text overlap with arXiv:2104.0564
Metrics reloaded : recommendations for image analysis validation
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases. [Abstract copyright: © 2024. Springer Nature America, Inc.