Common Limitations of Image Processing Metrics:A Picture Story

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.Comment: This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact [email protected] or [email protected]. Substantial contributions to this document will be acknowledged with a co-authorshi

Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior authors: Paul F. J\"ager, Lena Maier-Hei

Continuous small-molecule monitoring with a digital single particle switch

The ability to continuously measure concentrations of small molecules is important for biomedical, environmental and industrial monitoring. However, due to their low molecular mass it is difficult to quantify concentrations of such molecules, particularly at low concentrations. Here we describe a small-molecule sensor that is generalizable, sensitive, specific, reversible, and suited for continuous monitoring over long durations. The sensor consists of particles attached to a sensing surface via a double stranded DNA tether. The particles transiently bind to the sensing surface via single molecular affinity interactions and the transient binding is optically detected as digital binding events via the Brownian motion of the particles. The rate of binding events decreases with increasing analyte concentration, because analyte molecules inhibit binding of the tethered particle to the surface. The sensor enables continuous measurements of analyte concentrations due to the reversibility of the inter-molecular bonds and digital read-out of particle motion. We show results for the monitoring of short single-stranded DNA sequences and creatinine, a small-molecule biomarker (113 Da) for kidney function, demonstrating a temporal resolution of a few minutes. The precision of the sensor is determined by the statistics of the digital switching events, which means that the precision is tunable by the number of particles and the measurement time

Continuous Small-Molecule Monitoring with a Digital Single-Particle Switch

The ability to continuously measure concentrations of small molecules is important for biomedical, environmental and industrial monitoring. However, due to their low molecular mass it is difficult to quantify concentrations of such molecules, particularly at low concentrations. Here we describe a small-molecule sensor that is generalizable, sensitive, specific, reversible, and suited for continuous monitoring over long durations. The sensor consists of particles attached to a sensing surface via a double stranded DNA tether. The particles transiently bind to the sensing surface via single molecular affinity interactions and the transient binding is optically detected as digital binding events via the Brownian motion of the particles. The rate of binding events decreases with increasing analyte concentration, because analyte molecules inhibit binding of the tethered particle to the surface. The sensor enables continuous measurements of analyte concentrations due to the reversibility of the inter-molecular bonds and digital read-out of particle motion. We show results for the monitoring of short single-stranded DNA sequences and creatinine, a small-molecule biomarker (113 Da) for kidney function, demonstrating a temporal resolution of a few minutes. The precision of the sensor is determined by the statistics of the digital switching events, which means that the precision is tunable by the number of particles and the measurement time

Reversible Immunosensor for the Continuous Monitoring of Cortisol in Blood Plasma Sampled with Microdialysis

Cortisol is a steroid hormone involved in a wide range of medical conditions. The level of the hormone fluctuates over time, but with traditional laboratory-based assays, such dynamics cannot be monitored in real time. Here, a reversible cortisol sensor is reported that allows continuous monitoring of cortisol in blood plasma using sampling by microdialysis. The sensor is based on measuring single-molecule binding and unbinding events of tethered particles. The particles are functionalized with antibodies and the substrate with cortisol-analogues, causing binding and unbinding events to occur between particles and substrate. The frequency of binding events is reduced when cortisol is present in the solution as it blocks the binding sites of the antibodies. The sensor responds to cortisol in the high nanomolar to low micromolar range and can monitor cortisol concentrations over multiple hours. Results are shown for cortisol monitoring in filtered and in microdialysis-sampled human blood plasma

Continuous biomarker monitoring with single molecule resolution by measuring free particle motion

There is a need for sensing technologies that can continuously monitor concentration levels of critical biomolecules in applications such as patient care, fundamental biological research, biotechnology and food industry, as well as the environment. However, it is fundamentally difficult to develop measurement technologies that are not only sensitive and specific, but also allow monitoring over a broad concentration range and over long timespans. Here we describe a continuous biomolecular sensing methodology based on the free diffusion of biofunctionalized particles hovering over a sensor surface. The method records digital events due to single-molecule interactions and enables biomarker monitoring at picomolar to micromolar concentrations without consuming any reagents. We demonstrate the affinity-based sensing methodology for DNA-based sandwich and competition assays, and for an antibody-based cortisol assay. Additionally, the sensor can be dried, facilitating storage over weeks while maintaining its sensitivity. We foresee that this will enable the development of continuous monitoring sensors for applications in fundamental research, for studies on organs on a chip, for the monitoring of patients in critical care, and for the monitoring of industrial processes and bioreactors as well as ecological systems

Do we become better prescribers after graduation : a 1-year international follow-up study among junior doctors

Aim: The aim of this study was to investigate how the prescribing knowledge and skills of junior doctors in the Netherlands and Belgium develop in the year after graduation. We also analysed differences in knowledge and skills between surgical and nonsurgical junior doctors. Methods: This international, multicentre (n = 11), longitudinal study analysed the learning curves of junior doctors working in various specialties via three validated assessments at about the time of graduation, and 6 months and 1 year after graduation. Each assessment contained 35 multiple choice questions (MCQs) on medication safety (passing grade ≥85%) and three clinical scenarios. Results: In total, 556 junior doctors participated, 326 (58.6%) of whom completed the MCQs and 325 (58.5%) the clinical case scenarios of all three assessments. Mean prescribing knowledge was stable in the year after graduation, with 69% (SD 13) correctly answering questions at assessment 1 and 71% (SD 14) at assessment 3, whereas prescribing skills decreased: 63% of treatment plans were considered adequate at assessment 1 but only 40% at assessment 3 (P <.001). While nonsurgical doctors had similar learning curves for knowledge and skills as surgical doctors (P =.53 and P =.56 respectively), their overall level was higher at all three assessments (all P <.05). Conclusion: These results show that junior doctors' prescribing knowledge and skills did not improve while they were working in clinical practice. Moreover, their level was under the predefined passing grade. As this might adversely affect patient safety, educational interventions should be introduced to improve the prescribing competence of junior doctors

Do we become better prescribers after graduation: A 1-year international follow-up study among junior doctors

Aim: The aim of this study was to investigate how the prescribing knowledge and skills of junior doctors in the Netherlands and Belgium develop in the year after graduation. We also analysed differences in knowledge and skills between surgical and nonsurgical junior doctors. Methods: This international, multicentre (n = 11), longitudinal study analysed the learning curves of junior doctors working in various specialties via three validated assessments at about the time of graduation, and 6 months and 1 year after graduation. Each assessment contained 35 multiple choice questions (MCQs) on medication safety (passing grade ≥85%) and three clinical scenarios. Results: In total, 556 junior doctors participated, 326 (58.6%) of whom completed the MCQs and 325 (58.5%) the clinical case scenarios of all three assessments. Mean prescribing knowledge was stable in the year after graduation, with 69% (SD 13) correctly answering questions at assessment 1 and 71% (SD 14) at assessment 3, whereas prescribing skills decreased: 63% of treatment plans were considered adequate at assessment 1 but only 40% at assessment 3 (P <.001). While nonsurgical doctors had similar learning curves for knowledge and skills as surgical doctors (P =.53 and P =.56 respectively), their overall level was higher at all three assessments (all P <.05). Conclusion: These results show that junior doctors' prescribing knowledge and skills did not improve while they were working in clinical practice. Moreover, their level was under the predefined passing grade. As this might adversely affect patient safety, educational interventions should be introduced to improve the prescribing competence of junior doctors