Potential climatic transitions with profound impact on Europe

We discuss potential transitions of six climatic subsystems with large-scale impact on Europe, sometimes denoted as tipping elements. These are the ice sheets on Greenland and West Antarctica, the Atlantic thermohaline circulation, Arctic sea ice, Alpine glaciers and northern hemisphere stratospheric ozone. Each system is represented by co-authors actively publishing in the corresponding field. For each subsystem we summarize the mechanism of a potential transition in a warmer climate along with its impact on Europe and assess the likelihood for such a transition based on published scientific literature. As a summary, the ‘tipping’ potential for each system is provided as a function of global mean temperature increase which required some subjective interpretation of scientific facts by the authors and should be considered as a snapshot of our current understanding. <br/

Common Limitations of Image Processing Metrics:A Picture Story

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.Comment: This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact [email protected] or [email protected]. Substantial contributions to this document will be acknowledged with a co-authorshi

Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior authors: Paul F. J\"ager, Lena Maier-Hei

Why rankings of biomedical image analysis competitions should be interpreted with care

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future

How to exploit weaknesses in biomedical challenge design and organization

\u3cp\u3eSince the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.\u3c/p\u3

How to exploit weaknesses in biomedical challenge design and organization

Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization

Prognostic value of improvement endpoints in pulmonary arterial hypertension trials: A COMPERA analysis

BACKGROUND: The prognostic value of improvement endpoints that have been used in clinical trials of treatments for pulmonary arterial hypertension (PAH) needs to be further investigated. METHODS: Using the COMPERA database, we evaluated the prognostic value of improvements in functional class (FC) and absolute or relative improvements in 6-min walking distance (6MWD) and N-terminal fragment of pro-brain natriuretic peptide (NT-proBNP). In addition, we investigated multicomponent endpoints based on prespecified improvements in FC, 6MWD and NT-proBNP that have been used in recent PAH trials. Finally, we assessed the predictive value of improvements determined by risk stratification tools. The effects of changes from baseline to first follow-up (3-12 months after initiation of PAH therapy) on consecutive survival were determined by Kaplan-Meier analysis with Log-Rank testing and Cox proportional hazard analyses. RESULTS: All analyses were based on 596 patients with newly diagnosed PAH for whom complete data were available at baseline and first follow-up. Improvements in FC were associated with improved survival, whereas absolute or relative improvements in 6MWD had no predictive value. For NT-proBNP, absolute declines conferred no prognostic information while relative declines by ≥35% were associated with better survival. Improvements in multicomponent endpoints were associated with improved survival and the same was found for risk stratification tools. CONCLUSION: While sole improvements in 6MWD and NT-proBNP had minor prognostic relevance, improvements in multicomponent endpoints and risk stratification tools based on FC, 6MWD, and NT-proBNP were associated with improved survival. These tools should be further explored as outcome measures in PAH trials