BIAS: Transparent reporting of biomedical image analysis challenges

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit

Optimizing expert and patient input in pediatric trial design : lessons learned and recommendations from a collaboration between conect4children and European Patient‐CEntric ClinicAl TRial PLatforms

Advice from multiple stakeholders is required to design the optimal pediatric clinical trial. We present recommendations for acquiring advice from trial experts and patients/caregivers, derived from advice meetings that were performed through a collaboration of the Collaborative Network for European Clinical Trials for Children (c4c) and the European Patient‐CEntric ClinicAl TRial PLatforms (EU‐PEARL). Three advice meetings were performed: (1) an advice meeting for clinical and methodology experts, (2) an advice meeting for patients/caregivers, and (3) a combined meeting with both experts and patients/caregivers. Trial experts were recruited from c4c database. Patients/caregivers were recruited through a patient organization. Participants were asked to provide input on a trial protocol, including endpoints, outcomes, and the assessment schedule. Ten experts, 10 patients, and 13 caregivers participated. The advice meetings resulted in modification of eligibility criteria and outcome measures. We have provided recommendations for the most effective meeting type per protocol topic. Topics with limited options for patient input were most efficiently discussed in expert advice meetings. Other topics benefit from patient/caregiver input, either through a combined meeting with experts or a patients/caregivers‐only advice meeting. Some topics, such as endpoints and outcome measures, are suitable for all meeting types. Combined sessions profit from synergy between experts and patients/caregivers, balancing input on protocol scientific feasibility and acceptability. Both experts and patients/caregivers provided critical input on the presented protocol. The combined meeting was the most effective methodology for most protocol topics. The presented methodology can be used effectively to acquire expert and patient feedback

A large annotated medical image dataset for the development and evaluation of segmentation algorithms

Semantic segmentation of medical images aims to associate a pixel with a label in a medical image without human initialization. The success of semantic segmentation algorithms is contingent on the availability of high-quality imaging data with corresponding labels provided by experts. We sought to create a large collection of annotated medical image datasets of various clinically relevant anatomies available under open source license to facilitate the development of semantic segmentation algorithms. Such a resource would allow: 1) objective assessment of general-purpose segmentation methods through comprehensive benchmarking and 2) open and free access to medical image data for any researcher interested in the problem domain. Through a multi-institutional effort, we generated a large, curated dataset representative of several highly variable segmentation tasks that was used in a crowd-sourced challenge - the Medical Segmentation Decathlon held during the 2018 Medical Image Computing and Computer Aided Interventions Conference in Granada, Spain. Here, we describe these ten labeled image datasets so that these data may be effectively reused by the research community

The Medical Segmentation Decathlon

International challenges have become the de facto standard for comparative assessment of image analysis algorithms. Although segmentation is the most widely investigated medical image processing task, the various challenges have been organized to focus only on specific clinical tasks. We organized the Medical Segmentation Decathlon (MSD)—a biomedical image analysis challenge, in which algorithms compete in a multitude of both tasks and modalities to investigate the hypothesis that a method capable of performing well on multiple tasks will generalize well to a previously unseen task and potentially outperform a custom-designed solution. MSD results confirmed this hypothesis, moreover, MSD winner continued generalizing well to a wide range of other clinical problems for the next two years. Three main conclusions can be drawn from this study: (1) state-of-the-art image segmentation algorithms generalize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasks is a strong surrogate of algorithmic generalizability; (3) the training of accurate AI segmentation models is now commoditized to scientists that are not versed in AI model training

E6 and E7 from Beta Hpv38 Cooperate with Ultraviolet Light in the Development of Actinic Keratosis-Like Lesions and Squamous Cell Carcinoma in Mice

Cutaneous beta human papillomavirus (HPV) types appear to be involved in the development of non-melanoma skin cancer (NMSC); however, it is not entirely clear whether they play a direct role. We have previously shown that E6 and E7 oncoproteins from the beta HPV type 38 display transforming activities in several experimental models. To evaluate the possible contribution of HPV38 in a proliferative tissue compartment during carcinogenesis, we generated a new transgenic mouse model (Tg) where HPV38 E6 and E7 are expressed in the undifferentiated basal layer of epithelia under the control of the Keratin 14 (K14) promoter. Viral oncogene expression led to increased cellular proliferation in the epidermis of the Tg animals in comparison to the wild-type littermates. Although no spontaneous formation of tumours was observed during the lifespan of the K14 HPV38 E6/E7-Tg mice, they were highly susceptible to 7,12-dimethylbenz(a)anthracene (DMBA)/12-0-tetradecanoylphorbol-13-acetate (TPA) two-stage chemical carcinogenesis. In addition, when animals were exposed to ultraviolet light (UV) irradiation, we observed that accumulation of p21WAF1 and cell-cycle arrest were significantly alleviated in the skin of Tg mice as compared to wild-type controls. Most importantly, chronic UV irradiation of Tg mice induced the development of actinic keratosis-like lesions, which are considered in humans as precursors of squamous cell carcinomas (SCC), and subsequently of SCC in a significant proportion of the animals. In contrast, wild-type animals subjected to identical treatments did not develop any type of skin lesions. Thus, the oncoproteins E6 and E7 from beta HPV38 significantly contribute to SCC development in the skin rendering keratinocytes more susceptible to UV-induced carcinogenesis

Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior authors: Paul F. J\"ager, Lena Maier-Hei

Common Limitations of Image Processing Metrics:A Picture Story

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.Comment: This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact [email protected] or [email protected]. Substantial contributions to this document will be acknowledged with a co-authorshi