Why rankings of biomedical image analysis competitions should be interpreted with care

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future

Reproducibility of hemodynamic simulations of cerebral aneurysms across imaging modalities 3DRA and CTA: Geometric and hemodynamic data

<p>This dataset was created as part of the following study:</p> <p>Geers AJ, Larrabide I, Radaelli AG, Bogunovic H, Kim M, Gratama van Andel HAF, Majoie CB, VanBavel E, Frangi AF. Patient-specific computational hemodynamics of intracranial aneurysms from 3D rotational angiography and CT angiography: An in vivo reproducibility study. American Journal of Neuroradiology, 32(3):581–586, 2011.</p> <p>The goal of the study was to determine the reproducibility of blood flow simulations of cerebral aneurysms. Patients with a total of 10 cerebral aneurysms were imaged with both 3D rotational angiography (3DRA) and computed tomographic angiography (CTA). Each image independently was segmented to obtain a vascular model, the same boundary conditions were imposed, and a CFD simulation was obtained.</p> <p>The dataset contains values for geometric and hemodynamic variables that were derived from the CFD simulations. The variables are defined as follows (TA: time-averaged; PS: peak systole; ED: end diastole):</p> <p>* A_N: Aneurysm neck area<br>* V_A: Aneurysm volume<br>* Q_P: TA flow rate in the parent vessel just proximal to the aneurysm<br>* Q_A: TA flow rate into the aneurysm<br>* NQ_A: Q_A / Q_P<br>* WSS_P: Average TA WSS on the wall of a parent vessel segment just proximal to the aneurysm<br>* WSS_A: Average TA WSS on the aneurysm wall<br>* NWSS_A: WSS_A / WSS_P<br>* LWSS_A: Portion of the aneurysm wall with WSS < 0.4 Pa at ED<br>* MWSS_A: Maximum WSS on the aneurysm wall at PS<br>* 90WSS_A: 90th percentile value of the WSS on the aneurysm wall at PS<br>* N90WSS_A: 90WSS_A normalized by the average WSS on the aneurysm wall at PS</p> <p>See https://github.com/ajgeers/3dracta for an analysis of the data.</p> <p> </p