Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021

Imaging markers of cerebral small vessel disease provide valuable information on brain health, but their manual assessment is time-consuming and hampered by substantial intra- and interrater variability. Automated rating may benefit biomedical research, as well as clinical assessment, but diagnostic reliability of existing algorithms is unknown. Here, we present the results of the VAscular Lesions DetectiOn and Segmentation (Where is VALDO?) challenge that was run as a satellite event at the international conference on Medical Image Computing and Computer Aided Intervention (MICCAI) 2021. This challenge aimed to promote the development of methods for automated detection and segmentation of small and sparse imaging markers of cerebral small vessel disease, namely enlarged perivascular spaces (EPVS) (Task 1), cerebral microbleeds (Task 2) and lacunes of presumed vascular origin (Task 3) while leveraging weak and noisy labels. Overall, 12 teams participated in the challenge proposing solutions for one or more tasks (4 for Task 1-EPVS, 9 for Task 2-Microbleeds and 6 for Task 3-Lacunes). Multi-cohort data was used in both training and evaluation. Results showed a large variability in performance both across teams and across tasks, with promising results notably for Task 1-EPVS and Task 2-Microbleeds and not practically useful results yet for Task 3-Lacunes. It also highlighted the performance inconsistency across cases that may deter use at an individual level, while still proving useful at a population level

Detecting Perivascular Spaces: a Geodesic Deep Learning Approach

Perivascular spaces (PVS) visible on MRI are currently emerging as an important potential neuroimaging marker for several pathologies in the brain like Alzheimer’s disease and cerebral small vessel disease. PVS are fluid-filled spaces surrounding vessels as they enter the brain. Although PVS are normally not noticeable on MRI scans acquired at clinical field strengths, when these spaces increase in size they become increasingly visible and quantifiable. To study these spaces it is important to have a robust method for quantifying PVS. Manual quantification of PVS is challenging, time-consuming and subject to observer bias due to the difficulty of distinguishing PVS from mimics and the large number of PVS that can occur in MRI scans. Many promising (semi-)automated methods have been proposed recently to decrease annotation time and intra- and inter-observer variability while providing more information about EPVS. However there are still various limitations in the current methods that need to be overcome. An important limitation is that most of the methods are based on elaborate preprocessing steps, feature extraction and heuristic fine-tuning of parameters, making the use of these methods on new datasets cumbersome. Furthermore the majority of the currently proposed methods have been evaluated on small sets of barely 30 images, as most of these methods aim to segment PVS and require voxel-wise annotations for evaluation. In this thesis we propose a method for automated detection of perivascular spaces that combines a convolutional neural network and geodesic distance transform (GDT). We propose to use dot annotations instead of voxel-wise segmentations as this is less time-consuming than fully segmenting PVS while still providing the location of PVS. This enables us to use a considerably larger dataset with ground truth locations than is used in all previously proposed (semi-)automatic methods that provide the location of PVS. We investigated two approaches of using geodesic distance transform to optimize the CNN to detect PVS. The first approach focuses on optimizing the CNN for voxel-wise regression of the geodesic distance map (GDM) computed from the dots and the intensity image. The second approach aims to predict segmentations of the PVS using a CNN that is trained on approximated segmentations obtained by thresholding GDMs. We use 1202 proton density-weighted (PDw) MRI scans to develop our methods and 1000 other scans are used to evaluate the performance of the methods. We show that our methods match human intra-rater performance on detecting PVS without the need for any user interaction. Additionally we show that GDMs are extremely useful for capturing complex morphologies when computed from dot annotations. Our experiments indicate that GDMs can be used to provide valuable additional information to CNNs during training

Training dataset for the VALDO 2021 challenge - Vascular Lesion Detection

VALDO Challenge 2021 Training dataset This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. This dataset corresponds to the training data of the VALDO Challenge 2021 - For more information about the challenge and to participate, please see website In order to register a team, remember to fill in the registration form The challenge is separated in 3 tasks Task 1 - Perivascular spaces segmentation Task 2 - Cerebral micro bleeds segmentation Task 3 - Lacunes segmentation Details of the contents and organisation of the data is described in the README.md file Data reference and acknowledgement When using this data, we kindly ask for the following funding sources to be acknowledged: Wellcome Trust (082464/Z/07/Z), British Heart Foundation (SP/07/001/23603, PG/08/103, PG/12/29/29497 and CS/13/1/30327), Erasmus MC University Medical Center, the Erasmus University Rotterdam, the Netherlands Organization for Scientific Research (NWO) Grant 918-46-615, the Netherlands Organization for Health Research and Development (ZonMW), the Research Institute for Disease in the Elderly (RIDE), and the European Union Seventh Framework Programme (FP7/2007–2013) under grant agreement No. 601055, VPHDARE@IT, the Dutch Technology Foundation STW For all publications, please add references noted below in the reference section When using data from Task 2, please add "for the ALFA Study" as corporate author in your publication and indicate the following list of contributors: Müge Akinci, Annabella Beteta, Raffaele Cacciaglia, Alba Cañas, Irene Cumplido, Carme Deulofeu, Ruth Dominguez, Maria Emilio, Carles Falcón, Karine Fauria, Sherezade Fuentes, Juan Domingo Gispert, Oriol Grau-Rivera, José M. González-de-Echávarri, Laura Hernandez, Gema Huesa, Jordi Huguet, Iva Knezevic, Eider M. Arenaza-Urquijo, Eva M Palacios, Paula Marne, Tania Menchón, Marta Milà-Alomà, Carolina Minguillon, José Luis Molinuevo, Grégory Operto, Albina Polo, Gemma Salvadó, Sandra Pradas, Blanca Rodríguez, Aleix Sala-Vila, Gonzalo Sánchez-Benavides, Mahnaz Shekari, Anna Soteras, Marc Suárez-Calvet, Laura Stankeviciute, Marc Vilanova and Natalia Vilor-Tejedor

Training dataset for the VALDO 2021 challenge - Vascular Lesion Detection

VALDO Challenge 2021 Training dataset This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. This dataset corresponds to the training data of the VALDO Challenge 2021 - For more information about the challenge and to participate, please see website In order to register a team, remember to fill in the registration form The challenge is separated in 3 tasks Task 1 - Perivascular spaces segmentation Task 2 - Cerebral micro bleeds segmentation Task 3 - Lacunes segmentation Details of the contents and organisation of the data is described in the README.md file Data reference and acknowledgement When using this data, we kindly ask for the following funding sources to be acknowledged: Wellcome Trust (082464/Z/07/Z), British Heart Foundation (SP/07/001/23603, PG/08/103, PG/12/29/29497 and CS/13/1/30327), Erasmus MC University Medical Center, the Erasmus University Rotterdam, the Netherlands Organization for Scientific Research (NWO) Grant 918-46-615, the Netherlands Organization for Health Research and Development (ZonMW), the Research Institute for Disease in the Elderly (RIDE), and the European Union Seventh Framework Programme (FP7/2007–2013) under grant agreement No. 601055, VPHDARE@IT, the Dutch Technology Foundation STW For all publications, please add references noted below in the reference section When using data from Task 2, please add "for the ALFA Study" as corporate author in your publication and indicate the following list of contributors: Müge Akinci, Annabella Beteta, Raffaele Cacciaglia, Alba Cañas, Irene Cumplido, Carme Deulofeu, Ruth Dominguez, Maria Emilio, Carles Falcón, Karine Fauria, Sherezade Fuentes, Juan Domingo Gispert, Oriol Grau-Rivera, José M. González-de-Echávarri, Laura Hernandez, Gema Huesa, Jordi Huguet, Iva Knezevic, Eider M. Arenaza-Urquijo, Eva M Palacios, Paula Marne, Tania Menchón, Marta Milà-Alomà, Carolina Minguillon, José Luis Molinuevo, Grégory Operto, Albina Polo, Gemma Salvadó, Sandra Pradas, Blanca Rodríguez, Aleix Sala-Vila, Gonzalo Sánchez-Benavides, Mahnaz Shekari, Anna Soteras, Marc Suárez-Calvet, Laura Stankeviciute, Marc Vilanova and Natalia Vilor-Tejedor

Determinants of Perivascular Spaces in the General Population: A Pooled Cohort Analysis of Individual Participant Data

BACKGROUND AND OBJECTIVES: Perivascular spaces (PVS) are emerging markers of cerebral small vessel disease (CSVD), but research on their determinants has been hampered by conflicting results from small single studies using heterogeneous rating methods. In this study, we therefore aimed to identify determinants of PVS burden in a pooled analysis of multiple cohort studies using 1 harmonized PVS rating method. METHODS: Individuals from 10 population-based cohort studies with adult participants from the Uniform Neuro-Imaging of Virchow-Robin Spaces Enlargement consortium and the UK Biobank were included. On MRI scans, we counted PVS in 4 brain regions (mesencephalon, hippocampus, basal ganglia, and centrum semiovale) according to a uniform and validated rating protocol, both manually and automated using a deep learning algorithm. As potential determinants, we considered demographics, cardiovascular risk factors, APOE genotypes, and other imaging markers of CSVD. Negative binomial regression models were used to examine the association between these determinants and PVS counts. RESULTS: In total, 39,976 individuals were included (age range 20-96 years). The average count of PVS in the 4 regions increased from the age 20 years (0-1 PVS) to 90 years (2-7 PVS). Men had more mesencephalic PVS (OR [95% CI] = 1.13 [1.08-1.18] compared with women), but less hippocampal PVS (0.82 [0.81-0.83]). Higher blood pressure, particularly diastolic pressure, was associated with more PVS in all regions (ORs between 1.04-1.05). Hippocampal PVS showed higher counts with higher high-density lipoprotein cholesterol levels (1.02 [1.01-1.02]), glucose levels (1.02 [1.01-1.03]), and APOE ε4-alleles (1.02 [1.01-1.04]). Furthermore, white matter hyperintensity volume and presence of lacunes were associated with PVS in multiple regions, but most strongly with the basal ganglia (1.13 [1.12-1.14] and 1.10 [1.09-1.12], respectively). DISCUSSION: Various factors are associated with the burden of PVS, in part regionally specific, which points toward a multifactorial origin beyond what can be expected from PVS-related risk factor profiles. This study highlights the power of collaborative efforts in population neuroimaging research

<i>Where is VALDO? </i>:VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021

Imaging markers of cerebral small vessel disease provide valuable information on brain health, but their manual assessment is time-consuming and hampered by substantial intra-and interrater variability. Automated rating may benefit biomedical research, as well as clinical assessment, but diagnostic reliability of existing algorithms is unknown. Here, we present the results of the VAscular Lesions DetectiOn and Segmentation (Where is VALDO?) challenge that was run as a satellite event at the international conference on Medical Image Computing and Computer Aided Intervention (MICCAI) 2021. This challenge aimed to promote the development of methods for automated detection and segmentation of small and sparse imaging markers of cerebral small vessel disease, namely enlarged perivascular spaces (EPVS) (Task 1), cerebral microbleeds (Task 2) and lacunes of presumed vascular origin (Task 3) while leveraging weak and noisy labels. Overall, 12 teams participated in the challenge proposing solutions for one or more tasks (4 for Task 1-EPVS, 9 for Task 2-Microbleeds and 6 for Task 3Lacunes). Multi-cohort data was used in both training and evaluation. Results showed a large variability in performance both across teams and across tasks, with promising results notably for Task 1-EPVS and Task 2-Microbleeds and not practically useful results yet for Task 3-Lacunes. It also highlighted the performance inconsistency across cases that may deter use at an individual level, while still proving useful at a population level

Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021

Imaging markers of cerebral small vessel disease provide valuable information on brain health, but their manual assessment is time-consuming and hampered by substantial intra- and interrater variability. Automated rating may benet biomedical research, as well as clinical assessment, but diagnostic reliability of existing algorithms is unknown. Here, we present the results of the VAscular Lesions DetectiOn and Segmentation (Where is VALDO?) challenge that was run as a satellite event at the international conference on Medical Image Computing and Computer Aided Intervention (MICCAI) 2021. This challenge aimed to promote the development of methods for automated detection and segmentation of small and sparse imaging markers of cerebral small vessel disease, namely enlarged perivascular spaces (EPVS) (Task 1), cerebral microbleeds (Task 2) and lacunes of presumed vascular origin (Task 3) while leveraging weak and noisy labels. Overall, 12 teams participated in the challenge proposing solutions for one or more tasks (4 for Task 1 - EPVS, 9 for Task 2 - Microbleeds and 6 for Task 3 - Lacunes). Multi-cohort data was used in both training and evaluation. Results showed a large variability in performance both across teams and across tasks, with promising results notably for Task 1 - EPVS and Task 2 - Microbleeds and not practically useful results yet for Task 3 - Lacunes. It also highlighted the performance inconsistency across cases that may deter use at an individual level, while still proving useful at a population level. Keywords: CSVD, brain, MRI, microbleeds, enlarged perivascular spaces, lacunes, automated, segmentation, detection, challeng

Why is the winner the best?

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work