Performance evaluation of automated white matter hyperintensity segmentation algorithms in a multicenter cohort on cognitive impairment and dementia

Background: White matter hyperintensities (WMH), a biomarker of small vessel disease, are often found in Alzheimer’s disease (AD) and their advanced detection and quantification can be beneficial for research and clinical applications. To investigate WMH in large-scale multicenter studies on cognitive impairment and AD, appropriate automated WMH segmentation algorithms are required. This study aimed to compare the performance of segmentation tools and provide information on their application in multicenter research. Methods: We used a pseudo-randomly selected dataset (n = 50) from the DZNE-multicenter observational Longitudinal Cognitive Impairment and Dementia Study (DELCODE) that included 3D fluid-attenuated inversion recovery (FLAIR) images from participants across the cognitive continuum. Performances of top-rated algorithms for automated WMH segmentation [Brain Intensity Abnormality Classification Algorithm (BIANCA), lesion segmentation toolbox (LST), lesion growth algorithm (LGA), LST lesion prediction algorithm (LPA), pgs, and sysu_media] were compared to manual reference segmentation (RS). Results: Across tools, segmentation performance was moderate for global WMH volume and number of detected lesions. After retraining on a DELCODE subset, the deep learning algorithm sysu_media showed the highest performances with an average Dice’s coefficient of 0.702 (±0.109 SD) for volume and a mean F1-score of 0.642 (±0.109 SD) for the number of lesions. The intra-class correlation was excellent for all algorithms (>0.9) but BIANCA (0.835). Performance improved with high WMH burden and varied across brain regions. Conclusion: To conclude, the deep learning algorithm, when retrained, performed well in the multicenter context. Nevertheless, the performance was close to traditional methods. We provide methodological recommendations for future studies using automated WMH segmentation to quantify and assess WMH along the continuum of cognitive impairment and AD dementia

Example of pain picture and neutral picture.

<p>Example of pain picture and neutral picture.</p

Correlation of personal distress with the slopes of the BOLD response.

<p>The statistical maps are shown superimposed on the averaged T1-weighted dataset of all subjects. Yellow/orange colours signify positive correlations.</p

One sample t-test against zero for the slopes of the BOLD response (pain pictures > neutral pictures) across the whole sample.

<p>The statistical maps are shown superimposed on the averaged T1-weighted dataset of all subjects. Blue/green colours signify negative slopes of the BOLD response significantly different from zero, i.e. neural habituation.</p

Cluster maxima of clusters for which the ratings of trait fantasy correlated with the slopes of the BOLD response across the whole sample (n = 62).

<p>Thresholds are based on cluster-level thresholding with an initial threshold of <i>p</i> < .01 and a cluster threshold of <i>p</i> < .05 (minimum cluster size: 1944 mm³).</p

Cluster maxima of clusters for which the slopes of the BOLD response for the contrast pain > no pain were ≠ 0 (one sample <i>t</i>-test) across the whole sample (n = 62).

<p>All t values were negative, indicating negative slopes and thus habituation. Thresholds are based on cluster-level thresholding with an initial threshold of <i>p</i> < .01 and a cluster threshold of <i>p</i> < .05 (minimum cluster size: 1944 mm³).</p

Correlation of trait fantasy with the slopes of the BOLD response.

<p>The statistical maps are shown superimposed on the averaged T1-weighted dataset of all subjects. Yellow/orange colours signify positive correlations.</p

Stimulation paradigm.

<p>Stimulation paradigm.</p

Means (<i>M</i>), standard deviations (<i>SD</i>) and differences between the groups (<i>t</i>-tests) in age, self-pain ratings, POM ratings and the SPF.

<p><i>Note</i>. <i>POM</i>, <i>Pain of model; SPF</i>, <i>Saarbruecker Persoenlichkeitsfragebogen</i>. Self-pain ratings and POM ratings were each measured on 11-point NRSs (0–10).</p><p>Means (<i>M</i>), standard deviations (<i>SD</i>) and differences between the groups (<i>t</i>-tests) in age, self-pain ratings, POM ratings and the SPF.</p

Two sample t-test between the groups’ slopes of the BOLD response (pain exposure versus touch exposure).

<p>The statistical maps are shown superimposed on the averaged T1-weighted dataset of all subjects. Yellow/orange colours signify larger values of the slopes of the BOLD response, i.e. less neural habituation.</p