Performance evaluation of automated white matter hyperintensity segmentation algorithms in a multicenter cohort on cognitive impairment and dementia

Background: White matter hyperintensities (WMH), a biomarker of small vessel disease, are often found in Alzheimer’s disease (AD) and their advanced detection and quantification can be beneficial for research and clinical applications. To investigate WMH in large-scale multicenter studies on cognitive impairment and AD, appropriate automated WMH segmentation algorithms are required. This study aimed to compare the performance of segmentation tools and provide information on their application in multicenter research. Methods: We used a pseudo-randomly selected dataset (n = 50) from the DZNE-multicenter observational Longitudinal Cognitive Impairment and Dementia Study (DELCODE) that included 3D fluid-attenuated inversion recovery (FLAIR) images from participants across the cognitive continuum. Performances of top-rated algorithms for automated WMH segmentation [Brain Intensity Abnormality Classification Algorithm (BIANCA), lesion segmentation toolbox (LST), lesion growth algorithm (LGA), LST lesion prediction algorithm (LPA), pgs, and sysu_media] were compared to manual reference segmentation (RS). Results: Across tools, segmentation performance was moderate for global WMH volume and number of detected lesions. After retraining on a DELCODE subset, the deep learning algorithm sysu_media showed the highest performances with an average Dice’s coefficient of 0.702 (±0.109 SD) for volume and a mean F1-score of 0.642 (±0.109 SD) for the number of lesions. The intra-class correlation was excellent for all algorithms (>0.9) but BIANCA (0.835). Performance improved with high WMH burden and varied across brain regions. Conclusion: To conclude, the deep learning algorithm, when retrained, performed well in the multicenter context. Nevertheless, the performance was close to traditional methods. We provide methodological recommendations for future studies using automated WMH segmentation to quantify and assess WMH along the continuum of cognitive impairment and AD dementia

ROC curves for the differentiation between reduced and normal DAT availability by the SBR of the caudate (a) and the putamen (b) (minimum over both hemispheres).

<p>ROC curves for the differentiation between reduced and normal DAT availability by the SBR of the caudate (a) and the putamen (b) (minimum over both hemispheres).</p

Example page from the pdf document for visual scoring.

<p>The pdf document comprised one page for each I-123-ioflupane SPECT image showing a 12 mm thick slab (<b>a,</b><b>left</b>) and 4x4 slices of 4 mm thickness (<b>a,</b><b>right</b>). Example I-123-ioflupane SPECTs used as reference images for the visual scoring (<b>b</b>).</p

Results of the semi-quantitative analysis of the phantom studies.

<p>The specific-to-background ratio (SBR) was measured using the same whole striatum ROIs as in the patient studies (<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0108328#pone-0108328-g001" target="_blank">Fig. 1</a>).</p>a<p>measured SBR: mean over left and right hemisphere.</p><p>Results of the semi-quantitative analysis of the phantom studies.</p

Bland-Altman plots comparing the SBR of the caudate (a) and the putamen (b) between CT-based and Chang AC (SBRs of both hemispheres were included independently, i.e. n = 124).

<p>Different scales were chosen for abscissae and ordinates in <b>a</b> and <b>b</b> for display purposes. The horizontal continuous line represents the mean difference, the dashed lines indicate the 95% confidence interval. The given p-value corresponds to the one-sample t-test for zero mean.</p

SPECT underestimates the true activity concentration in small structures such as the striatum and its substructures due to the limited spatial resolution in the reconstructed SPECT image (partial volume effect, PVE).

<p>In order to estimate the extent of underestimation in the present study, the reconstructed spatial resolution was estimated on the basis of a line source measurement using the same acquisition and reconstruction protocol as in the measurements of the striatal phantom and the patients included in the present study (no AC). Spatial resolution was found to be about 8 mm full-width-at-half-maximum (FWHM). Then a high-resolution CT of the striatal phantom was segmented manually (top row; from left to right: transversal, sagittal and coronal slice). Voxel values in the striatum were set to 6.5, voxel values in the background to 1.0 in order to simulate the actual SBR of about 5.5 in the phantom studies (<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0108328#pone-0108328-t001" target="_blank">Table 1</a>). Then the segmented CT image was smoothed with a 3-dimensional Gaussian kernel with 8 mm FWHM to simulate the PVE in SPECT (bottom row). ROI analysis of the smoothed image resulted in a striatal SBR of 3.2 which underestimates the actual SBR of 5.5 by about 42%.</p

Cohen’s unweighted κ (mean ±1 standard deviation) for inter-rater agreement of visual scoring of DAT availability.

<p>Cohen’s unweighted κ (mean ±1 standard deviation) for inter-rater agreement of visual scoring of DAT availability.</p

Custom-made I-123-ioflupane template.

<p>Transversal slice (<b>left</b>). ROIs for left/right caudate and putamen used for hottest voxel analysis, and ROI for the reference region used for intensity scaling, all defined in MNI space (<b>middle</b>). The union of caudate and putamen ROI was used as ROI for the whole striatum. Fusion image (<b>right</b>).</p

Image quality.

<p>Slab of 12 mm thickness of the scaled, stereotactically normalized I-123-ioflupane SPECT averaged over all patients with normal DAT availability (<b>top</b>). Slab displaying the coefficient of variation (%) of the DVR over all patients with normal DAT availability (<b>bottom</b>).</p

Results of the semi-quantitative analysis of the patient studies.

<p>Given are mean values ±1 standard deviation (minimum over both hemispheres for the SBRs, maximum over both hemispheres for the caudate-to-putamen ratio). Subjects were categorized as ‘reduced’ or ‘normal’ DAT availability according to the written report in the patient’s file.</p><p>Results of the semi-quantitative analysis of the patient studies.</p