Reproducibility of hippocampal atrophy rates measured with manual, FreeSurfer, AdaBoost, FSL/FIRST and the MAPS-HBSI methods in Alzheimer's disease

The purpose of this study is to assess the reproducibility of hippocampal atrophy rate measurements of commonly used fully-automated algorithms in Alzheimer disease (AD). The reproducibility of hippocampal atrophy rate for FSL/FIRST, AdaBoost, FreeSurfer, MAPS independently and MAPS combined with the boundary shift integral (MAPS-HBSI) were calculated. Back-to-back (BTB) 3D T1-weighted MPRAGE MRI from the Alzheimer's Disease Neuroimaging Initiative (ADNI1) study at baseline and year one were used. Analysis on 3 groups of subjects was performed – 562 subjects at 1.5 T, a 75 subject group that also had manual segmentation and 111 subjects at 3 T. A simple and novel statistical test based on the binomial distribution was used that handled outlying data points robustly. Median hippocampal atrophy rates were −1.1%/year for healthy controls, −3.0%/year for mildly cognitively impaired and −5.1%/year for AD subjects. The best reproducibility was observed for MAPS-HBSI (1.3%), while the other methods tested had reproducibilities at least 50% higher at 1.5 T and 3 T which was statistically significant. For a clinical trial, MAPS-HBSI should require less than half the subjects of the other methods tested. All methods had good accuracy versus manual segmentation. The MAPS-HBSI method has substantially better reproducibility than the other methods considered

Performance of five research-domain automated WM lesion segmentation methods in a multi-center MS study

Background and Purpose: In vivoidentification of white matter lesions plays a key-role in evaluation of patients with multiple sclerosis (MS). Automated lesion segmentation methods have been developed to substitute manual outlining, but evidence of their performance in multi-center investigations is lacking. In this work, five research-domain automated segmentation methods were evaluated using a multi-center MS dataset. / Methods: 70 MS patients (median EDSS of 2.0 [range 0.0–6.5]) were included from a six-center dataset of the MAGNIMS Study Group (www.magnims.eu) which included 2D FLAIR and 3D T1 images with manual lesion segmentation as a reference. Automated lesion segmentations were produced using five algorithms: Cascade; Lesion Segmentation Toolbox (LST) with both the Lesion growth algorithm (LGA) and the Lesion prediction algorithm (LPA); Lesion-Topology preserving Anatomical Segmentation (Lesion-TOADS); and k-Nearest Neighbor with Tissue Type Priors (kNN-TTP). Main software parameters were optimized using a training set (N = 18), and formal testing was performed on the remaining patients (N = 52). To evaluate volumetric agreement with the reference segmentations, intraclass correlation coefficient (ICC) as well as mean difference in lesion volumes between the automated and reference segmentations were calculated. The Similarity Index (SI), False Positive (FP) volumes and False Negative (FN) volumes were used to examine spatial agreement. All analyses were repeated using a leave-one-center-out design to exclude the center of interest from the training phase to evaluate the performance of the method on ‘unseen’ center. / Results: Compared to the reference mean lesion volume (4.85 ± 7.29 mL), the methods displayed a mean difference of 1.60 ± 4.83 (Cascade), 2.31 ± 7.66 (LGA), 0.44 ± 4.68 (LPA), 1.76 ± 4.17 (Lesion-TOADS) and −1.39 ± 4.10 mL (kNN-TTP). The ICCs were 0.755, 0.713, 0.851, 0.806 and 0.723, respectively. Spatial agreement with reference segmentations was higher for LPA (SI = 0.37 ± 0.23), Lesion-TOADS (SI = 0.35 ± 0.18) and kNN-TTP (SI = 0.44 ± 0.14) than for Cascade (SI = 0.26 ± 0.17) or LGA (SI = 0.31 ± 0.23). All methods showed highly similar results when used on data from a center not used in software parameter optimization. / Conclusion: The performance of the methods in this multi-center MS dataset was moderate, but appeared to be robust even with new datasets from centers not included in training the automated methods

Can measuring hippocampal atrophy with a fully automatic method be substantially less noisy than manual segmentation over both 1 and 3 years?

To quantify the "segmentation noise" of several widely used fully automatic methods for measuring longitudinal hippocampal atrophy in Alzheimer's disease and compare the results to the segmentation noise of manual segmentation over both 1 and 3 years. The segmentation noise of 5 longitudinal hippocampal atrophy measurement methods was quantified, including checking its Gaussianity, using 264 subjects from the ADNI1 back-to-back (BTB) data set over both 1 year and 3 year intervals. The segmentation methods were FreeSurfer 5.3.0 both cross sectional and longitudinal, FreeSurfer 6.0.0 longitudinal, MAPS-HBSI and FSL/FIRST 5.0.8. The BTB manual segmentation of 75 ADNI subjects from a previous study provided the manual distributions for comparison. All methods, including the manual segmentation, violated the Gaussianity assumption. Two methods, FreeSurfer 6.0.0 and MAPS-HBSI, had a segmentation noise substantially less than a surrogate for manual segmentation. FreeSurfer 5.3.0 longitudinal was confirmed as a surrogate for manual segmentation. The violation of the Gaussian assumption by the segmentation methods assessed, including manual, suggests results of previous studies that assumed Gaussian statistics without confirmation may need review. Fully automatic FreeSurfer 6.0.0 and MAPS-HBSI both have lower segmentation noise than manual requiring less than two thirds of the subjects to detect the same treatment effect

Reproducibility of hippocampal atrophy rates measured with manual, FreeSurfer, AdaBoost, FSL/FIRST and the MAPS-HBSI methods in Alzheimer's disease

The purpose of this study is to assess the reproducibility of hippocampal atrophy rate measurements of commonly used fully-automated algorithms in Alzheimer disease (AD). The reproducibility of hippocampal atrophy rate for FSL/FIRST, AdaBoost, FreeSurfer, MAPS independently and MAPS combined with the boundary shift integral (MAPS-HBSI) were calculated. Back-to-back (BTB) 3D T1-weighted MPRAGE MRI from the Alzheimer's Disease Neuroimaging Initiative (ADNI1) study at baseline and year one were used. Analysis on 3 groups of subjects was performed - 562 subjects at 1.5T, a 75 subject group that also had manual segmentation and 111 subjects at 3T. A simple and novel statistical test based on the binomial distribution was used that handled outlying data points robustly. Median hippocampal atrophy rates were -1.1%/year for healthy controls, -3.0%/year for mildly cognitively impaired and -5.1%/year for AD subjects. The best reproducibility was observed for MAPS-HBSI (1.3%), while the other methods tested had reproducibilities at least 50% higher at 1.5T and 3T which was statistically significant. For a clinical trial, MAPS-HBSI should require less than half the subjects of the other methods tested. All methods had good accuracy versus manual segmentation. The MAPS-HBSI method has substantially better reproducibility than the other methods considered