Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge

Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. The automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their methods on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge. Sixty T1 + FLAIR images from three MR scanners were released with the manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. The segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: 1) Dice similarity coefficient; 2) modified Hausdorff distance (95th percentile); 3) absolute log-transformed volume difference; 4) sensitivity for detecting individual lesions; and 5) F1-score for individual lesions. In addition, the methods were ranked on their inter-scanner robustness; 20 participants submitted their methods for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all the methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation

Validation of White Matter Hyperintensities automatic segmentation methods

Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona, Any: 2020, Tutor: Eloi Puertas i Prats i Joaquim Raduà[en] This master’s thesis seeks to review and objectively evaluate the current white matter hyperintensities (WMH) automatic segmentation methods published journals. To this end, the methods have been systematically searched in scientific databases, and those meeting inclusion criteria have been evaluated. The evaluation has consisted in applying the method to detect WMH in our dataset of patients with bipolar disorder and healthy controls, in which an experienced neuroradiologist had manually coded all WMH. After the systematic search, we selected all available methods that were ready for use with standard MRI data by a standard user. Four methods met these criteria. We then applied these methods to detect WMH in our dataset, and compared the results with the neuroradiologist-based ground truth deriving several evaluation metrics. This master’s thesis also include a discussion section, in which we compare the results of our evaluations with the results of the WMH Segmentation Challenge held in 2017, which included substantially different datasets. The most relevant conclusion of this master’s thesis is that no method seems to be accurate enough for clinical implementation, although the low performance of the methods may be related to the differences between our data and the data that were used to train them. Besides, realizing the huge improvement made in the field during the last few years after the appearance of deep neural networks, we anticipate that a method with sufficient accuracy might be available soon. The codes used to obtain the results and graphs displayed in this project together with some guidelines to run them are available through PFM-WMH 1

MRI white matter lesion segmentation using an ensemble of neural networks and overcomplete patch-based voting

[EN] Accurate quantification of white matter hyperintensities (WMH) from Magnetic Resonance Imaging (MRI) is a valuable tool for the analysis of normal brain ageing or neurodegeneration. Reliable automatic extraction of WMH lesions is challenging due to their heterogeneous spatial occurrence, their small size and their diffuse nature. In this paper, we present an automatic method to segment these lesions based on an ensemble of overcomplete patch-based neural networks. The proposed method successfully provides accurate and regular segmentations due to its overcomplete nature while minimizing the segmentation error by using a boosted ensemble of neural networks. The proposed method compared favourably to state of the art techniques using two different neurodegenerative datasets. (C) 2018 Elsevier Ltd. All rights reserved.This research has been done thanks to the Australian distinguished visiting professor grant from the CSIRO (Commonwealth Scientific and Industrial Research Organisation) and the Spanish "Programa de apoyo a la investigacion y desarrollo (PAID-00-15)" of the Universidad Politecnica de Valencia. This research was partially supported by the Spanish grant TIN2013-43457-R from the Ministerio de Economia y competitividad. This study has been carried out also with support from the French State, managed by the French National Research Ageny in the frame of the Investments for the future Program IdEx Bordeaux (ANR-10-IDEX-03-02, HL-MRI Project), Cluster of excellence CPU and TRAIL (HR-DTI ANR-10-LABX-57) and the CNRS multidisciplinary project Defi imag'In. Some of the data used in this work was collected by the AIBL study group. Funding for the AIBL study is provided by the CSIRO Flagship Collaboration Fund and the Science and Industry Endowment Fund (SIEF) in partnership with Edith Cowan University (ECU), Mental Health Research Institute (MHRI), Alzheimer's Australia (AA), National Ageing Research Institute (NARI), Austin Health, Macquarie University, CogState Ltd, Hollywood Private Hospital, and Sir Charles Gairdner Hospital.Manjón Herrera, JV.; Coupe, P.; Raniga, P.; Xia, Y.; Desmond, P.; Fripp, J.; Salvado, O. (2018). MRI white matter lesion segmentation using an ensemble of neural networks and overcomplete patch-based voting. Computerized Medical Imaging and Graphics. 69:43-51. https://doi.org/10.1016/j.compmedimag.2018.05.001S43516

Quality control for more reliable integration of deep learning-based image segmentation into medical workflows

Machine learning algorithms underpin modern diagnostic-aiding software, whichhas proved valuable in clinical practice, particularly in radiology. However,inaccuracies, mainly due to the limited availability of clinical samples fortraining these algorithms, hamper their wider applicability, acceptance, andrecognition amongst clinicians. We present an analysis of state-of-the-artautomatic quality control (QC) approaches that can be implemented within thesealgorithms to estimate the certainty of their outputs. We validated the mostpromising approaches on a brain image segmentation task identifying whitematter hyperintensities (WMH) in magnetic resonance imaging data. WMH are acorrelate of small vessel disease common in mid-to-late adulthood and areparticularly challenging to segment due to their varied size, anddistributional patterns. Our results show that the aggregation of uncertaintyand Dice prediction were most effective in failure detection for this task.Both methods independently improved mean Dice from 0.82 to 0.84. Our workreveals how QC methods can help to detect failed segmentation cases andtherefore make automatic segmentation more reliable and suitable for clinicalpractice.<br

Assessment of White Matter Hyperintensity, Cerebral Blood Flow, and Cerebral Oxygenation in Older Subjects Stratified by Cerebrovascular Risk

Objective: Cerebrovascular disease (CVD) is the fifth most common cause of mortality in the United States. Diagnosis of CVD at an early stage is critical for optimal intervention designed to prevent ongoing and future brain injury. CVD is commonly associated with abnormalities of the cerebral microvasculature leading to tissue dysfunction, neuronal injury and death, and resultant clinical symptoms, which in turn, further impacts cerebral autoregulation (CA). This series of studies aims to test the hypothesis that white matter hyperintensities (WMH) and cerebral hemodynamics (quantified by magnetic resonance imaging (MRI) and an by innovative hybrid near-infrared diffuse optical instrument) can be used as biomarkers to distinguish cognitively healthy older subjects with high or low risk for developing CVD. Methods: Using functional MRI, WMH and cerebral blood flow (CBF) were quantified in 26 cognitively healthy older subjects (age: 77.8 ± 6.8 years). In a follow-up study, significant variability in WMH quantification methodology was addressed, with sources of variability identified in selecting image center of gravity, software compatibility, thresholding techniques, and manual editing procedures. Accordingly, post-acquisition processing methods were optimized to develop a standardized protocol with less than 0.5% inter-rater variance. Using a novel laboratory-made hybrid near-infrared spectroscopy/diffuse correlation spectroscopy (NIRS/DCS) and a finger plethysmograph, low-frequency oscillations (LFOs) of CBF, cerebral oxygenation, and main arterial pressure (MAP) were simultaneously measured before, during, and after 70° head-up-tilting (HUT). Gains (associated with CAs) to magnify LFOs were determined by transfer function analyses with MAP as the input and cerebral hemodynamic parameters as the outputs. In a follow-up study, a fast software correlator for DCS and a parallel detection technique for NIRS/DCS were adapted to improve the sampling rate of hybrid optical measurements. In addition, a new DCS probe was developed to measure CBF at the occipital lobe, which represents a novel application of the NIRS/DCS technique. Results: MRI measurements demonstrate that deep WMH (dWMH) and periventricular WMH (pWMH) volumetric measures are associated with reduced regional cortical CBF in patients at high-risk of CVD. Moreover, CBF in white matter (WM) was reduced in regions demonstrating both pWMH and dWMHs. NIRS/DCS optical measurements demonstrate that at resting baseline, LFO gains in the high-risk group were relatively lower compared to the low-risk group. The lower baseline gains in the high-risk group may be attributed to compensatory mechanisms that allow the maintenance of a stronger steady-state CA. However, HUT resulted in smaller gain reductions in the high-risk group compared to the low-risk group, suggesting weaker dynamic CA in association with increased CVD risks. A noteworthy finding in these experiments was that CVD risk more strongly influenced CBF than cerebral oxygenation. Conclusions: Regional WMH volumes, cortical and WM CBF values, and LFO gains of cerebral hemodynamics demonstrate specific associations with CA and may serve as important potential biomarkers for early diagnosis of CVD. The high spatial resolution, large penetration depth, and variety of imaging-sequences afforded by MRI make it an appealing imaging modality for evaluation of CVD, although MRI is costly, time-limited, and requires transfer of subjects from bed to imaging facility. In contrast, low-cost, portable, mobile diffuse optical technologies provide a complementary alternative for early screening of CVD, that can further allow continuous monitoring of disease attenuation or progression at the subject’s bedside. Thus, development of both methodologies is essential for progress in our future understanding of CVD as a major contributor to the morbidity and mortality associated with CVD today

Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities; Results of the WMH Segmentation Challenge

Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge (https://wmh.isi.uu.nl/). Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. Segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice similarity coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute log-transformed volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness. Twenty participants submitted their method for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation