17 research outputs found

    Variability in the analysis of a single neuroimaging dataset by many teams

    Get PDF
    Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed

    Variability in the analysis of a single neuroimaging dataset by many teams

    Get PDF
    Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed

    Integration of multimodal MRI data via PCA to explain language performance

    Get PDF
    Objective/methods: Neuroimaging research has predominantly focused on exploring how cortical or subcortical brain abnormalities are related to language dysfunction in patients with neurological disease through the use of single modality imaging. Still, limited knowledge exists on how various MRI measures relate to each other and to patients' language performance. In this study, we explored the relationship between measures of regional cortical thickness, gray–white matter contrast (GWMC), white matter diffusivity [mean diffusivity (MD) and fractional anisotropy (FA)] and the relative contributions of these MRI measures to predicting language function across patients with temporal lobe epilepsy (TLE) and healthy controls. T1- and diffusion-weighted MRI data were collected from 56 healthy controls and 52 patients with TLE. By focusing on frontotemporal regions implicated in language function, we reduced each domain of MRI data to its principal component (PC) and quantified the correlations among these PCs and the ability of these PCs to explain the variation in vocabulary, naming and fluency. We followed up our significant findings by assessing the predictive power of the implicated PCs with respect to language impairment in our sample. Results: We found significant positive associations between PCs representing cortical thickness, GWMC and FA that appeared to be partially mediated by changes in total brain volume. We also found a significant association between reduced FA and increased MD after controlling for confounding factors (e.g., age, field strength, total brain volume). Reduced FA was significantly associated with reductions in visual naming while increased MD was associated with reductions in auditory naming scores, even after controlling for the variability explained by reductions in hippocampal volumes. Inclusion of FA and MD PCs in predictive models of language impairment resulted in significant improvements in sensitivity and specificity of the predictions. Conclusions: Quantitative MRI measures from T1 and diffusion-weighted scans are unlikely to represent perfectly orthogonal vectors of disease in individuals with epilepsy. On the contrary, they exhibit highly intercorrelated PCs in their factor structures, which is consistent with an underlying pathological process that affects both the cortical and the subcortical structures simultaneously. In addition to hippocampal volume, the PCs of diffusion weighted measures (FA and MD) increase the sensitivity and specificity for determining naming impairment in patients with TLE. These findings underline the importance of combining multimodal imaging measures to better predict language performance in TLE that could extend to other patients with prominent language impairments
    corecore