Variability in the analysis of a single neuroimaging dataset by many teams

Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed

Variability in the analysis of a single neuroimaging dataset by many teams

Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed

Composition, crystallization conditions and genesis of sulfide-saturated parental melts of olivine-phyric rocks from Kamchatsky Mys (Kamchatka, Russia)

Highlights • Parental melts of sulfide-bearing KM rocks have near primary MORB-like composition. • Crystallization of these S-saturated melts occurred in near-surface conditions. • Extensive fractionation and crustal assimilation are not the causes of S-saturation. • S content in melts can be restored by accounting for daughter sulfide globules. Abstract Sulfide liquids that immiscibly separate from silicate melts in different magmatic processes accumulate chalcophile metals and may represent important sources of the metals in Earth's crust for the formation of ore deposits. Sulfide phases commonly found in some primitive mid-ocean ridge basalts (MORB) may support the occurrence of sulfide immiscibility in the crust without requiring magma contamination and/or extensive fractionation. However, the records of incipient sulfide melts in equilibrium with primitive high-Mg olivine and Cr-spinel are scarce. Sulfide globules in olivine phenocrysts in picritic rocks of MORB-affinity at Kamchatsky Mys (Eastern Kamchatka, Russia) represent a well-documented example of natural immiscibility in primitive oceanic magmas. Our study examines the conditions of silicate-sulfide immiscibility in these magmas by reporting high precision data on the compositions of Cr-spinel and silicate melt inclusions, hosted in Mg-rich olivine (86.9–90 mol% Fo), which also contain globules of magmatic sulfide melt. Major and trace element contents of reconstructed parental silicate melts, redox conditions (ΔQFM = +0.1 ± 0.16 (1σ) log. units) and crystallization temperature (1200–1285 °C), as well as mantle potential temperatures (~1350 °C), correspond to typical MORB values. We show that nearly 50% of sulfur could be captured in daughter sulfide globules even in reheated melt inclusions, which could lead to a significant underestimation of sulfur content in reconstructed silicate melts. The saturation of these melts in sulfur appears to be unrelated to the effects of melt crystallization and crustal assimilation, so we discuss the reasons for the S variations in reconstructed melts and the influence of pressure and other parameters on the SCSS (Sulfur Content at Sulfide Saturation)

Variability in the analysis of a single neuroimaging dataset by many teams

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.Depto. de Psicobiología y Metodología en Ciencias del ComportamientoFac. de PsicologíaTRUEpu