To test the validity of statistical methods for fMRI data analysis, Eklund et al. (1) used, for the first time, large-scale experimental data rather than simulated data. Using resting-state fMRI measurements to represent a null hypothesis of no task-induced activation, the authors compare familywise error rates for voxel-based and cluster-based inferences for both parametric and nonparametric methods. Eklund et al.’s study used three fMRI statistical analysis packages. They found that, for a target familywise error rate of 5%, the parametric methods gave invalid cluster-based inferences and conservative voxel-based inferences

Behrmann, Marlene

Brown, Emery Neal

Proceedings of the National Academy of Sciences

DSpace@MIT

LETTERControversy in statistical analysis of functionalmagnetic resonance imaging dataEmery N. Browna,b,1 and Marlene BehrmanncTo test the validity of statistical methods for fMRI dataanalysis, Eklund et al. (1) used, for the first time, large-scale experimental data rather than simulated data.Using resting-state fMRI measurements to represent anull hypothesis of no task-induced activation, the au-thors compare familywise error rates for voxel-basedand cluster-based inferences for both parametric andnonparametric methods. Eklund et al.’s study usedthree fMRI statistical analysis packages. They foundthat, for a target familywise error rate of 5%, the para-metric methods gave invalid cluster-based inferencesand conservative voxel-based inferences.Eklund et al. (1) attribute the invalid cluster-basedinferences to the incorrect assumption of squared ex-ponential structure in the spatial autocorrelation func-tion of the parametric models. The authors suggestnonparametric methods as a more appropriate wayto achieve targeted error rates, and conclude that sta-tistical methods for fMRI data analysis should be vali-dated. In addition, Eklund et al. state that their findings“question the validity of some 40,000 fMRI studies andmay have a large impact on the interpretation of neuro-imaging results” (1). This sentence from the Signifi-cance section of the original paper was picked up bythe press and yielded the alarming negative headlinethat fMRI analyses produce incorrect results because ofa bug in a widely used data analysis package (2–4).Eklund et al. revised their extrapolation regarding theimplication of their findings in a correction to their arti-cle (5) and report that their analysis might apply to3,500 rather than 40,000 fMRI studies (6). However,before this revision was published, the original state-ments created considerable debate about data analysisand the accuracy of fMRI findings (2–4).The overstatements of the original paper and thesubsequent media attention cast doubt on fMRI as atechnique for studying brain function, and possiblyeven caused damage to the field of cognitive neuro-science (2–4). In PNAS, Cox et al. (7) and Kessler et al. (8)offer clarifications about the original paper and its re-vision. Eklund et al. have added their rejoinder (9). Sev-eral scientific points have now been mostly resolved.The remaining question is: What else can be learnedfrom this controversy?fMRI is a highly valued methodology for under-standing brain function and its relationship to behavior.During the last 25 y, significant scientific advances havebeen made using this technique. To ensure continuedprogress, fMRI experimentalists want to be assured thatthe instruments, experimental protocols, and dataanalysis paradigms have been vetted by experts andwork correctly. At the same time, experimentalists mustbe well informed about the fMRI process, and have asolid understanding of how to apply and interpret com-monly used statistical methods (10–12). The ease ofanalysis afforded by some of the software programsbelies the complexity of the methods. This ease of usedoes not release experimentalists from their responsibil-ity to validate findings using established statistical prin-ciples (12, 13). Judicious use of nonparametric methodscan, as Eklund et al. (1) suggest, improve the currentanalysis paradigm in certain cases. However, applicationof nonparametric methods cannot be the universal so-lution, nor did Eklund et al. suggest that it could be.The current discussion shows that the validity of fMRIdata analysis paradigms has not been uniformly estab-lished and needs continued in-depth investigation.fMRI is a complex process that involves biophysics,neuroananatomy, neurophysiology, and statistics (ex-perimental design, statistical modeling, and data anal-ysis). fMRI data have a low signal-to-noise ratio (14, 15).As a consequence, all of the biophysics, neurophysiol-ogy, and neuroanatomy that underlie fMRI should beused to design experiments, formulate statistical mod-els, and analyze the data to increase the signal-to-noiseratio and information extraction. Achieving more ac-curate fMRI data analyses is a challenging interdisciplinarytask that requires concerted collaborations among phys-icists, statisticians, and neuroscientists who, together, canquestion the current approaches more deeply and con-struct more accurate analysis methods.In an ideal fMRI statistical analysis, the relationshipsamong the voxels would take account of the spatial andtemporal properties of the experiment and the scanneraDepartment of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114;bDepartment of Brain and Cognitive Sciences, Institute for Medical Engineering and Science and Picower Institute for Learning and Memory,Massachusetts Institute of Technology, Cambridge, MA 02139; and cDepartment of Psychology and Center for the Neural Basis of Cognition,Carnegie Mellon University, Pittsburgh, PA 15213Author contributions: E.N.B. and M.B. wrote the paper.The authors declare no conflict of interest.1To whom correspondence should be addressed. Email: enb@neurostat.mit.edu.E3368–E3369 | PNAS | April 25, 2017 | vol. 114 | no. 17 www.pnas.org/cgi/doi/10.1073/pnas.1705513114LETTERthermal noise (16). The experiment’s spatial and temporal proper-ties are dictated by the physiological changes (neural activity, bloodflow, and blood oxygenation levels) induced by the particular be-havioral task and background physiological activity and anatomy(white matter, gray matter, the ventricles, and blood vessels) ofthe relevant brain regions. The ideal fMRI acquisition schemewouldbe accompanied by a characterization of these spatial and temporalprocesses so that the subsequent data analysis can correctly takethem into account (16). Improving fMRI statistical methods mustcombine research to decipher the meaning/origins of the bloodoxygen level-dependent signal with characterizations of the spatio-temporal properties of task-related activity, background physiolog-ical activity, and scanner properties. Sharing data and methodswould greatly expedite validation (9).BRAIN 2025, the report of the NIH Brain Initiative, recommendsfostering interdisciplinary collaborations among neuroscientists,physicists, engineers, statisticians, and mathematicians to properlycollect, analyze, and interpret the data that result from the devel-opment of new neuroscience tools (https://www.braininitiative.nih.gov/2025/). The current exchange identifies fMRI as an existing toolthat is perfect for pursuing such a collaboration. A possible goalcould be to increase fMRI signal-to-noise ratios so that the tech-nique can be used reliably to make inferences about an individualsubject in a given paradigm.Developing statistical methods based on detailed modeling ofthe fMRI process opens the door to using more direct, informativeinference paradigms based on estimated effect sizes, confidenceintervals, and Bayesian posterior assessments rather than moreindirect approaches based on significance tests and P values. Link-ing statistical methodology development and fundamental fMRIresearch is crucial for developing more accurate analysis methods,attributing accurate scientific interpretations to results, and ensur-ing the reliability and reproducibility of fMRI studies. These pointshave been made before. However, their significance has perhapsnot been considered to the extent required.AcknowledgmentsWe thank Bruce Rosen, Michael Tarr, and Larry Wald for helpful comments.1 Eklund A, Nichols TE, Knutsson H (2016) Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proc Natl Acad Sci USA 113:7900–7905. Erratum in Proc Natl Acad Sci USA 113:E4929.2 Crew B (2016) A bug in fMRI software could invalidate 15 years of brain research. Available at www.sciencealert.com/a-bug-in-fmri-software-could-invalidate-decades-of-brain-research-scientists-discover. Accessed April 10, 2017.3 Murphy K (2016) Do you believe in god or is that a software glitch? Available at https://www.nytimes.com/2016/08/28/opinion/sunday/do-you-believe-in-god-or-is-that-a-software-glitch.html?_r=0. Accessed April 10, 2017.4 Biello D (2016) Much of what we know about the brain may be wrong: The problem with fMRI. Available at ideas.ted.com/much-of-what-we-know-about-the-brain-may-be-wrong-the-problem-with-fmri/. Accessed April 10, 2017.5 Eklund A, Nichols TE, Knutsson H (2016) Correction for Eklund et al. Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. ProcNatl Acad Sci USA 113:E4929.6 Nichols T (2016) Bibliometrics of cluster inference. Available at blogs.warwick.ac.uk/nichols/entry/bibliometrics_of_cluster/. Accessed April 10, 2017.7 Cox RW, Chen G, Glen DR, Reynolds RC, Taylor PA (2017) FMRI clustering and false positive rates. Proc Natl Acad Sci USA 114:E3370–E3371.8 Kessler D, Angstadt M, Sripada C (2017) Reevaluating “cluster failure” using nonparametric control of false discovery rate. Proc Natl Acad Sci USA114:E3372–E3373.9 Eklund A, Nichols TE, Knutsson H (2017) Reply to Cox et al. and Kessler et al.: Data and code sharing is the way forward for fMRI. Proc Natl Acad Sci USA114:E3374–E3375.10 Poldrack RA, et al. (2017) Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nat Rev Neurosci 18:115–126.11 Munafo MR, et al. (2017) A manifesto for reproducible science. Nature Human Behaviour 1:0021.12 Kass RE, et al. (2016) Ten simple rules for effective statistical practice. PLOS Comput Biol 12:e1004961.13 Kass RE, Ventura V, Brown EN (2005) Statistical issues in the analysis of neuronal data. J Neurophysiol 94:8–25.14 Parrish TB, Gitelman DR, LaBar KS, Mesulam MM (2000) Impact of signal-to-noise on functional MRI. Magn Reson Med 44:925–932.15 Welvaert M, Rosseel Y (2013) On the definition of signal-to-noise ratio and contrast-to-noise ratio for FMRI data. PLoS One 8:e77089.16 Wald LL, Rosseel Y (2017) Impacting the effect of fMRI noise through hardware and acquisition choices-impliations for controlling false positive rates.Neuroimage,10.1016/j.neuroimage.2016.12.057.Brown and Behrmann PNAS | April 25, 2017 | vol. 114 | no. 17 | E3369

Controversy in statistical analysis of functional magnetic resonance imaging data

Emery N. Brown

Marlene Behrmann

Crossref

http://dspace.mit.edu/bitstream/1721.1/112127/1/PNAS-2017-Brown-E3368-9.pdf

Controversy in statistical analysis of functional magnetic resonance imaging data

Abstract

Similar works

Full text

Available Versions

DSpace@MIT

Crossref