One-sided t-tests are commonly used in the neuroimaging field, but two-sided
tests should be the default unless a researcher has a strong reason for using a
one-sided test. Here we extend our previous work on cluster false positive
rates, which used one-sided tests, to two-sided tests. Briefly, we found that
parametric methods perform worse for two-sided t-tests, and that non-parametric
methods perform equally well for one-sided and two-sided tests

Eklund, Anders

Knutsson, Hans

Nichols, Thomas E.

Human Brain Mapping

English

arXiv

One‐sided t‐tests are commonly used in the neuroimaging field, but two‐sided tests should be the default unless a researcher has a strong reason for using a one‐sided test. Here we extend our previous work on cluster false positive rates, which used one‐sided tests, to two‐sided tests. Briefly, we found that parametric methods perform worse for two‐sided t‐tests, and that nonparametric methods perform equally well for one‐sided and two‐sided tests

Eklund, A

Knutsson, H

Nichols, TEP

Oxford University Research Archive (ORA)

Reply to Chen et al.: Parametric methods for cluster inference perform worse for two‐sided t‐tests

Nichols, Thomas

Oxford University Research Archive

COMMENTReply to Chen et al.: Parametric methods for cluster inferenceperform worse for two-sided t-testsAnders Eklund1,2,3 | Hans Knutsson1,3 | Thomas E. Nichols4,5,61Division of Medical Informatics, Department of Biomedical Engineering, Linköping University, Linköping, Sweden2Division of Statistics and Machine Learning, Department of Computer and Information Science, Linköping University, Linköping, Sweden3Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden4Big Data Institute, University of Oxford, Oxford, United Kingdom5Wellcome Trust Centre for Integrative Neuroimaging (WIN-FMRIB), University of Oxford, Oxford, United Kingdom6Department of Statistics, University of Warwick, Coventry, United KingdomCorrespondenceAnders Eklund, Division of MedicalInformatics, Department of BiomedicalEngineering, Linköping University, Linköping,Sweden.Email: anders.eklund@liu.seFunding informationNIH, Grant/Award Number: R01 EB015611;Wellcome Trust, Grant/Award Number:100309/Z/12/Z; Knut och Alice WallenbergsStiftelse; Linköping University; SwedishResearch Council, Grant/Award Numbers:2017-04889, 2013-5229; “la Caixa”Foundation; VetenskapsrådetAbstractOne-sided t-tests are commonly used in the neuroimaging field, but two-sided tests should bethe default unless a researcher has a strong reason for using a one-sided test. Here we extendour previous work on cluster false positive rates, which used one-sided tests, to two-sided tests.Briefly, we found that parametric methods perform worse for two-sided t-tests, and that non-parametric methods perform equally well for one-sided and two-sided tests.KEYWORDScluster inference, false positives, fMRI, one-sided, permutation, two-sided1 | INTRODUCTIONChen et al. (2018) discuss an important topic which is often neglectedin the neuroimaging field, the use of one-sided or two-sided tests andthe lack of multiple comparison correction for two one-sided tests. Asmentioned in their article, in our work on massive empirical evaluationof task fMRI inference methods with resting state fMRI (Eklund,Nichols, & Knutsson, 2016) we used one-sided tests (familywise errorrate αFWE = 0.05). We made this choice for two reasons. The first rea-son was simply that for analyses of randomly created groups ofhealthy controls, it should make no difference if one uses a one-sidedor a two-sided test. The second reason was more practical. FSL andSPM both run one-sided tests by default, and we wished to reflectthe typical (if ill-advised) practices of the community. Furthermore, toperform a two-sided permutation test (Winkler, Ridgway, Webster,Smith, & Nichols, 2014), it would be necessary to run two permutationtests per group analysis (which would double the processing time),since normally only the maximum test value over the brain (or thelargest cluster) is saved for every permutation (to form the maximumnull distribution).2 | METHODSTo investigate if performing a two-sided test (as implemented by twotests at αFWE = 0.025) lead to different false positive rates comparedwith a single one-sided test (at αFWE = 0.05), we performed new groupanalyses for a subset of all the parameter settings used in our previouswork (Eklund et al., 2016; Eklund, Knutsson, & Nichols, 2018). Specifi-cally, we only performed two-sample t-tests for the Beijing data(Biswal, Mennes, Zuo, & Milham, 2010), using 40 subjects (i.e., 20 sub-jects per group) and a cluster defining threshold of p = .001. All groupanalyses were performed for 4, 6, 8, and 10 mm FWHM of smoothing.See our recent work (Eklund et al., 2018) for a description of the sixdesigns (B1, B2, E1, E2, E3, and E4) applied to every subject in thefirst level analysis.Received: 6 October 2018 Revised: 19 October 2018 Accepted: 30 October 2018DOI: 10.1002/hbm.24465This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium,provided the original work is properly cited.© 2018 The Authors. Human Brain Mapping published by Wiley Periodicals, Inc.Hum Brain Mapp. 2019;40:1689–1691. wileyonlinelibrary.com/journal/hbm 1689For FSL, group analyses were only performed using FSL OLS, andnot using FLAME1 (which is the default option); FLAME1 leads toconservative results if resting state fMRI data is used, while null taskfMRI analyses (control–control) with FLAME1 gives FWE rates com-parable to FSL OLS (Eklund et al., 2016). For AFNI, we used the newautocorrelation function (ACF) option in 3dClustSim (Cox, Chen, Glen,Reynolds, & Taylor, 2017), which uses a long-tail spatial ACF insteadof a Gaussian one. It should be noted that AFNI provides anotherfunction for cluster thresholding, equitable thresholding and clustering(ETAC) (Cox, 2018), which may perform better than the long-tail ACFfunction used here, but we used the ACF approach to be able to com-pare the two-sided results to our recent work (Eklund et al., 2018).Contrary to Chen et al. (2018), we did not change the cluster definingthreshold to p = .0005 when performing two one-sided tests (forSPM, FSL, or AFNI), as this represents yet another change in the infer-ence configuration that we rather leave fixed to facilitate the compari-son of these results to previous one-sided findings.3 | RESULTSFigure 1 shows estimated familywise error rates for one-sided andtwo-sided tests, where both should exhibit a nominal 5% familywisefalse positive rate. The nonparametric permutation test produces simi-lar results in both cases, while the parametric methods perform worsefor two-sided tests.4 | DISCUSSIONWe have extended our original work on cluster false positive rates(Eklund et al., 2016, 2018) to two-sided tests, showing that parametricmethods perform worse for two-sided tests. RFT p-values depend ona number of approximations:1. Joint normality over the image,2. Sufficient smoothness for lattice images to behave like continuousprocesses,3. Homogeneous smoothness (stationarity), so that the null distribu-tion of cluster size does not vary over space,4. Spatial dependence mostly local, that is, the spatial autocorrela-tion function is proportional to a Gaussian density, and5. Sufficiently high cluster-forming threshold so that the approxi-mate distribution for cluster size is accurate.On this last assumption, the control of FWE depends on the accu-racy of the cluster size distribution in its tail. For example, it is of littleconsequence if the true cluster size FWE p-value is .6 and RFT esti-mates it as .5; in contrast, two-sided inference demands accuracy inthe RFT approximation down to FWE 0.025, and then any inaccura-cies are doubled as both positive and negative excursions are consid-ered. In our findings, it appears that modest inaccuracies in the nullcluster size distribution corresponding to FWE 0.05 (see Figure 1a,and general tendency to overestimate FWE) grow into larger inaccura-cies when the more stringent FWE level 0.025 is used (the inferenceused twice for each result contributing to Figure 1b).In contrast, the nonparametric permutation test for a two-samplet-test is only based on the assumption of exchangeability betweensubjects, and therefore performs equally well for two one-sided testsat αFWE = 0.025.ACKNOWLEDGMENTSThe authors have no conflict of interest to declare. This study wassupported by Swedish Research Council grants 2013-5229 and2017-04889. Funding was also provided by the Center for IndustrialInformation Technology (CENIIT) at Linköping University, and theKnut och Alice Wallenbergs Stiftelse project “Seeing organ function”.B1 B2 E1 E2 E3 E4 B1 B2 E1 E2 E3 E4 B1 B2 E1 E2 E3 E4 B1 B2 E1 E2 E3 E4SPM             FSL OLS             3dttest                 Perm01020304050607080Familywise error rate (%)Beijing, two sample t-test, 40 subjects,CDT p = 0.001, two-sided4 mm6 mm8 mm10 mmExpected95% CIB1 B2 E1 E2 E3 E4 B1 B2 E1 E2 E3 E4 B1 B2 E1 E2 E3 E4 B1 B2 E1 E2 E3 E4SPM             FSL OLS             3dttest                 Perm01020304050607080Familywise error rate (%)Beijing, two sample t-test, 40 subjects,CDT p = 0.001, one-sided4 mm6 mm8 mm10 mmExpected95% CIFIGURE 1 A comparison of empirical familywise error rates for one-sided (left) and two-sided (right) tests, for a cluster defining threshold ofp = .001. Designs B1 and B2 represent two block based activity paradigms, while E1, E2, E3, and E4 represent event related paradigms. DesignE4 is randomized over subjects, while all other designs are the same for all subjects. The parametric methods perform worse for two one-sidedtests at αFWE = 0.025, compared with a single one-sided test at αFWE = 0.05, while the permutation test produces nominal results in both cases[Color figure can be viewed at wileyonlinelibrary.com]1690 EKLUND ET AL.Thomas E. Nichols was supported by the Wellcome Trust Trust(100309/Z/12/Z) and the NIH (R01 EB015611). The Nvidia Corpora-tion, who donated the Nvidia Quadro P6000 graphics card used torun all permutation tests, is also acknowledged. This study would notbe possible without the recent data-sharing initiatives in the neuroim-aging field. We therefore thank the Neuroimaging Informatics Toolsand Resources Clearinghouse and all of the researchers who havecontributed with resting-state data to the 1,000 Functional Connec-tomes Project.ORCIDAnders Eklund https://orcid.org/0000-0001-7061-7995REFERENCESBiswal, B., Mennes, M., Zuo, X., & Milham, M. (2010). Toward discoveryscience of human brain function. Proceedings of the National Academyof Sciences of the United States of America, 107, 4734–4739.Chen, G., Cox, R. W., Glen, D. R., Rajendra, J. K., Reynolds, R. C., &Taylor, P. A. (2018). A tail of two sides: Artificially doubled false posi-tive rates in neuroimaging due to the sidedness choice with t-tests.Human Brain Mapping, 1–7. https://doi.org/10.1002/hbm.24399Cox, R., Chen, G., Glen, D., Reynolds, R., & Taylor, P. (2017). FMRI cluster-ing in AFNI: False-positive rates redux. Brain Connectivity, 7, 152–171.Cox, R. W. (2018). Equitable thresholding and clustering. bioRxiv. https://doi.org/10.1101/295931Eklund, A., Knutsson, H., & Nichols, T. (2018). Cluster failure revisited: Impactof first level design and physiological noise on cluster false positive rates.Human Brain Mapping, 1–16. https://doi.org/10.1002/hbm.24350Eklund, A., Nichols, T., & Knutsson, H. (2016). Cluster failure: Why fMRIinferences for spatial extent have inflated false positive rates. Proceed-ings of the National Academy of Sciences of the United States of America,113, 7900–7905.Winkler, A., Ridgway, G., Webster, M., Smith, S., & Nichols, T. (2014). Permu-tation inference for the general linear model. NeuroImage, 92, 381–397.How to cite this article: Eklund A, Knutsson H, Nichols TE.Reply to Chen et al.: Parametric methods for cluster inferenceperform worse for two-sided t-tests. Hum Brain Mapp. 2019;40:1689–1691. https://doi.org/10.1002/hbm.24465EKLUND ET AL. 1691

ORA - Oxford University Research Archive

Warwick Research Archives Portal Repository

Reply to Chen et al. : parametric methods for cluster inference perform worse for two‐sided t ‐ tests

Reply to Chen et al.: Parametric methods for cluster inference perform
  worse for two-sided t-tests

Reply to Chen et al.: Parametric methods for cluster inference perform worse for two-sided t-tests

Abstract

Similar works

Full text

Available Versions

Oxford University Research Archive (ORA)

Oxford University Research Archive

ORA - Oxford University Research Archive

Warwick Research Archives Portal Repository