Global permutation tests for multivariate ordinal data: alternatives, test statistics, and the null dilemma

We discuss two-sample global permutation tests for sets of multivariate ordinal data in possibly high-dimensional setups, motivated by the analysis of data collected by means of the World Health Organisation's International Classification of Functioning, Disability and Health. The tests do not require any modelling of the multivariate dependence structure. Specifically, we consider testing for marginal inhomogeneity and direction-independent marginal order. Max-T test statistics are known to lead to good power against alternatives with few strong individual effects. We propose test statistics that can be seen as their counterparts for alternatives with many weak individual effects. Permutation tests are valid only if the two multivariate distributions are identical under the null hypothesis. By means of simulations, we examine the practical impact of violations of this exchangeability condition. Our simulations suggest that theoretically invalid permutation tests can still be 'practically valid'. In particular, they suggest that the degree of the permutation procedure's failure may be considered as a function of the difference in group-specific covariance matrices, the proportion between group sizes, the number of variables in the set, the test statistic used, and the number of levels per variable

The use of permutation tests for the analysis of parallel and stepped-wedge cluster randomized trials

We investigate the use of permutation tests for the analysis of parallel and stepped-wedge cluster randomized trials. Permutation tests for parallel designs with exponential family endpoints have been extensively studied. The optimal permutation tests developed for exponential family alternatives require information on intraclass correlation, a quantity not yet defined for time-to-event endpoints. Therefore, it is unclear how efficient permutation tests can be constructed for cluster-randomized trials with such endpoints. We consider a class of test statistics formed by a weighted average of pair-specific treatment effect estimates and offer practical guidance on the choice of weights to improve efficiency. We apply the permutation tests to a cluster-randomized trial evaluating the effect of an intervention to reduce the incidence of hospital-acquired infection. In some settings, outcomes from different clusters may be correlated, and we evaluate the validity and efficiency of permutation test in such settings. Lastly, we propose a permutation test for stepped-wedge designs and compare its performance to mixed effect modeling, and illustrate its superiority when sample sizes are small, the underlying distribution is skewed, or there is correlation across clusters

Global tests of association for multivariate ordinal data

Global tests are in demand whenever it is of interest to draw inferential conclusions about sets of variables as a whole. The present thesis attempts to develop such tests for the case of multivariate ordinal data in possibly high-dimensional set-ups, and has primarily been motivated by research questions that arise from data collected by means of the 'International Classification of Functioning, Disability and Health'. The thesis essentially comprises two parts. In the first part two tests are discussed, each of which addresses one specific problem in the classical two-group scenario. Since both are permutation tests, their validity relies on the condition that, under the null hypothesis, the joint distribution of the variables in the set to be tested is the same in both groups. Extensive simulation studies on the basis of the tests proposed suggest, however, that violations of this condition, from the purely practical viewpoint, do not automatically lead to invalid tests. Rather, two-sample permutation tests' failure appears to depend on numerous parameters, such as the proportion between group sizes, the number of variables in the set of interest and, importantly, the test statistic used. In the second part two further tests are developed which both can be used to test for association, if desired after adjustment for certain covariates, between a set of ordinally scaled covariates and an outcome variable within the range of generalized linear models. The first test rests upon explicit assumptions on the distances between the covariates' categories, and is shown to be a proper generalization of the traditional Cochran-Armitage test to higher dimensions, covariate-adjusted scenarios and generalized linear model-specific outcomes. The second test in turn parametrizes these distances and thus keeps them flexible. Based on the tests' power properties, practical recommendations are provided on when to favour one or the other, and connections with the permutation tests from the first part of the thesis are pointed out. For illustration of the methods developed, data from two studies based on the 'International Classification of Functioning, Disability and Health' are analyzed. The results promise vast potential of the proposed tests in this data context and beyond

Invalid Permutation Tests

Permutation tests are often presented in a rather casual manner, in both introductory and advanced statistics textbooks. The appeal of the cleverness of the procedure seems to replace the need for a rigorous argument that it produces valid hypothesis tests. The consequence of this educational failing has been a widespread belief in a “permutation principle”, which is supposed invariably to give tests that are valid by construction, under an absolute minimum of statistical assumptions. Several lines of argument are presented here to show that the permutation principle itself can be invalid, concentrating on the Fisher-Pitman permutation test for two means. A simple counterfactual example illustrates the general problem, and a slightly more elaborate counterfactual argument is used to explain why the main mathematical proof of the validity of permutation tests is mistaken. Two modifications of the permutation test are suggested to be valid in a very modest simulation. In instances where simulation software is readily available, investigating the validity of a specific permutation test can be done easily, requiring only a minimum understanding of statistical technicalities