At what sample size do correlations stabilize?

Sample correlations converge to the population value with increasing sample size, but the estimates are often inaccurate in small samples. In this report we use Monte-Carlo simulations to determine the critical sample size from which on the magnitude of a correlation can be expected to be stable. The necessary sample size to achieve stable estimates for correlations depends on the effect size, the width of the corridor of stability (i.e., a corridor around the true value where deviations are tolerated), and the requested confidence that the trajectory does not leave this corridor any more. Results indicate that in typical scenarios the sample size should approach 250 for stable estimates

Testing for negligible interaction: A coherent and robust approach.

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap-based intersection-union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a nonsignificant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per-contrast power of the proposed approach to the noncentrality-based negligible interaction test of Cheng and Shao (2007). For 2 x 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied, however only our proposed bootstrapping approach was satisfactory under nonnormality and/or variance heterogeneity. In general a x b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection-union approach.Social Sciences and Humanities Research Council of Canad

Robust statistical methods: a primer for clinical psychology and experimental psychopathology researchers

This paper reviews and offers tutorials on robust statistical methods relevant to clinical and experimental psychopathology researchers. We review the assumptions of one of the most commonly applied models in this journal (the general linear model, GLM) and the effects of violating them. We then present evidence that psychological data are more likely than not to violate these assumptions. Next, we overview some methods for correcting for violations of model assumptions. The final part of the paper presents 8 tutorials of robust statistical methods using R that cover a range of variants of the GLM (t-tests, ANOVA, multiple regression, multilevel models, latent growth models). We conclude with recommendations that set the expectations for what methods researchers submitting to the journal should apply and what they should report

Speed Effect Analysis Using the CFA Framework

The paper outlines a method for investigating the speed effect due to a time limit in testing. It is assumed that the time limit enables latent processing speed to influence responses by causing omissions in the case of insufficient speed. Because of processing speed as additional latent source, the customary confirmatory factor model is enlarged by a second latent variable representing latent processing speed. For distinguishing this effect from other method effects, the factor loadings are fixed according to the cumulative normal distribution. With the second latent variable added, confirmatory factor analysis of reasoning data (N=518) including omissions because of a time limit yielded good model fit and discriminated the speed effect from other possible effects due to the item difficulty, the homogeneity of an item subset and the item positions. Because of the crucial role of the cumulative normal distribution for fixing the factor loadings a check of the normality assumption is also reported

The effect of skewness and kurtosis on the robustness of linear mixed models

This study analyzes the robustness of the linear mixed model (LMM) with the Kenward–Roger (KR) procedure to violations of normality and sphericity when used in split-plot designs with small sample sizes. Specifically, it explores the independent effect of skewness and kurtosis on KR robustness for the values of skewness and kurtosis coefficients that are most frequently found in psychological and educational research data. To this end, a Monte Carlo simulation study was designed, considering a split-plot design with three levels of the between-subjects grouping factor and four levels of the within-subjects factor. Robustness is assessed in terms of the probability of type I error. The results showed that (1) the robustness of the KR procedure does not differ as a function of the violation or satisfaction of the sphericity assumption when small samples are used; (2) the LMM with KR can be a good option for analyzing total sample sizes of 45 or larger when their distributions are normal, slightly or moderately skewed, and with different degrees of kurtosis violation; (3) the effect of skewness on the robustness of the LMM with KR is greater than the corresponding effect of kurtosis for common values; and (4) when data are not normal and the total sample size is 30, the procedure is not robust. Alternative analyses should be performed when the total sample size is 30