Detecting a lack of association: An equivalence testing approach

Researchers often test for a lack of association between variables. A lack of association is usually established by demonstrating a non-significant relationship with a traditional test (e.g., Pearson’s r). However, for logical as well as statistical reasons, such conclusions are problematic. In this paper, we discuss and compare the empirical Type I error and power rates of three lack of association tests. The results indicate that large, sometimes very large , sample sizes are required for the test statistics to be appropriate. What is especially problematic is that the required sample sizes may exceed what is practically feasible for the conditions that are expected to be common among researchers in psychology. This paper highlights the importance of using available lack of association tests, instead of traditional tests of association, for demonstrating the independence of variables, and qualifies the conditions under which these tests are appropriate.Social Sciences and Humanities Research Council (SSHRC

Effects of nonnormality on test statistics for one-way independent groups designs

The data obtained from one-way independent groups designs is typically non-normal inform and rarely equally variable across treatment populations (i.e. population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e. the analysis of variance F test) typically provides invalid results (e.g. too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non-normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e. trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances,though not non-normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non-normal and heterogeneous.The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non-normal.Social Sciences and Humanities Research Council (SSHRC

Are we really that different from each other? The difficulties of focusing on similarities in cross-cultural research.

In this article we argue that there are 2 dominant underlying themes in discussions of strategies for dealing with diversity—similarity and difference. When we are dealing with social groups, a number of basic psychological processes, as well as popular media and research-based narratives, make it easier to highlight difference rather than similarity. This difference-based approach in research is inherently divisive, but the training that we receive as researchers in the field of psychology has taken us down this path. As a first step, we propose that researchers working in the area of cultural diversity should start making explicit attempts to highlight similarities between groups, even if such similarities are only based on the absence of observed statistical differences. Moreover, if we are going to be serious about demonstrating similarity between groups and certain types of universals in behavior, we should start embracing new approaches to data analyses and consider using statistical procedures that test for equivalence. We illustrate these new techniques using our own data. Finally, we argue that shifting our primary focus from difference to similarity is a worthwhile direction to pursue for successfully managing diversity in multicultural societies.Social Sciences and Humanities Research Council (SSHRC

Testing for negligible interaction: A coherent and robust approach.

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap-based intersection-union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a nonsignificant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per-contrast power of the proposed approach to the noncentrality-based negligible interaction test of Cheng and Shao (2007). For 2 x 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied, however only our proposed bootstrapping approach was satisfactory under nonnormality and/or variance heterogeneity. In general a x b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection-union approach.Social Sciences and Humanities Research Council of Canad

Factor structure of the Beck Hopelessness Scale in individuals with advanced cancer

Although the Beck Hopelessness Scale is often used with the seriously ill, its factor structure has been given relatively little consideration in this context. The factor structure of this scale was examined in a sample of 406 ambulatory patients with advanced lung or gastrointestinal cancer, using a sequential exploratory confirmatory factor analysis procedure. A two-factor model was consistent with the data: The first factor reflected a negative outlook and was labeled ‘negative expectations’; the second factor identified a sense of resignation and was labeled ‘loss of motivation.’ Implications regarding scoring of the scale in this population are discussed, as are implications of the two-factor structure for our understanding of hopelessness in individuals with advanced cancer.Social Sciences and Humanities Research Council (SSHRC

Robust tests of equivalence for k independent groups

A common question of interest to researchers in psychology is the equivalence of two or more groups. Failure to reject the null hypothesis of traditional hypothesis tests such as the ANOVA F‐test (i.e., H0: μ1 = … = μ k ) does not imply the equivalence of the population means. Researchers interested in determining the equivalence of k independent groups should apply a one‐way test of equivalence (e.g., Wellek, 2003). The goals of this study were to investigate the robustness of the one‐way Wellek test of equivalence to violations of homogeneity of variance assumption, and compare the Type I error rates and power of the Wellek test with a heteroscedastic version which was based on the logic of the one‐way Welch (1951) F‐test. The results indicate that the proposed Wellek–Welch test was insensitive to violations of the homogeneity of variance assumption, whereas the original Wellek test was not appropriate when the population variances were not equal.Social Sciences and Humanities Research Council (SSHRC

Latent variables and structural equation models for longitudinal relationships: an illustration in nutritional epidemiology

<p>Abstract</p> <p>Background</p> <p>The use of structural equation modeling and latent variables remains uncommon in epidemiology despite its potential usefulness. The latter was illustrated by studying cross-sectional and longitudinal relationships between eating behavior and adiposity, using four different indicators of fat mass.</p> <p>Methods</p> <p>Using data from a longitudinal community-based study, we fitted structural equation models including two latent variables (respectively baseline adiposity and adiposity change after 2 years of follow-up), each being defined, by the four following anthropometric measurement (respectively by their changes): body mass index, waist circumference, skinfold thickness and percent body fat. Latent adiposity variables were hypothesized to depend on a cognitive restraint score, calculated from answers to an eating-behavior questionnaire (TFEQ-18), either cross-sectionally or longitudinally.</p> <p>Results</p> <p>We found that high baseline adiposity was associated with a 2-year increase of the cognitive restraint score and no convincing relationship between baseline cognitive restraint and 2-year adiposity change could be established.</p> <p>Conclusions</p> <p>The latent variable modeling approach enabled presentation of synthetic results rather than separate regression models and detailed analysis of the causal effects of interest. In the general population, restrained eating appears to be an adaptive response of subjects prone to gaining weight more than as a risk factor for fat-mass increase.</p

Equivalence tests for comparing correlation and regression coefficients

Equivalence tests are an alternative to traditional difference-based tests for demonstrating a lack of association between two variables. While there are several recent studies investigating equivalence tests for comparing means, little research has been conducted on equivalence methods for evaluating the equivalence or similarity of two correlation coefficients or two regression coefficients. The current project proposes novel tests for evaluating the equivalence of two regression or correlation coefficients derived from the two one-sided tests (TOST) method (Schuirmann, 1987, J. Pharmacokinet. Biopharm, 15, 657) and an equivalence test by Anderson and Hauck (1983, Stat. Commun., 12, 2663). A simulation study was used to evaluate the performance of these tests and compare them with the common, yet inappropriate, method of assessing equivalence using nonrejection of the null hypothesis in difference-based tests. Results demonstrate that equivalence tests have more accurate probabilities of declaring equivalence than difference-based tests. However, equivalence tests require large sample sizes to ensure adequate power. We recommend the Anderson–Hauck equivalence test over the TOST method for comparing correlation or regression coefficients.Social Sciences and Humanities Research Council (SSHRC

Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses

Articles published in several prominent educational journals were examined to investigate the use of data-analysis tools by researchers in four research paradigms: between-subjects univariate designs, between-subjects multivariate designs, repeated measures designs, and covariance designs. In addition to examining specific details pertaining to the research design (e.g., sample size, group size equality/inequality) and methods employed for data analysis, we also catalogued whether: (a) validity assumptions were examined, (b) effect size indices were reported, (c) sample sizes were selected based on power considerations, and (d) appropriate textbooks and/or articles were cited to communicate the nature of the analyses that were performed. Our analyses imply that researchers rarely verify that validity assumptions are satisfied and accordingly typically use analyses that are nonrobust to assumption violations. In addition, researchers rarely report effect size statistics, nor do they routinely perform power analyses to determine sample size requirements. We offer many recommendations to rectify these shortcomings.Social Sciences and Humanities Research Counci

Data from an International Multi-Centre Study of Statistics and Mathematics Anxieties and Related Variables in University Students (the SMARVUS Dataset)

This large, international dataset contains survey responses from N = 12,570 students from 100 universities in 35 countries, collected in 21 languages. We measured anxieties (statistics, mathematics, test, trait, social interaction, performance, creativity, intolerance of uncertainty, and fear of negative evaluation), self-efficacy, persistence, and the cognitive reflection test, and collected demographics, previous mathematics grades, self-reported and official statistics grades, and statistics module details. Data reuse potential is broad, including testing links between anxieties and statistics/mathematics education factors, and examining instruments’ psychometric properties across different languages and contexts. Data and metadata are stored on the Open Science Framework website [https://osf.io/mhg94/]