Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.

There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses. We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions. The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively. Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects

A framework for power analysis using a structural equation modelling procedure

BACKGROUND: This paper demonstrates how structural equation modelling (SEM) can be used as a tool to aid in carrying out power analyses. For many complex multivariate designs that are increasingly being employed, power analyses can be difficult to carry out, because the software available lacks sufficient flexibility. Satorra and Saris developed a method for estimating the power of the likelihood ratio test for structural equation models. Whilst the Satorra and Saris approach is familiar to researchers who use the structural equation modelling approach, it is less well known amongst other researchers. The SEM approach can be equivalent to other multivariate statistical tests, and therefore the Satorra and Saris approach to power analysis can be used. METHODS: The covariance matrix, along with a vector of means, relating to the alternative hypothesis is generated. This represents the hypothesised population effects. A model (representing the null hypothesis) is then tested in a structural equation model, using the population parameters as input. An analysis based on the chi-square of this model can provide estimates of the sample size required for different levels of power to reject the null hypothesis. CONCLUSIONS: The SEM based power analysis approach may prove useful for researchers designing research in the health and medical spheres

The psychometric properties of the subscales of the GHQ-28 in a multi-ethnic maternal sample: results from the Born in Bradford cohort

Background: Poor maternal mental health can impact on children’s development and wellbeing; however, there is concern about the comparability of screening instruments administered to women of diverse ethnic origin. Methods: We used confirmatory factor analysis (CFA) and exploratory factor analysis (EFA) to examine the subscale structure of the GHQ-28 in an ethnically diverse community cohort of pregnant women in the UK (N = 5,089). We defined five groups according to ethnicity and language of administration, and also conducted a CFA between four groups of 1,095 women who completed the GHQ-28 both during and after pregnancy. Results: After item reduction, 17 of the 28 items were considered to relate to the same four underlying concepts in each group; however, there was variation in the response to individual items by women of different ethnic origin and this rendered between group comparisons problematic. The EFA revealed that these measurement difficulties might be related to variation in the underlying concepts being measured by the factors. Conclusions: We found little evidence to recommend the use of the GHQ-28 subscales in routine clinical or epidemiological assessment of maternal women in populations of diverse ethnicity