Hierarchical Meta-Analysis: A Simulation Study Comparing Classical Random Effects and Fully Bayesian Methods

Meta-analytic data have a natural hierarchical structure to them, where individuals are nested within studies, and have both within-and between-study variation to model. A random-effects hierarchical linear model is useful to conduct a meta-analysis because it allows one to appropriately parse out the two components of variation that exist within and across studies to determine an observed effect. Empirical Bayes estimation considers the reliability of variance estimates; when the reliability of the effect size estimate for a study is high, substantial weight is placed on that estimate. However, problems with estimation arise when the number of studies and their sample size is small. Although time-consuming to employ, fully Bayesian methods offer a solution, but few studies systematically compare random-effects to fully Bayesian methods. A simulation study was performed varying certain characteristics of meta-analyses, such as the number of studies, their sample size and level of heterogeneity across studies, to determine under which condition(s) a fully Bayesian method improves meta-analytic findings. Results are unexpectedly inconsistent, whereby certain scenarios in which the number of studies is small and level of heterogeneity large, show that empirical Bayes performs better than the fully Bayesian method. Despite this, bias and mean-squared error are lower, on average, among the fully Bayesian models, with a model specifying a Cauchy prior on τ performing best for the most favorable scenario. Implications and areas for future study are discussed