A Unified Approach to Estimating the Intraclass Correlation Coefficient and Its Bias: An Exploratory Study

Many methods are used to measure interrater reliability for studies where each target receives ratings by a different set of judges. The purpose of this study is to explore the use of hierarchical linear modeling for estimating interrater reliability using the intraclass correlation coefficient. This study provides a description of how the ICC can be estimated using hierarchical linear modeling, recommends an appropriate non-parametric bootstrapping method, illustrates how both can be implemented to obtain a point estimate and an estimate of bias, and explores the viability of using these statistical tools to obtain such estimates. Results indicated that hierarchical linear modeling and the non-parametric bootstrap method can be used on both continuous and binary data to provide point and bias estimates of interrater reliability