In several observational contexts where different raters evaluate a set of
items, it is common to assume that all raters draw their scores from the same
underlying distribution. However, a plenty of scientific works have evidenced
the relevance of individual variability in different type of rating tasks. To
address this issue the intra-class correlation coefficient (ICC) has been used
as a measure of variability among raters within the Hierarchical Linear Models
approach. A common distributional assumption in this setting is to specify
hierarchical effects as independent and identically distributed from a normal
with the mean parameter fixed to zero and unknown variance. The present work
aims to overcome this strong assumption in the inter-rater agreement estimation
by placing a Dirichlet Process Mixture over the hierarchical effects' prior
distribution. A new nonparametric index λ is proposed to quantify
raters polarization in presence of group heterogeneity. The model is applied on
a set of simulated experiments and real world data. Possible future directions
are discussed