An important challenge in building automatic affective state
recognition systems is establishing the ground truth. When the groundtruth
is not available, observers are often used to label training and testing
sets. Unfortunately, inter-rater reliability between observers tends to
vary from fair to moderate when dealing with naturalistic expressions.
Nevertheless, the most common approach used is to label each expression
with the most frequent label assigned by the observers to that expression.
In this paper, we propose a general pattern recognition framework
that takes into account the variability between observers for automatic
affect recognition. This leads to what we term a multi-score learning
problem in which a single expression is associated with multiple values
representing the scores of each available emotion label. We also propose
several performance measurements and pattern recognition methods for
this framework, and report the experimental results obtained when testing
and comparing these methods on two affective posture datasets
