764 research outputs found
Grounding truth via ordinal annotation
The question of how to best annotate affect within
available content has been a milestone challenge for affective
computing. Appropriate methods and tools addressing that question
can provide better estimations of the ground truth which, in
turn, may lead to more efficient affect detection and more reliable
models of affect. This paper introduces a rank-based real-time
annotation tool, we name AffectRank, and compares it against the
popular rating-based real-time FeelTrace tool through a proofof-
concept video annotation experiment. Results obtained suggest
that the rank-based (ordinal) annotation approach proposed
yields significantly higher inter-rater reliability and, thereby,
approximation of the underlying ground truth. The key findings
of the paper demonstrate that the current dominant practice
in continuous affect annotation via rating-based labeling is
detrimental to advancements in the field of affective computing.The authors would like to thank all annotators that participated
in the reported experiments. We would also like to
thank Gary Hili and Ryan Abela for providing access to the
Eryi dataset. The work is supported, in part, by the EU-funded
FP7 ICT iLearnRW project (project no: 318803).peer-reviewe
The ordinal nature of emotions
Representing computationally everyday emotional
states is a challenging task and, arguably, one of the most fundamental
for affective computing. Standard practice in emotion annotation
is to ask humans to assign an absolute value of intensity
to each emotional behavior they observe. Psychological theories
and evidence from multiple disciplines including neuroscience,
economics and artificial intelligence, however, suggest that the
task of assigning reference-based (relative) values to subjective
notions is better aligned with the underlying representations
than assigning absolute values. Evidence also shows that we
use reference points, or else anchors, against which we evaluate
values such as the emotional state of a stimulus; suggesting
again that ordinal labels are a more suitable way to represent
emotions. This paper draws together the theoretical reasons to
favor relative over absolute labels for representing and annotating
emotion, reviewing the literature across several disciplines. We
go on to discuss good and bad practices of treating ordinal
and other forms of annotation data, and make the case for
preference learning methods as the appropriate approach for
treating ordinal labels. We finally discuss the advantages of
relative annotation with respect to both reliability and validity
through a number of case studies in affective computing, and
address common objections to the use of ordinal data. Overall,
the thesis that emotions are by nature relative is supported by
both theoretical arguments and evidence, and opens new horizons
for the way emotions are viewed, represented and analyzed
computationally.peer-reviewe
RankTrace : relative and unbounded affect annotation
How should annotation data be processed so that it
can best characterize the ground truth of affect? This paper
attempts to address this critical question by testing various
methods of processing annotation data on their ability to capture
phasic elements of skin conductance. Towards this goal the paper
introduces a new affect annotation tool, RankTrace, that allows
for the annotation of affect in a continuous yet unbounded
fashion. RankTrace is tested on first-person annotation lines
(traces) of tension elicited from a horror video game. The key
findings of the paper suggest that the relative processing of traces
via their mean gradient yields the best and most robust predictors
of phasic manifestations of skin conductance.peer-reviewe
Towards general models of player affect
While the primary focus of affective computing has
been on constructing efficient and reliable models of affect,
the vast majority of such models are limited to a specific task
and domain. This paper, instead, investigates how computational
models of affect can be general across dissimilar tasks; in
particular, in modeling the experience of playing very different
video games. We use three dissimilar games whose players
annotated their arousal levels on video recordings of their own
playthroughs. We construct models mapping ranks of arousal to
skin conductance and gameplay logs via preference learning and
we use a form of cross-game validation to test the generality of the
obtained models on unseen games. Our initial results comparing
between absolute and relative measures of the arousal annotation
values indicate that we can obtain more general models of player
affect if we process the model output in an ordinal fashion.peer-reviewe
Knowing Your Annotator: Rapidly Testing the Reliability of Affect Annotation
The laborious and costly nature of affect annotation is a key detrimental factor for obtaining large scale corpora with valid and reliable affect labels. Motivated by the lack of tools that can effectively determine an annotator's reliability, this paper proposes general quality assurance (QA) tests for real-time continuous annotation tasks. Assuming that the annotation tasks rely on stimuli with audiovisual components, such as videos, we propose and evaluate two QA tests: a visual and an auditory QA test. We validate the QA tool across 20 annotators that are asked to go through the test followed by a lengthy task of annotating the engagement of gameplay videos. Our findings suggest that the proposed QA tool reveals, unsurprisingly, that trained annotators are more reliable than the best of untrained crowdworkers we could employ. Importantly, the QA tool introduced can predict effectively the reliability of an affect annotator with 80% accuracy, thereby, saving on resources, effort and cost, and maximizing the reliability of labels solicited in affective corpora. The introduced QA tool is available and accessible through the PAGAN annotation platform
- …