1 research outputs found
An analysis of observation length requirements for machine understanding of human behaviors from spoken language
The task of quantifying human behavior by observing interaction cues is an
important and useful one across a range of domains in psychological research
and practice. Machine learning-based approaches typically perform this task by
first estimating behavior based on cues within an observation window, such as a
fixed number of words, and then aggregating the behavior over all the windows
in that interaction. The length of this window directly impacts the accuracy of
estimation by controlling the amount of information being used. The exact link
between window length and accuracy, however, has not been well studied,
especially in spoken language. In this paper, we investigate this link and
present an analysis framework that determines appropriate window lengths for
the task of behavior estimation. Our proposed framework utilizes a two-pronged
evaluation approach: (a) extrinsic similarity between machine predictions and
human expert annotations, and (b) intrinsic consistency between intra-machine
and intra-human behavior relations. We apply our analysis to real-life
conversations that are annotated for a large and diverse set of behavior codes
and examine the relation between the nature of a behavior and how long it
should be observed. We find that behaviors describing negative and positive
affect can be accurately estimated from short to medium-length expressions
whereas behaviors related to problem-solving and dysphoria require much longer
observations and are difficult to quantify from language alone. These findings
are found to be generally consistent across different behavior modeling
approaches.Comment: converted to CSL format, restructured presentation of analysis and
methodology, moved finer details to Appendix, enlarged figures and text,
fixed typos and notational inconsistenc