Analyzing Nonverbal Listener Responses using Parallel Recordings of Multiple Listeners

In this paper we study nonverbal listener responses on a corpus with multiple parallel recorded listeners. These listeners were meant to believe that they were the sole listener, while in fact there were three persons listening to the same speaker. The speaker could only see one of the listeners. We analyze the impact of the particular setup of the corpus on the behavior and perception of the two types of listeners; the listeners that could be seen by the speaker and the listeners that could not be seen. Furthermore we compare the nonverbal listening behaviors of these three listeners to each other with regard to timing and form. We correlate these behaviors with behaviors of the speaker, like pauses and whether the speaker is looking at the listeners or not

A Survey on Evaluation Metrics for Backchannel Prediction Models

In this paper we give an overview of the evaluation metrics used to measure the performance of backchannel prediction models. Both objective and subjective evaluation metrics are discussed. The survey shows that almost every backchannel prediction model is evaluated with a different evaluation metric. This makes comparison between developed models unreliable, even beside the other variables in play, such as different corpora, language, conversational setting, amount of data and/or definition of the term backchannel

Observations on listener responses from multiple perspectives

Proceedings of the 3rd Nordic Symposium on Multimodal Communication. Editors: Patrizia Paggio, Elisabeth Ahlsén, Jens Allwood, Kristiina Jokinen, Costanza Navarretta. NEALT Proceedings Series, Vol. 15 (2011), 48–55. © 2011 The editors and contributors. Published by Northern European Association for Language Technology (NEALT) http://omilia.uio.no/nealt . Electronically published at Tartu University Library (Estonia) http://hdl.handle.net/10062/22532

A Multimodal Analysis of Vocal and Visual Backchannels in Spontaneous Dialogs

Backchannels (BCs) are short vocal and visual listener responses that signal attention, interest, and understanding to the speaker. Previous studies have investigated BC prediction in telephone-style dialogs from prosodic cues. In contrast, we consider spontaneous face-to-face dialogs. The additional visual modality allows speaker and listener to monitor each other's attention continuously, and we hypothesize that this affects the BC-inviting cues. In this study, we investigate how gaze, in addition to prosody, can cue BCs. Moreover, we focus on the type of BC performed, with the aim to find out whether vocal and visual BCs are invited by similar cues. In contrast to telephone-style dialogs, we do not find rising/falling pitch to be a BC-inviting cue. However, in a face-to-face setting, gaze appears to cue BCs. In addition, we find that mutual gaze occurs significantly more often during visual BCs. Moreover, vocal BCs are more likely to be timed during pauses in the speaker's speech

Challenges in Synchronized Behavior Realization for Different Robotic Embodiments

de Kok I, Hemminghaus J, Kopp S. Challenges in Synchronized Behavior Realization for Different Robotic Embodiments. Proceedings of the Interaction with Agents and Robots: Different Embodiments, Common Challenges. 2017

Timing and Grounding in Motor Skill Coaching Interaction: Consequences for the Information State

Hough J, de Kok I, Schlangen D, Kopp S. Timing and Grounding in Motor Skill Coaching Interaction: Consequences for the Information State. In: Proceedings of the 19th SemDial Workshop on the Semantics and Pragmatics of Dialogue (goDIAL). 2015: 86-94

Deictic gestures in coaching interactions

de Kok I, Hough J, Schlangen D, Kopp S. Deictic gestures in coaching interactions. In: Proceedings of the Workshop on Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction - MA3HMI '16. New York, NY: Association for Computing Machinery (ACM); 2016: 1

Dialogue Structure of Coaching Sessions

de Kok I, Hough J, Frank C, Schlangen D, Kopp S. Dialogue Structure of Coaching Sessions. In: Proceedings of the 18th SemDial Workshop on the Semantics and Pragmatics of Dialogue (DialWatt), Posters. Herriot-Watt University; 2014: 167-169