Controlling the Gaze of Conversational Agents

We report on a pilot experiment that investigated the effects of different eye gaze behaviours of a cartoon-like talking face on the quality of human-agent dialogues. We compared a version of the talking face that roughly implements some patterns of human-like behaviour with\ud two other versions. In one of the other versions the shifts in gaze were kept minimal and in the other version the shifts would occur randomly. The talking face has a number of restrictions. There is no speech recognition, so questions and replies have to be typed in by the users\ud of the systems. Despite this restriction we found that participants that conversed with the agent that behaved according to the human-like patterns appreciated the agent better than participants that conversed with the other agents. Conversations with the optimal version also\ud proceeded more efficiently. Participants needed less time to complete their task

Experimenting with the Gaze of a Conversational Agent

We have carried out a pilot experiment to investigate the effects of different eye gaze behaviors of a cartoon-like talking face on the quality of human-agent dialogues. We compared a version of the talking face that roughly implements some patterns of humanlike behavior with two other versions. We called this the optimal version. In one of the other versions the shifts in gaze were kept minimal and in the other version the shifts would occur randomly. The talking face has a number of restrictions. There is no speech recognition, so questions and replies have to\ud be typed in by the users of the systems. Despite this restriction we found that participants that conversed with the optimal agent appreciated the agent more than participants that conversed with the other agents. Conversations with the optimal version proceeded more efficiently. Participants needed less time to complete their task

A Multimodal Analysis of Teacher-Student Interactions in Reading Recovery Writing Sessions

The day-to-day use of language and social actions in the classroom represents different modes of communication and are used as tools to negotiate the academic demands of the curriculum (Bodrova & Leong, 2007; Vygotsky, 1986). Teachers and students rely on verbal and nonverbal modes of communication to navigate teaching and learning in the classroom (Johnston, 2004; Lose, 2008; Mercer, 2008). Thus, a range of communicative interactions were examined through the lens of sociocultural theory within a worldview of constructivism as teachers and children engaged in learning. The purpose of this study was to identify the modes through which teachers and students communicated and interacted to co-construct meaning, and the extent to which these modes were read, interpreted, and understood by each other in a Reading Recovery writing lesson. The following questions guided the research: What modes do teachers and students use to communicate in the writing portion of Reading Recovery lessons? To what extent do teachers and students read, interpret, and understand each other’s modal interactions in the writing portion of Reading Recovery lessons? What modal adjustments do teachers make to scaffold and adapt instruction for student learning? This qualitative multi-case study design (Yin, 2014) investigated different modes of communication used with Reading Recovery teachers and their respective students in a one-on-one instructional setting focused on writing. Data collection included field notes, audio/video recordings, student work samples, and the researcher’s journal. Data was analyzed using multimodal interaction analysis (Norris, 2004) along with Navarro (2008) who also studies body language. The study found literacy learning involves a complex set of communicative practices and Reading Recovery teachers and students used a range of modes to communicate and respond to each other in the writing portion of the lesson. These modal responses foster or inhibit the co-construction of meaning in teaching and learning. This study adds to the literature that considers interactional and social dimensions of learning for students who struggle with some aspect of literacy learning thus preventing literacy failure and referral to special education

Designing and Implementing a Platform for Collecting Multi-Modal Data of Human-Robot Interaction

This paper details a method of collecting video and audio recordings of people inter- acting with a simple robot interlocutor. The interaction is recorded via a number of cameras and microphones mounted on and around the robot. The system utilised a number of technologies to engage with interlocutors including OpenCV, Python, and Max MSP. Interactions over a three month period were collected at The Science Gallery in Trinity College Dublin. Visitors to the gallery freely engaged with the robot, with interactions on their behalf being spontaneous and non-scripted. The robot dialogue was a set pattern of utterances to engage interlocutors in a simple conversation. A large number of audio and video recordings were collected over a three month period

Spotting Agreement and Disagreement: A Survey of Nonverbal Audiovisual Cues and Tools

While detecting and interpreting temporal patterns of non–verbal behavioral cues in a given context is a natural and often unconscious process for humans, it remains a rather difficult task for computer systems. Nevertheless, it is an important one to achieve if the goal is to realise a naturalistic communication between humans and machines. Machines that are able to sense social attitudes like agreement and disagreement and respond to them in a meaningful way are likely to be welcomed by users due to the more natural, efficient and human–centered interaction they are bound to experience. This paper surveys the nonverbal cues that could be present during agreement and disagreement behavioural displays and lists a number of tools that could be useful in detecting them, as well as a few publicly available databases that could be used to train these tools for analysis of spontaneous, audiovisual instances of agreement and disagreement

Proceedings

Proceedings of the 3rd Nordic Symposium on Multimodal Communication. Editors: Patrizia Paggio, Elisabeth Ahlsén, Jens Allwood, Kristiina Jokinen, Costanza Navarretta. NEALT Proceedings Series, Vol. 15 (2011), vi+87 pp. © 2011 The editors and contributors. Published by Northern European Association for Language Technology (NEALT) http://omilia.uio.no/nealt . Electronically published at Tartu University Library (Estonia) http://hdl.handle.net/10062/22532

A Review of Verbal and Non-Verbal Human-Robot Interactive Communication

In this paper, an overview of human-robot interactive communication is presented, covering verbal as well as non-verbal aspects of human-robot interaction. Following a historical introduction, and motivation towards fluid human-robot communication, ten desiderata are proposed, which provide an organizational axis both of recent as well as of future research on human-robot communication. Then, the ten desiderata are examined in detail, culminating to a unifying discussion, and a forward-looking conclusion