Teaching Virtual Characters to use Body Language

Non-verbal communication, or “body language”, is a critical component in constructing believable virtual characters. Most often, body language is implemented by a set of ad-hoc rules.We propose a new method for authors to specify and refine their character’s body-language responses. Using our method, the author watches the character acting in a situation, and provides simple feedback on-line. The character then learns to use its body language to maximize the rewards, based on a reinforcement learning algorithm

Analyzing Nonverbal Listener Responses using Parallel Recordings of Multiple Listeners

In this paper we study nonverbal listener responses on a corpus with multiple parallel recorded listeners. These listeners were meant to believe that they were the sole listener, while in fact there were three persons listening to the same speaker. The speaker could only see one of the listeners. We analyze the impact of the particular setup of the corpus on the behavior and perception of the two types of listeners; the listeners that could be seen by the speaker and the listeners that could not be seen. Furthermore we compare the nonverbal listening behaviors of these three listeners to each other with regard to timing and form. We correlate these behaviors with behaviors of the speaker, like pauses and whether the speaker is looking at the listeners or not

How Do I Address You? Modelling addressing behavior based on an analysis of a multi-modal corpora of conversational discourse

Addressing is a special kind of referring and thus principles of multi-modal referring expression generation will also be basic for generation of address terms and addressing gestures for conversational agents. Addressing is a special kind of referring because of the different (second person instead of object) role that the referent has in the interaction. Based on an analysis of addressing behaviour in multi-party face-to-face conversations (meetings, TV discussions as well as theater plays), we present outlines of a model for generating multi-modal verbal and non-verbal addressing behaviour for agents in multi-party interactions

Continuous Interaction with a Virtual Human

Attentive Speaking and Active Listening require that a Virtual Human be capable of simultaneous perception/interpretation and production of communicative behavior. A Virtual Human should be able to signal its attitude and attention while it is listening to its interaction partner, and be able to attend to its interaction partner while it is speaking – and modify its communicative behavior on-the-fly based on what it perceives from its partner. This report presents the results of a four week summer project that was part of eNTERFACE’10. The project resulted in progress on several aspects of continuous interaction such as scheduling and interrupting multimodal behavior, automatic classification of listener responses, generation of response eliciting behavior, and models for appropriate reactions to listener responses. A pilot user study was conducted with ten participants. In addition, the project yielded a number of deliverables that are released for public access

A Review of Verbal and Non-Verbal Human-Robot Interactive Communication

In this paper, an overview of human-robot interactive communication is presented, covering verbal as well as non-verbal aspects of human-robot interaction. Following a historical introduction, and motivation towards fluid human-robot communication, ten desiderata are proposed, which provide an organizational axis both of recent as well as of future research on human-robot communication. Then, the ten desiderata are examined in detail, culminating to a unifying discussion, and a forward-looking conclusion

Multimodal language processing in human communication

Multiple layers of visual (and vocal) signals, plus their different onsets and offsets, represent a significant semantic and temporal binding problem during face-to-face conversation. Despite this complex unification process, multimodal messages appear to be processed faster than unimodal messages. Multimodal gestalt recognition and multilevel prediction are proposed to play a crucial role in facilitating multimodal language processing. The basis of the processing mechanisms involved in multimodal language comprehension is hypothesized to be domain general, coopted for communication, and refined with domain-specific characteristics. A new, situated framework for understanding human language processing is called for that takes into consideration the multilayered, multimodal nature of language and its production and comprehension in conversational interaction requiring fast processing

Creating pre-Evaluation opportunity spaces in IRE sequences: Evidence from Italian L2 classrooms in a University Context.

This thesis explores L2 classroom teacher-fronted activities organised in Initiation-Response- Evaluation (IRE) sequences, during beginner and intermediate lessons of Italian at the University level. More specifically, the study analyses the ways in which teachers address a variety of pedagogical contingencies while simultaneously progressing the interaction. It is argued that the tripartite sequential structure provides the teachers with pre-evaluative moments - here defined as pre-Evaluation opportunity spaces - emerging between the student’s responsive move (R) and the teacher’s third positioned evaluation (E). The research draws upon 30 hours of video- and audio-recorded interactions from two University Italian L2 classrooms. The study is informed by multimodal Conversation Analysis and socio-interactional approaches to language learning. Classroom interaction is, thus, regarded as one institutional type of social interaction and - as such - is viewed as jointly achieved by participants, sequentially organised, and relentlessly negotiated on a moment-by-moment basis. The findings show that the teachers regularly exploit specific IRE sequential affordances, such as the inter-move space between the student’s responsive move and the teacher’s evaluation. In particular, the fine-grained analysis of the teachers’ multimodal conduct uncovers how such opportunity space arising between Response and Evaluation may be employed in order to invite peer-correction practices, manage shifting classroom participation frameworks, distribute agency in the L2 classroom, and orient to the omnirelevant property of sequential progressivity while attending to concurrent institutional pressures. Furthermore, the analysis unearths how such intra-move space might be organised through the mobilisation of different semiotic material, such as head nods, pointing gestures, gaze, and body orientation. The findings confirm the adaptive quality of the IRE sequence organisation as one fundamental infrastructure that embodies the reflexive relationship between pedagogy and interaction

Embodied Communicative Activity in Cooperative Conversational Interactions - studies in Visual Interaction Management

Peer reviewe

Social Communication And Interaction Amongst Children And Young People With Severe Learning Difficulties

This study explores the social and communicative interactions among children and young people in special schools who have severe learning difficulties. It considers the concept ofphatic communion or 'small talk' and its importance as a social and cultural practice. The study focuses on the actual communication of the children and young people and investigates the functions, patterns and strategies they employ. It considers the development of language and communication from birth onwards, particularly the processes involved in the progress of increasing skills in socially interactive communication. The study reviews literature concerning the development of communication in proto-conversations to the talk of children of typical development and children and adults with learning difficulties. It also investigates the role of non-verbal communication and context and how these can affect and support the social interaction process. The study employed an observational approach to data collection,recording predominantly by field notes, but also utilising some video and audio recording.The data were transcribed and analysed to provide examples of the range and variety of socio-communicative interactions between the children and young people. The findings demonstrate the multiplexity of skill involved in creating and perpetuating the social dynamics of amicable interaction. Finally conclusions are drawn to provide some significance of the impact that this study may offer to professionals working in the special educational sector, particularly with children and young people with severe learning difficulties