Social behavior modeling based on Incremental Discrete Hidden Markov Models

12 pagesInternational audienceModeling multimodal face-to-face interaction is a crucial step in the process of building social robots or users-aware Embodied Conversational Agents (ECA). In this context, we present a novel approach for human behavior analysis and generation based on what we called "Incremental Discrete Hidden Markov Model" (IDHMM). Joint multimodal activities of interlocutors are first modeled by a set of DHMMs that are specific to supposed joint cognitive states of the interlocutors. Respecting a task-specific syntax, the IDHMM is then built from these DHMMs and split into i) a recognition model that will determine the most likely sequence of cognitive states given the multimodal activity of the in- terlocutor, and ii) a generative model that will compute the most likely activity of the speaker given this estimated sequence of cognitive states. Short-Term Viterbi (STV) decoding is used to incrementally recognize and generate behav- ior. The proposed model is applied to parallel speech and gaze data of interact- ing dyads

Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems

International audienceA person's emotions and state of mind are apparent in their face and eyes. As a Latin proverb states: "The face is the portrait of the mind; the eyes, its informers.". This presents a huge challenge for computer graphics researchers in the generation of artificial entities that aim to replicate the movement and appearance of the human eye, which is so important in human-human interactions. This State of the Art Report provides an overview of the efforts made on tackling this challenging task. As with many topics in Computer Graphics, a cross-disciplinary approach is required to fully understand the workings of the eye in the transmission of information to the user. We discuss the movement of the eyeballs, eyelids, and the head from a physiological perspective and how these movements can be modelled, rendered and animated in computer graphics applications. Further, we present recent research from psychology and sociology that seeks to understand higher level behaviours, such as attention and eye-gaze, during the expression of emotion or during conversation, and how they are synthesised in Computer Graphics and Robotics

The internet of toys: a posthuman and multimodal analysis of connected play

Background: The study that is reported in this paper focuses on an exploration of the role and nature of play in young children’s use of toys that connect physical and digital domains. Purpose: The purpose of the paper is to explore the nature of the connections that are made in play that transverses physical and virtual domains. The paper draws on post-human theory in order to explain some of the complexity of the play that occurs in these contexts. Research Design: The research took place in the UK and the overall study consisted of four distinct stages: (i) A survey of 2000 parents of children aged 0-5, focusing on children’s access to and use of tablet apps; (ii) Case studies of pre-school children’s use of apps in six families (iii) Observations of children aged 3-5 in a school using apps (iv) Content and multimodal analysis of apps. The focus of this paper is on (ii), although some of the survey data from the first stage of the study are also shared in order to provide context. Data Collection and Analysis: The focus for this paper is the play of a threeyear old girl, Amy. In addition to ethnographic data constructed over a 2- month period (field notes, interviews, photographs and films), Amy’s mother collected data between the researchers’ visit by making films of her daughter’s use of apps. Amy also collected data herself by wearing a GoPro Chestcam. The data that informs the analysis in this paper is a film created by Amy (11.05 minutes) and a video filmed by Amy’s mother (5.21 minutes). Data were both inductively analyzed using multimodal (inter)action analysis and deductively analysed, utilizing a posthumanist approach. Findings: Amy’s play connected digital and non-digital components in complex ways. An app and related physical object that typify the Internet of Toys provided opportunities for Amy’s play to take place across physical and digital domains, and the inorganic objects embedded in the electronic toy and related app were an important element of this play, shaping Amy’s responses at times. However, Amy’s play was not always determined by the design of the electronic objects and she demonstrated agency within play episodes. There were multiple connections made across a variety of domains/ dimensions, which added to the complexity of the play. Conclusions/Recommendations: Young children’s play increasingly connects digital and non-digital domains and post-humanist theories can enhance understanding of how connections across these time/spaces are made