Multi-party Interaction in a Virtual Meeting Room

This paper presents an overview of the work carried out at the HMI group of the University of Twente in the domain of multi-party interaction. The process from automatic observations of behavioral aspects through interpretations resulting in recognized behavior is discussed for various modalities and levels. We show how a virtual meeting room can be used for visualization and evaluation of behavioral models as well as a research tool for studying the effect of modified stimuli on the perception of behavior

The dynamics of gender perception and status in email-mediated group interaction V1.0

This paper takes the position that identity is not located in the individual but in the community in which each individual is recognised as a legitimate participant. Markers of identity such as gender, race, and socio-economic class are not visible in written interaction, but socialisation produces expectations regarding the positions, status and behaviour of dual gender roles, and such expectations can colour the ways in which participants in a mailing list respond to each other. Despite the fact that technological mediation appears to render social markers invisible, social categories such as gender can become even more relevant for interactants in these contexts. The study of interactional patterns on the mailing list Cybermind uses a scheme to classify posts in order to highlight participants’ responses to their projected addressees according to perceived gender. The findings indicate that style of response both to and by each ‘identified’ gender can be differentiated, and suggests that interactive behaviour contributes to the legitimation of local status of participants, but that this is not just a function of gender alone

Building Embodied Conversational Agents:Observations on human nonverbal behaviour as a resource for the development of artificial characters

"Wow this is so cool!" This is what I most probably yelled, back in the 90s, when my first computer program on our MSX computer turned out to do exactly what I wanted it to do. The program contained the following instruction: COLOR 10(1.1) After hitting enter, it would change the screen color from light blue to dark yellow. A few years after that experience, Microsoft Windows was introduced. Windows came with an intuitive graphical user interface that was designed to allow all people, so also those who would not consider themselves to be experienced computer addicts, to interact with the computer. This was a major step forward in human-computer interaction, as from that point forward no complex programming skills were required anymore to perform such actions as adapting the screen color. Changing the background was just a matter of pointing the mouse to the desired color on a color palette. "Wow this is so cool!". This is what I shouted, again, 20 years later. This time my new smartphone successfully skipped to the next song on Spotify because I literally told my smartphone, with my voice, to do so. Being able to operate your smartphone with natural language through voice-control can be extremely handy, for instance when listening to music while showering. Again, the option to handle a computer with voice instructions turned out to be a significant optimization in human-computer interaction. From now on, computers could be instructed without the use of a screen, mouse or keyboard, and instead could operate successfully simply by telling the machine what to do. In other words, I have personally witnessed how, within only a few decades, the way people interact with computers has changed drastically, starting as a rather technical and abstract enterprise to becoming something that was both natural and intuitive, and did not require any advanced computer background. Accordingly, while computers used to be machines that could only be operated by technically-oriented individuals, they had gradually changed into devices that are part of many people’s household, just as much as a television, a vacuum cleaner or a microwave oven. The introduction of voice control is a significant feature of the newer generation of interfaces in the sense that these have become more "antropomorphic" and try to mimic the way people interact in daily life, where indeed the voice is a universally used device that humans exploit in their exchanges with others. The question then arises whether it would be possible to go even one step further, where people, like in science-fiction movies, interact with avatars or humanoid robots, whereby users can have a proper conversation with a computer-simulated human that is indistinguishable from a real human. An interaction with a human-like representation of a computer that behaves, talks and reacts like a real person would imply that the computer is able to not only produce and understand messages transmitted auditorily through the voice, but also could rely on the perception and generation of different forms of body language, such as facial expressions, gestures or body posture. At the time of writing, developments of this next step in human-computer interaction are in full swing, but the type of such interactions is still rather constrained when compared to the way humans have their exchanges with other humans. It is interesting to reflect on how such future humanmachine interactions may look like. When we consider other products that have been created in history, it sometimes is striking to see that some of these have been inspired by things that can be observed in our environment, yet at the same do not have to be exact copies of those phenomena. For instance, an airplane has wings just as birds, yet the wings of an airplane do not make those typical movements a bird would produce to fly. Moreover, an airplane has wheels, whereas a bird has legs. At the same time, an airplane has made it possible for a humans to cover long distances in a fast and smooth manner in a way that was unthinkable before it was invented. The example of the airplane shows how new technologies can have "unnatural" properties, but can nonetheless be very beneficial and impactful for human beings. This dissertation centers on this practical question of how virtual humans can be programmed to act more human-like. The four studies presented in this dissertation all have the equivalent underlying question of how parts of human behavior can be captured, such that computers can use it to become more human-like. Each study differs in method, perspective and specific questions, but they are all aimed to gain insights and directions that would help further push the computer developments of human-like behavior and investigate (the simulation of) human conversational behavior. The rest of this introductory chapter gives a general overview of virtual humans (also known as embodied conversational agents), their potential uses and the engineering challenges, followed by an overview of the four studies

Measuring perceived social presence in distributed learning groups

Kreijns, K., Kirschner, P. A., Jochems, W. M. G., & Van Buuren, H. (2011). Measuring perceived social presence in distributed learning groups. Education and Information Technologies, 16, 365-381.Social presence – the degree to which „the other‟ in a communication appears to be a „real‟ person – has captured the attention of those dealing with learning in groups through computer-supported collaborative learning environments. The concept is important because it affects participation and social interaction, both necessary for effective collaboration and knowledge construction. This article reports on the construction and validation of a self-reporting (Dutch-language) Social Presence Scale to determine perceived social presence in distributed learning groups using computer-supported collaborative learning environments. The result is a onedimensional scale consisting of five items with an internal consistency of .81. We used a nomological network of similar constructs for further validation. The findings suggest that the Social Presence Scale has potential to be useful as a measure for social presence

Building Embodied Conversational Agents:Observations on human nonverbal behaviour as a resource for the development of artificial characters

"Wow this is so cool!" This is what I most probably yelled, back in the 90s, when my first computer program on our MSX computer turned out to do exactly what I wanted it to do. The program contained the following instruction: COLOR 10(1.1) After hitting enter, it would change the screen color from light blue to dark yellow. A few years after that experience, Microsoft Windows was introduced. Windows came with an intuitive graphical user interface that was designed to allow all people, so also those who would not consider themselves to be experienced computer addicts, to interact with the computer. This was a major step forward in human-computer interaction, as from that point forward no complex programming skills were required anymore to perform such actions as adapting the screen color. Changing the background was just a matter of pointing the mouse to the desired color on a color palette. "Wow this is so cool!". This is what I shouted, again, 20 years later. This time my new smartphone successfully skipped to the next song on Spotify because I literally told my smartphone, with my voice, to do so. Being able to operate your smartphone with natural language through voice-control can be extremely handy, for instance when listening to music while showering. Again, the option to handle a computer with voice instructions turned out to be a significant optimization in human-computer interaction. From now on, computers could be instructed without the use of a screen, mouse or keyboard, and instead could operate successfully simply by telling the machine what to do. In other words, I have personally witnessed how, within only a few decades, the way people interact with computers has changed drastically, starting as a rather technical and abstract enterprise to becoming something that was both natural and intuitive, and did not require any advanced computer background. Accordingly, while computers used to be machines that could only be operated by technically-oriented individuals, they had gradually changed into devices that are part of many people’s household, just as much as a television, a vacuum cleaner or a microwave oven. The introduction of voice control is a significant feature of the newer generation of interfaces in the sense that these have become more "antropomorphic" and try to mimic the way people interact in daily life, where indeed the voice is a universally used device that humans exploit in their exchanges with others. The question then arises whether it would be possible to go even one step further, where people, like in science-fiction movies, interact with avatars or humanoid robots, whereby users can have a proper conversation with a computer-simulated human that is indistinguishable from a real human. An interaction with a human-like representation of a computer that behaves, talks and reacts like a real person would imply that the computer is able to not only produce and understand messages transmitted auditorily through the voice, but also could rely on the perception and generation of different forms of body language, such as facial expressions, gestures or body posture. At the time of writing, developments of this next step in human-computer interaction are in full swing, but the type of such interactions is still rather constrained when compared to the way humans have their exchanges with other humans. It is interesting to reflect on how such future humanmachine interactions may look like. When we consider other products that have been created in history, it sometimes is striking to see that some of these have been inspired by things that can be observed in our environment, yet at the same do not have to be exact copies of those phenomena. For instance, an airplane has wings just as birds, yet the wings of an airplane do not make those typical movements a bird would produce to fly. Moreover, an airplane has wheels, whereas a bird has legs. At the same time, an airplane has made it possible for a humans to cover long distances in a fast and smooth manner in a way that was unthinkable before it was invented. The example of the airplane shows how new technologies can have "unnatural" properties, but can nonetheless be very beneficial and impactful for human beings. This dissertation centers on this practical question of how virtual humans can be programmed to act more human-like. The four studies presented in this dissertation all have the equivalent underlying question of how parts of human behavior can be captured, such that computers can use it to become more human-like. Each study differs in method, perspective and specific questions, but they are all aimed to gain insights and directions that would help further push the computer developments of human-like behavior and investigate (the simulation of) human conversational behavior. The rest of this introductory chapter gives a general overview of virtual humans (also known as embodied conversational agents), their potential uses and the engineering challenges, followed by an overview of the four studies

Moving together: the organisation of non-verbal cues during multiparty conversation

PhDConversation is a collaborative activity. In face-to-face interactions interlocutors have mutual access to a shared space. This thesis aims to explore the shared space as a resource for coordinating conversation. As well demonstrated in studies of two-person conversations, interlocutors can coordinate their speech and non-verbal behaviour in ways that manage the unfolding conversation. However, when scaling up from two people to three people interacting, the coordination challenges that the interlocutors face increase. In particular speakers must manage multiple listeners. This thesis examines the use of interlocutors’ bodies in shared space to coordinate their multiparty dialogue. The approach exploits corpora of motion captured triadic interactions. The thesis first explores how interlocutors coordinate their speech and non-verbal behaviour. Inter-person relationships are examined and compared with artificially created triples who did not interact. Results demonstrate that interlocutors avoid speaking and gesturing over each other, but tend to nod together. Evidence is presented that the two recipients of an utterance have different patterns of head and hand movement, and that some of the regularities of movement are correlated with the task structure. The empirical section concludes by uncovering a class of coordination events, termed simultaneous engagement events, that are unique to multiparty dialogue. They are constructed using combinations of speaker head orientation and gesture orientation. The events coordinate multiple recipients of the dialogue and potentially arise as a result of the greater coordination challenges that interlocutors face. They are marked in requiring a mutually accessible shared space in order to be considered an effective interactional cue. The thesis provides quantitative evidence that interlocutors’ head and hand movements are organised by their dialogue state and the task responsibilities that the bear. It is argued that a shared interaction space becomes a more important interactional resource when conversations scale up to three people

Contextual Factors Affecting Information Sharing Patterns in Technology Mediated Communication

In this thesis, we investigate how and what contextual factors affect user’s information sharing. We build our work on six individual research projects which cover a variety of systems (search engines, social network sites, teleconferencing systems, monitoring technology, and general purpose conversational agents) in a variety of communication scenarios with diverse relationships and dispositions of users. Alongside detailed findings for particular systems and communication scenarios from each individual project, we provide a consolidated analysis of these results across systems and scenarios, which allows us to identify patterns specific for different system types and aspects shared between systems. In particular, we show that depending on the system’s position between a user and an intended information receiving agent – whether communication happens through, around, or directly with the system – the system should have different patterns of operational adaptation to communication context. Specifically, when communication happens through the system, the system needs to gather communication context unavailable to the user and integrate it into information communication; when communication happens around the system, the system should adapt its operations to provide information in the most contextually suitable format; finally, when a user communicates with the system, the role of the system is to “match” this context in communication with the user. We then argue that despite the differences between system types in patterns of required context-based adaptation, there are contextual factors affecting user’s information sharing intent that should be acknowledged across systems. Grounded in our cumulative findings and analysis of related literature, we identify four such high-level contextual factors. We then present these four factors synthesized into an early design framework, which we call SART according to the included factors of space, addressee, reason, and time. Each factor in SART is presented as a continuum defined through a descriptive dichotomy: perceived breadth of communication space (public to private); perceived specificity of an information addressee (defined to undefined); intended reason for information sharing (instrumental to objective); and perceived time of information relevance and life-span (immediate to indefinite)

CMC-core: a schema for the representation of CMC corpora in TEI

Dans cet article, nous décrivons un schéma et des modèles de représentation développés pour structurer les corpus de communication médiée par ordinateur (CMC) en suivant les recommandations de la Text Encoding Initiative (TEI). Nous considérons le discours CMC comme un échange dialogique entre humains, organisé de manière séquentielle. Nous insistons d’abord sur le fait que de nombreuses caractéristiques de la CMC ne sont pas traitées de manière adéquate par les schémas et les outils actuels d’encodage de corpus. Nous formulons donc un ensemble de recommandations pour représenter la CMC avec des schémas d’encodage, en insistant sur le fait que la TEI nous semble être un cadre particulièrement approprié pour l’encodage des corpus CMC. Nous proposons une modélisation des unités de base de la CMC (énoncés, messages et actions non verbales) ainsi que des structures de niveaux macro- et micro des interactions dans les environnements de la CMC. À partir de ces modèles, nous introduisons le CMC-core, un noyau TEI pour l’encodage des corpus CMC, qui définit un ensemble de traits d’encodage spécifiques à la CMC sur quatre niveaux: (i) éléments, (ii) classes de modèles, (iii) classes d'attributs et (iv) modules de l'infrastructure TEI. La description du noyau proposé est illustrée au moyen d’exemples extraits des corpus des chercheurs du groupe SIG TEI CMC, représentant une grande variété de genres de la CMC (le chat, le wiki talk, le tweet, le blog, les interactions Second Life…). Le matériel décrit, i.e. les schémas, les exemples d’encodage et la documentation, est disponible sur le Wiki du SIG CMC TEI et accompagnera une demande d’enrichissement de la TEI (TEI feature request) au conseil de la TEI à la fin de l’année 2019.In this Paper, we describe a schema and models which have been developed for the representation of corpora of computer-mediated communicatin (CMC corpora) using the representation framework provided by the Text Encoding Initiative (TEI). We characterise CMC discourse as dialogic, sequentially organised interchange between humans and point out that many features of CMC are not adequately handled by current corpus encoding schemas and tools. We formulate desiderata for a representation of CMC in encoding schemes and argue why the TEI is a suitable framework for the encoding of CMC corpora. We propose a model of basic CMC units (utterances, posts, and nonverbal activities) and the macro- and micro-level structures of interactions in CMC environments. Based on these models, we introduce CMC-core, a TEI customisation for the encoding of CMC corpora, which defines CMC-specific encoding features on the four levels of elements, model classes, attribute classes, and modules of the TEI infrastructure. The description of our customisation is illustrated by encoding examples from corpora by researchers of the TEI SIG CMC, representing a variety of CMC genres, i.e. chat, wiki talk, twitter, blog, and Second Life interactions. The material described, i.e. schemata, encoding examples, and documentation, is available from the of the TEI CMC SIG Wiki and will accompany a feature request to the TEI council in late 2019

Gender variation in writing : analyzing online dating ads

This dissertation presents a study of gendered language variation and linguistic indexicality in computer-mediated communication. A two-pronged approach combining the analysis of language production in a corpus of 103,000 English-language online dating ads with a language perception study (891 participants) is taken towards identifying the usage patterns and social meanings of nine features of e-grammar (Herring 2012). The indexicalities of features exhibiting gendered patterns in production as well as perception, emoticons (e.g. : ) ) and prosodic items (e.g. haha), are discussed in light of their linguistic and social context. Drawing on empirical research on American gender ideologies, the study argues that they index characteristics such as friendliness and emotional expressiveness, both stereotypically associated with women. In an instance of indirect indexicality (Ochs 1992), they are then linked to femininity in this type of computer-mediated communication. In production, the same features exhibit a strong audience effect (Bell 1984): women, for instance, use them more frequently in ads directed at other women. Throughout the analysis, the study makes use of and illustrates use cases for computational tools such as machine learning algorithms or automatic part-of-speech tagging in sociolinguistic research. At the same time, it attempts to strike a balance between a quantitative, data-driven approach and the nuanced analysis of gender identities and linguistic indexicality in the performance of gendered identitiesEnglis