Social Interactions in Immersive Virtual Environments: People, Agents, and Avatars

Immersive virtual environments (IVEs) have received increased popularity with applications in many fields. IVEs aim to approximate real environments, and to make users react similarly to how they would in everyday life. An important use case is the users-virtual characters (VCs) interaction. We interact with other people every day, hence we expect others to appropriately act and behave, verbally and non-verbally (i.e., pitch, proximity, gaze, turn-taking). These expectations also apply to interactions with VCs in IVEs, and this thesis tackles some of these aspects. We present three projects that inform the area of social interactions with a VC in IVEs, focusing on non-verbal behaviours. In our first study on interactions between people, we collaborated with the Social Neuroscience group at the Institute of Cognitive Neuroscience from UCL on a dyad multi-modal interaction. This aims to understand the conversation dynamics, focusing on gaze and turn-taking. The results show that people have a higher frequency of gaze change (from averted to direct and vice versa) when they are being looked at compared to when they are not. When they are not being looked at, they are also directing their gaze to their partners more compared to when they are being looked at. Another contribution of this work is the automated method of annotating speech and gaze data. Next, we consider agents’ higher-level non-verbal behaviours, covering social attitudes. We present a pipeline to collect data and train a machine learning (ML) model that detects social attitudes in a user-VC interaction. Here we collaborated with two game studios: Dream Reality Interaction and Maze Theory. We present a case study for the ML pipeline on social engagement recognition for the Peaky Blinders narrative VR game from Maze Theory studio. We use a reinforcement learning algorithm with imitation learning rewards and a temporal memory element. The results show that the model trained with raw data does not generalise and performs worse (60% accuracy) than the one trained with socially meaningful data (83% accuracy). In IVEs, people embody avatars and their appearance can impact social interactions. In collaboration with Microsoft Research, we report a longitudinal study in mixed-reality on avatar appearance in real-work meetings between co-workers comparing personalised full-body realistic and cartoon avatars. The results imply that when participants use realistic avatars first, they may have higher expectations and they perceive their colleagues’ emotional states with less accuracy. Participants may also become more accustomed to cartoon avatars as time passes and the overall use of avatars may lead to less accurately perceiving negative emotions. The work presented here contributes towards the field of detecting and generating nonverbal cues for VCs in IVEs. These are also important building blocks for creating autonomous agents for IVEs. Additionally, this work contributes to the games and work industry fields through an immersive ML pipeline for detecting social attitudes and through insights into using different avatar styles over time in real-world meetings

Nice is Different than Good: Longitudinal Communicative Effects of Realistic and Cartoon Avatars in Real Mixed Reality Work Meetings

We report a within-subjects study of the effect of realistic and cartoon avatars on communication, task satisfaction, and perceived sense of presence in mixed reality meetings. For 2 − 3 weeks, six groups of co-workers (14 people) held a recurring real work meeting using Microsoft HoloLens2 devices. Each person embodied a personalised full-body avatar with a realistic face and another with a cartoon face. Half the groups started in the realistic condition and the other half started in the cartoon condition; all groups switched conditions half-way. Initial results show that, overall, participants found the realistic avatars’ nonverbal behaviour more appropriate for the interaction and more useful for understanding their colleagues compared to the cartoon one. Regarding the results over time, we identify different insights for cartoon and realistic avatars based on the type of avatar was embodied first. We discuss the implications of these results for mixed and virtual reality meetings

More than buttons on controllers: engaging social interactions in narrative VR games through social attitudes detection

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program agents or non-player characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We present a collaborative work between two game studios (Maze Theory and Dream Reality Interactive) and academia to develop an immersive machine learning (ML) pipeline for detecting social engagement. Here we introduce the motivation and the methodology of the immersive ML pipeline, then we cover the motivation for the industry-academia collaboration, how it progressed, the implications of joined work on the industry and reflective insights on the collaboration. Overall, we highlight the industry-academia collaborative work on an immersive ML pipeline for detecting social engagement. We demonstrate how creatives could use ML and VR to expand their ability to design more engaging commercial games

Direct Gaze Triggers Higher Frequency of Gaze Change: An Automatic Analysis of Dyads in Unstructured Conversation

Nonverbal cues have multiple roles in social encounters, with gaze behaviour facilitating interactions and conversational flow. In this work, we explore the conversation dynamics in dyadic settings in a free-flow discussion. Using automatic analysis (rather than manual labelling), we investigate how the gaze behaviour of one person is related to how much the other person changes their gaze (frequency in gaze change) and what their gaze target is (direct or avert gaze). Our results show that when one person is looked at they change their gaze direction with a higher frequency compared to when they are not looked at. They also tend to maintain a direct gaze to the other person when they are not looked at

Using machine learning to generate engaging behaviours in immersive virtual environments

Our work aims at implementing autonomous agents for Immersive Virtual Reality (IVR). With the advances in IVR environments, users can be more engaged and respond realistically to the events delivered in IVR, a state described in literature as presence. Agents with engaging verbal and nonverbal behaviour help preserve the sense of presence in IVR. For instance, gaze behaviour plays an important role, having monitoring and communicative functions. The initial step is to look at a machine learning model that generates flexible and contextual gaze behaviour and takes into account the rapport between the user and the agent. In this paper, we present our progress to date on the problem of creating realistic nonverbal behaviour. This includes analysing a multimodal dyad data, creating a data-processing pipeline, implementing a Hidden Markov Model and linking the Python scripts with the VR game engine (Unity3D). Future work consists of using richer data for more complex machine learning models, with a final aim of integrating the gaze model (plus future nonverbal behaviour models) into an autonomous virtual character framework

Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g., head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes

Rolling horizon co-evolution in two-player general video game playing

Artificial Intelligence for General Video Game Playing (GVGP) is challenging not only because agents must adapt to a range of different games, but they must also make decisions within the time constraints of real-time video games. The General Video Game Artificial Intelligence framework (GVGAI) is a popular framework for GVGP. It features a two-player track where two agents play a game together, either competitively or cooperatively, which poses the additional challenge of considering another player. Commonly, agents only consider their own moves in these two-player games. In this paper we discuss and assess Rolling Horizon Co-evolutionary Planning (a modification to Rolling Horizon Evolutionary Algorithms) for two player GVGAI. We present experimental results on its effectiveness against other agents playing GVGAI games and show that co-evolution can improve results compared to a RHEA agent