Social Interactions in Immersive Virtual Environments: People, Agents, and Avatars

Immersive virtual environments (IVEs) have received increased popularity with applications in many fields. IVEs aim to approximate real environments, and to make users react similarly to how they would in everyday life. An important use case is the users-virtual characters (VCs) interaction. We interact with other people every day, hence we expect others to appropriately act and behave, verbally and non-verbally (i.e., pitch, proximity, gaze, turn-taking). These expectations also apply to interactions with VCs in IVEs, and this thesis tackles some of these aspects. We present three projects that inform the area of social interactions with a VC in IVEs, focusing on non-verbal behaviours. In our first study on interactions between people, we collaborated with the Social Neuroscience group at the Institute of Cognitive Neuroscience from UCL on a dyad multi-modal interaction. This aims to understand the conversation dynamics, focusing on gaze and turn-taking. The results show that people have a higher frequency of gaze change (from averted to direct and vice versa) when they are being looked at compared to when they are not. When they are not being looked at, they are also directing their gaze to their partners more compared to when they are being looked at. Another contribution of this work is the automated method of annotating speech and gaze data. Next, we consider agents’ higher-level non-verbal behaviours, covering social attitudes. We present a pipeline to collect data and train a machine learning (ML) model that detects social attitudes in a user-VC interaction. Here we collaborated with two game studios: Dream Reality Interaction and Maze Theory. We present a case study for the ML pipeline on social engagement recognition for the Peaky Blinders narrative VR game from Maze Theory studio. We use a reinforcement learning algorithm with imitation learning rewards and a temporal memory element. The results show that the model trained with raw data does not generalise and performs worse (60% accuracy) than the one trained with socially meaningful data (83% accuracy). In IVEs, people embody avatars and their appearance can impact social interactions. In collaboration with Microsoft Research, we report a longitudinal study in mixed-reality on avatar appearance in real-work meetings between co-workers comparing personalised full-body realistic and cartoon avatars. The results imply that when participants use realistic avatars first, they may have higher expectations and they perceive their colleagues’ emotional states with less accuracy. Participants may also become more accustomed to cartoon avatars as time passes and the overall use of avatars may lead to less accurately perceiving negative emotions. The work presented here contributes towards the field of detecting and generating nonverbal cues for VCs in IVEs. These are also important building blocks for creating autonomous agents for IVEs. Additionally, this work contributes to the games and work industry fields through an immersive ML pipeline for detecting social attitudes and through insights into using different avatar styles over time in real-world meetings

Filters for Wi-Fi Generated Crowd Movement Data

Cities represent large groups of people that share a common infrastructure, common social groups and/or common interests. With the development of new technologies current cities aim to become what is known as smart cities, in which all the small details of these large constructs are controlled to better improve the quality of life of its inhabitants. One of the important gears that powers a city is given by traffic, be it vehicular or pedestrian. As such traffic is closely related to all other activities that take place inside of a city. Understanding traffic is still a difficult process as we have to be able to not only measure it in the sense of how many people are using a particular path but also in analyzing where people are going and when, while still maintaining individual privacy. And all this has to be done at a scale that would cover most if not all individuals in a city. With the high increase in smartphones adoption we can reliably assume that a large part of the population in cities are carrying with them, at all times, at least one Wi-Fi enabled device. Because Wi-Fi devices are regularly transmitting signals we can rely on these devices to detect individual's movements unobtrusively without identifying or tracking any particular individual. Special sensors that monitor Wi-Fi frequencies can be placed around a city to gather data that can later be used to identify patterns in the traffic flows. We present a set of filters that can be used to minimize the amount of data needed for processing and without negatively impacting the result or the information that can be extracted from this data. Part of the filters we present can be deployed at the sensor level, making the entire system more scalable, while a different part can be executed before data processing thus enabling real time information extraction and a broader temporal and spatial range for data analysis. Some of these filters are particular to Wi-Fi but some of them can be applied to any detection system

Proximity Graphs for Crowd Movement Sensors

Sensors are now common, they span over different applications, different purposes and some over large geospatial areas. Most data produced by these sensors needs to be linked to the physical location of the sensor itself. By using the location of a sensor we can construct (mathematically) proximity graphs that have the sensors as nodes. These graphs have a wide variety of applications including visualization, packet routing, and spatial data analysis. We consider a sensor network that measures detections of WiFi packets transmitted by devices, such as smartphones. One important feature of sensors is given by the range in which they can gather data. Algorithms that build proximity graphs do not take this radius into account. We present an approach to building proximity graph that takes sensor position and radius as input. Our goal is to construct a graph that contains edges between pairs of sensors that are correlated to crowd movements, reflecting paths that individuals are likely to take. Because we are considering crowd movement, it gives us the unique opportunity to construct graphs that show the connections between sensors using consecutive detections of the same device. We show that our approach is better than ones that are based on the positioning of sensors only

Nice is Different than Good: Longitudinal Communicative Effects of Realistic and Cartoon Avatars in Real Mixed Reality Work Meetings

We report a within-subjects study of the effect of realistic and cartoon avatars on communication, task satisfaction, and perceived sense of presence in mixed reality meetings. For 2 − 3 weeks, six groups of co-workers (14 people) held a recurring real work meeting using Microsoft HoloLens2 devices. Each person embodied a personalised full-body avatar with a realistic face and another with a cartoon face. Half the groups started in the realistic condition and the other half started in the cartoon condition; all groups switched conditions half-way. Initial results show that, overall, participants found the realistic avatars’ nonverbal behaviour more appropriate for the interaction and more useful for understanding their colleagues compared to the cartoon one. Regarding the results over time, we identify different insights for cartoon and realistic avatars based on the type of avatar was embodied first. We discuss the implications of these results for mixed and virtual reality meetings

More than buttons on controllers: engaging social interactions in narrative VR games through social attitudes detection

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program agents or non-player characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We present a collaborative work between two game studios (Maze Theory and Dream Reality Interactive) and academia to develop an immersive machine learning (ML) pipeline for detecting social engagement. Here we introduce the motivation and the methodology of the immersive ML pipeline, then we cover the motivation for the industry-academia collaboration, how it progressed, the implications of joined work on the industry and reflective insights on the collaboration. Overall, we highlight the industry-academia collaborative work on an immersive ML pipeline for detecting social engagement. We demonstrate how creatives could use ML and VR to expand their ability to design more engaging commercial games

Direct Gaze Triggers Higher Frequency of Gaze Change: An Automatic Analysis of Dyads in Unstructured Conversation

Nonverbal cues have multiple roles in social encounters, with gaze behaviour facilitating interactions and conversational flow. In this work, we explore the conversation dynamics in dyadic settings in a free-flow discussion. Using automatic analysis (rather than manual labelling), we investigate how the gaze behaviour of one person is related to how much the other person changes their gaze (frequency in gaze change) and what their gaze target is (direct or avert gaze). Our results show that when one person is looked at they change their gaze direction with a higher frequency compared to when they are not looked at. They also tend to maintain a direct gaze to the other person when they are not looked at

What caging force cells feel in 3D hydrogels: a rheological perspective

It is established that the mechanical properties of hydrogels control the fate of (stem) cells. However, despite its importance, a one-to-one correspondence between gels' stiffness and cell behaviour is still missing from literature. In this work, the viscoelastic properties of Poly(ethylene-glycol) (PEG)-based hydrogels - broadly used in 3D cell cultures and whose mechanical properties can be tuned to resemble those of different biological tissues - are investigated by means of rheological measurements performed at different length scales. When compared with literature values, the outcomes of this work reveal that conventional bulk rheology measurements may overestimate the stiffness of hydrogels by up to an order of magnitude. It is demonstrated that this apparent stiffening is caused by an induced 'tensional state' of the gel network, due to the application of a compressional normal force during measurements. Moreover, it is shown that the actual stiffness of the hydrogels is instead accurately determined by means of passive-video-particle-tracking (PVPT) microrheology measurements, which are inherently performed at cells length scales and in absence of any externally applied force. These results underpin a methodology for measuring the linear viscoelastic properties of hydrogels that are representative of the mechanical constraints felt by cells in 3D hydrogel cultures

Pervaporation of Aqueous Ethanol Solutions Through Pure and Composite Cellulose Membranes

A procedure for synthesis of pure and composite membranes based on cellulose dissolved in NaOH-urea/ thiourea solutions was developed. The phase inversion method was employed for cellulose solution conversion to supported and non-supported membranes. The use of tetraethyl orthosilicate (TEOS) as precursor in synthesis of composite cellulose membranes produced significant changes in their structure and pervaporation behaviour. The obtained membranes were tested for the pervaporation of ethanol-water system. Pervaporation performances, which were evaluated in terms of total permeate flux, separation factor and pervaporation separation index, strongly depended on TEOS loading, ethanol concentration and operation temperature