Eye tracking and avatar-mediated communication in immersive collaborative virtual environments

The research presented in this thesis concerns the use of eye tracking to both enhance and understand avatar-mediated communication (AMC) performed by users of immersive collaborative virtual environment (ICVE) systems. AMC, in which users are embodied by graphical humanoids within a shared virtual environment (VE), is rapidly emerging as a prevalent and popular form of remote interaction. However, compared with video-mediated communication (VMC), which transmits interactants’ actual appearance and behaviour, AMC fails to capture, transmit, and display many channels of nonverbal communication (NVC). This is a significant hindrance to the medium’s ability to support rich interpersonal telecommunication. In particular, oculesics (the communicative properties of the eyes), including gaze, blinking, and pupil dilation, are central nonverbal cues during unmediated social interaction. This research explores the interactive and analytical application of eye tracking to drive the oculesic animation of avatars during real-time communication, and as the primary method of experimental data collection and analysis, respectively. Three distinct but interrelated questions are addressed. First, the thesis considers the degree to which quality of communication may be improved through the use of eye tracking, to increase the nonverbal, oculesic, information transmitted during AMC. Second, the research asks whether users engaged in AMC behave and respond in a socially realistic manner in comparison with VMC. Finally, the degree to which behavioural simulations of oculesics can both enhance the realism of virtual humanoids, and complement tracked behaviour in AMC, is considered. These research questions were investigated over a series of telecommunication experiments investigating scenarios common to computer supported cooperative work (CSCW), and a further series of experiments investigating behavioural modelling for virtual humanoids. The first, exploratory, telecommunication experiment compared AMC with VMC in a three-party conversational scenario. Results indicated that users employ gaze similarly when faced with avatar and video representations of fellow interactants, and demonstrated how interaction is influenced by the technical characteristics and limitations of a medium. The second telecommunication experiment investigated the impact of varying methods of avatar gaze control on quality of communication during object-focused multiparty AMC. The main finding of the experiment was that quality of communication is reduced when avatars demonstrate misleading gaze behaviour. The final telecommunication study investigated truthful and deceptive dyadic interaction in AMC and VMC over two closely-related experiments. Results from the first experiment indicated that users demonstrate similar oculesic behaviour and response in both AMC and VMC, but that psychological arousal is greater following video-based interaction. Results from the second experiment found that the use of eye tracking to drive the oculesic behaviour of avatars during AMC increased the richness of NVC to the extent that more accurate estimation of embodied users’ states of veracity was enabled. Rather than directly investigating AMC, the second series of experiments addressed behavioural modelling of oculesics for virtual humanoids. Results from the these experiments indicated that oculesic characteristics are highly influential to the perceived realism of virtual humanoids, and that behavioural models are able to complement the use of eye tracking in AMC. The research presented in this thesis explores AMC and eye tracking over a range of collaborative and perceptual studies. The overall conclusion is that eye tracking is able to enhance AMC towards a richer medium for interpersonal telecommunication, and that users’ behaviour in AMC is no less socially ‘real’ than that demonstrated in VMC. However, there are distinct differences between the two communication mediums, and the importance of matching the characteristics of a planned communication with those of the medium itself is critical

A mixed reality telepresence system for collaborative space operation

This paper presents a Mixed Reality system that results from the integration of a telepresence system and an application to improve collaborative space exploration. The system combines free viewpoint video with immersive projection technology to support non-verbal communication, including eye gaze, inter-personal distance and facial expression. Importantly, these can be interpreted together as people move around the simulation, maintaining natural social distance. The application is a simulation of Mars, within which the collaborators must come to agreement over, for example, where the Rover should land and go. The first contribution is the creation of a Mixed Reality system supporting contextualization of non-verbal communication. Tw technological contributions are prototyping a technique to subtract a person from a background that may contain physical objects and/or moving images, and a light weight texturing method for multi-view rendering which provides balance in terms of visual and temporal quality. A practical contribution is the demonstration of pragmatic approaches to sharing space between display systems of distinct levels of immersion. A research tool contribution is a system that allows comparison of conventional authored and video based reconstructed avatars, within an environment that encourages exploration and social interaction. Aspects of system quality, including the communication of facial expression and end-to-end latency are reported

Communication in Immersive Social Virtual Reality: A Systematic Review of 10 Years' Studies

As virtual reality (VR) technologies have improved in the past decade, more research has investigated how they could support more effective communication in various contexts to improve collaboration and social connectedness. However, there was no literature to summarize the uniqueness VR provided and put forward guidance for designing social VR applications for better communication. To understand how VR has been designed and used to facilitate communication in different contexts, we conducted a systematic review of the studies investigating communication in social VR in the past ten years by following the PRISMA guidelines. We highlight current practices and challenges and identify research opportunities to improve the design of social VR to better support communication and make social VR more accessible.Comment: Chinese CHI '22: The Tenth International Symposium of Chinese CHI (Chinese CHI 2022

Effectiveness of Social Virtual Reality

A lot of work in social virtual reality, including our own group's, has focused on effectiveness of specific social behaviours such as eye-gaze, turn taking, gestures and other verbal and non-verbal cues. We have built upon these to look at emergent phenomena such as co-presence, leadership and trust. These give us good information about the usability issues of specific social VR systems, but they don't give us much information about the requirements for such systems going forward. In this short paper we discuss how we are broadening the scope of our work on social systems, to move out of the laboratory to more ecologically valid situations and to study groups using social VR for longer periods of time

Investigating Social Presence and Communication with Embodied Avatars in Room-Scale Virtual Reality

Submission includes video.Room-scale virtual reality (VR) holds great potential as a medium for communication and collaboration in remote and same-time, same-place settings. Related work has established that movement realism can create a strong sense of social presence, even in the absence of photorealism. Here, we explore the noteworthy attributes of communicative interaction using embodied minimal avatars in room-scale VR in the same-time, same-place setting. Our system is the first in the research community to enable this kind of interaction, as far as we are aware. We carried out an experiment in which pairs of users performed two activities in contrasting variants: VR vs. face-to-face (F2F), and 2D vs. 3D. Objective and subjective measures were used to compare these, including motion analysis, electrodermal activity, questionnaires, retrospective think-aloud protocol, and interviews. On the whole, participants communicated effectively in VR to complete their tasks, and reported a strong sense of social presence. The system's high fidelity capture and display of movement seems to have been a key factor in supporting this. Our results confirm some expected shortcomings of VR compared to F2F, but also some non-obvious advantages. The limited anthropomorphic properties of the avatars presented some difficulties, but the impact of these varied widely between the activities. In the 2D vs. 3D comparison, the basic affordance of freehand drawing in 3D was new to most participants, resulting in novel observations and open questions. We also present methodological observations across all conditions concerning the measures that did and did not reveal differences between conditions, including unanticipated properties of the think-aloud protocol applied to VR

An Exploration of Theatre Rehearsals in Social Virtual Reality

Virtual Reality (VR) offers potential for theatre makers to rehearse remotely in settings which are uniquely immersive. In collaboration with a major drama school in the United Kingdom, a longitudinal diary study was completed to examine the utility of consumer-grade VR for theatre rehearsals. Utilising commonly affordable headsets and general-purpose Social VR applications, 10 experienced students (2 directors, 8 actors) rehearsed scenes in VR over 3 weeks, before performing them in person. Participants detailed their experiences in diary logs and interviews, expressing the ability to work through spatial arrangements (blocking) as a full body avatar to be positively beneficial. Limitations included the absence of facial expressions and gestural nuance. Our overarching conclusion is that low-tech VR can be a useful aid in theatre rehearsals and early stages of production. In conclusion we outline design recommendations for a) using VR in theatre production and b) research and development of Social VR.</p

An Exploration of Theatre Rehearsals in Social Virtual Reality

Virtual Reality (VR) offers potential for theatre makers to rehearse remotely in settings which are uniquely immersive. In collaboration with a major drama school in the United Kingdom, a longitudinal diary study was completed to examine the utility of consumer-grade VR for theatre rehearsals. Utilising commonly affordable headsets and general-purpose Social VR applications, 10 experienced students (2 directors, 8 actors) rehearsed scenes in VR over 3 weeks, before performing them in person. Participants detailed their experiences in diary logs and interviews, expressing the ability to work through spatial arrangements (blocking) as a full body avatar to be positively beneficial. Limitations included the absence of facial expressions and gestural nuance. Our overarching conclusion is that low-tech VR can be a useful aid in theatre rehearsals and early stages of production. In conclusion we outline design recommendations for a) using VR in theatre production and b) research and development of Social VR.</p

Analysis domain model for shared virtual environments

The field of shared virtual environments, which also encompasses online games and social 3D environments, has a system landscape consisting of multiple solutions that share great functional overlap. However, there is little system interoperability between the different solutions. A shared virtual environment has an associated problem domain that is highly complex raising difficult challenges to the development process, starting with the architectural design of the underlying system. This paper has two main contributions. The first contribution is a broad domain analysis of shared virtual environments, which enables developers to have a better understanding of the whole rather than the part(s). The second contribution is a reference domain model for discussing and describing solutions - the Analysis Domain Model