What aspects of realism and faithfulness are relevant to supporting non-verbal communication through 3D mediums

This thesis investigates what aspects of realism and faithfulness are relevant to supporting non-verbal communication through visual mediums. The mediums examined are 2D video, 3D computer graphics and video based 3D reconstruction. The latter is 3D CGI derived from multiple streams of 2D video. People’s ability to identify behaviour of primates through gross non-verbal communication is compared across 2D video and 3D CGI. Findings suggest 3D CGI performs equally well to 2D video for the identification of gross non-verbal behaviour, however user feedback points to a lack of understanding of intent. Secondly, ability to detect truthfulness in humans across 2D video and video based 3D reconstruction mediums is examined. Effort of doing this is measured by studying changes in level of oxygenation to the prefrontal cortex. Discussion links to the literature to propose that tendency to over trust is inversely proportional to the range of non-verbal resources communicated through a medium. It is suggested that perhaps this is because “tells” are hidden. The third study identifies that video based 3D reconstruction can successfully illustrate subtle facial muscle movements on a par with 2D video, but does identify issues with the display of lower facial detail, due to a reconstruction error called droop. It is hoped that the combination of these strands of research will help users, and application developers, make more informed decisions when selecting which type of virtual character to implement for a particular application therefore contributing to the fields of virtual characters and virtual environments/serious gaming, by giving readers a greater understanding of virtual characters ability to convey non-verbal behaviour

Investigating Person Identification in Security Settings with Virtual Reality

Person identification at airports requires the comparison of a passport photograph with its bearer. In psychology, this process is typically studied with static pairs of face photographs that require identity match (same person shown) versus mismatch (two different people) decisions, but this approach provides a limited proxy for studying how environment and social interaction factors affect this task. This thesis explores the feasibility of virtual reality (VR) as a solution to this problem, by examining the identity matching of avatars in a VR airport. In Chapter 2, facial photographs of real people are successfully rendered into VR avatars in a manner that preserves image and identity information (Experiments 1 to 3). Furthermore, identity matching of avatar pairs reflects similar cognitive processes to the matching of face photographs (Experiments 4 and 5), a pattern which holds when assessed in a VR airport (Experiments 6 and 7). Chapter 3 then examines whether a simulation of a passport control task in VR can provide a useful tool for selecting personnel for real-world tasks (Experiment 8). The classification of identity mismatches, the detection of which is of paramount importance in security settings, correlated across conventional laboratory face matching tests and the VR passport control task. Social interaction factors, such as body language, may further influence face matching performance, which was explored in Chapter 4. Whilst performance was unaffected when observers were not instructed explicitly to utilise body language (Experiments 9 and 10), when instructed body language enhanced detection of identity mismatches yet also increased false classification of matches (Experiments 11 to 13). This effect was driven by increased activity levels rather than body language that simply differed from normal behaviour, and occurred independently of individuals' face-matching ability (Experiment 14). This thesis concludes with a summary of how VR can open up many avenues for face-matching research, by facilitating the study of new environment and social interaction factors that may be relevant in real-world operational settings