The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment

Objective and subjective measures of performance in virtual reality environments increase as more sensory cues are delivered and as simulation fidelity increases. Some cues (colour or sound) are easier to present than others (object weight, vestibular cues) so that substitute cues can be used to enhance informational content in a simulation at the expense of simulation fidelity. This study evaluates how substituting cues in one modality by alternative cues in another modality affects subjective and objective performance measures in a highly immersive virtual reality environment. Participants performed a wheel change in a virtual reality (VR) environment. Auditory, haptic and visual cues, signalling critical events in the simulation, were manipulated in a factorial design. Subjective ratings were recorded via questionnaires. The time taken to complete the task was used as an objective performance measure. The results show that participants performed best and felt an increased sense of immersion and involvement, collectively referred to as 'presence', when substitute multimodal sensory feedback was provided. Significant main effects of audio and tactile cues on task performance and on participants' subjective ratings were found. A significant negative relationship was found between the objective (overall completion times) and subjective (ratings of presence) performance measures. We conclude that increasing informational content, even if it disrupts fidelity, enhances performance and user's overall experience. On this basis we advocate the use of substitute cues in VR environments as an efficient method to enhance performance and user experience

Human Performance and Cognitive Workload in Haptic, Audio and Visual Environments

The ability to efficiently perform a task in a human-in-the-loop system and in multi-sensory virtual environments is highly dependent on the type of sensory feedback the operator is receiving and the amount of workload the operator is exposed to. Despite the vast amount of research on Collaborative Virtual Environments (CVEs) and Human Machine Interactions (HMIs), little is known about what type of feedback increases the performance of a human operator and what type of sensory feedback minimizes the amount of workload the operator is exposed to. While individual differences influence human performance outcomes, the physiological processes a human being set the fundamental guidelines for assessing human performance. The purpose of this study is to evaluate the performance of participants for a combination of sensory two feedback modes (audio-visual, haptic-visual or audio-haptic) in a primary task to find the optimum feedback model for CVE and HMI applications. A concurrent secondary task is also designed to evaluate workload of each feedback mode (audio, haptic or visual) and the effect of different levels of workload on task completion time and task accuracy. For example, a car driver performs a primary task by steering the car in the correct direction. A secondary task, in the same context, would be monitoring the fuel level or checking the speed limit. In the primary task, participants are required to press a virtual button from a set of three (right button, left button or up button). The secondary task evaluates the amount of workload the participant is exposed to in three different feedback modes (haptic, audio or visual). Each participant is required to recognize a Morse code. In this study, participants perform three trials. In first trial, participants perform one task the primary task alone. In the second and third trials, participants perform the primary task and the secondary task concurrently. The primary task evaluates human performance and includes combined sensory modalities as a feedback mode (audio-visual, haptic-visual or audio-haptic). The time it takes the participant to press the virtual button (primary task response time), the number of correct button presses (primary task accuracy), the time it takes the participant to recognize the Morse code (secondary task response time) and the number of the correct codes (secondary task accuracy) are all collected. In addition, NASA Task Load Index (TLX) questionnaire is used after each trial to assess the subjective performance and subjective workload of participants. The data collected is tested for normality using Lilliefors test, filtered using Grubb’s test to eliminate outlying data and analyzed using one-way ANOVA and multiple two-sample t-tests. A Tukey HSD is also used to show the differences between experimental conditions. The results of this study indicate that the hypothesis that all combinations of feedback provide the same performance can be rejected for the primary task response time. For instance, the results show that the there is a difference in response time between the audio-haptic and the audio-visual feedback modes in the first, second and third trials. The results of this study also indicate that the hypothesis that all sensory feedback modes provide the same workload can be rejected for the secondary task accuracy. Results show that there is a difference between haptic and auditory conditions and shows that visual condition has a lower accuracy than the other feedback modes

A Systematic Review of Social Presence: Definition, Antecedents, and Implications

Social presence, or the feeling of being there with a “real” person, is a crucial component of interactions that take place in virtual reality. This paper reviews the concept, antecedents, and implications of social presence, with a focus on the literature regarding the predictors of social presence. The article begins by exploring the concept of social presence, distinguishing it from two other dimensions of presence—telepresence and self-presence. After establishing the definition of social presence, the article offers a systematic review of 233 separate findings identified from 152 studies that investigate the factors (i.e., immersive qualities, contextual differences, and individual psychological traits) that predict social presence. Finally, the paper discusses the implications of heightened social presence and when it does and does not enhance one's experience in a virtual environment

The role of multisensory feedback in the objective and subjective evaluations of fidelity in virtual reality environments.

The use of virtual reality in academic and industrial research has been rapidly expanding in recent years therefore evaluations of the quality and effectiveness of virtual environments are required. The assessment process is usually done through user evaluation that is being measured whilst the user engages with the system. The limitations of this method in terms of its variability and user bias of pre and post-experience have been recognised in the research literature. Therefore, there is a need to design more objective measures of system effectiveness that could complement subjective measures and provide a conceptual framework for the fidelity assessment in VR. There are many technological and perceptual factors that can influence the overall experience in virtual environments. The focus of this thesis was to investigate how multisensory feedback, provided during VR exposure, can modulate a user’s qualitative and quantitative experience in the virtual environment. In a series of experimental studies, the role of visual, audio, haptic and motion cues on objective and subjective evaluations of fidelity in VR was investigated. In all studies, objective measures of performance were collected and compared to the subjective measures of user perception. The results showed that the explicit evaluation of environmental and perceptual factors available within VR environments modulated user experience. In particular, the results shown that a user’s postural responses can be used as a basis for the objective measure of fidelity. Additionally, the role of augmented sensory cues was investigated during a manual assembly task. By recording and analysing the objective and subjective measures it was shown that augmented multisensory feedback modulated the user’s acceptability of the virtual environment in a positive manner and increased overall task performance. Furthermore, the presence of augmented cues mitigated the negative effects of inaccurate motion tracking and simulation sickness. In the follow up study, the beneficial effects of virtual training with augmented sensory cues were observed in the transfer of learning when the same task was performed in a real environment. Similarly, when the effects of 6 degrees of freedom motion cuing on user experience were investigated in a high fidelity flight simulator, the consistent findings between objective and subjective data were recorded. By measuring the pilot’s accuracy to follow the desired path during a slalom manoeuvre while perceived task demand was increased, it was shown that motion cuing is related to effective task performance and modulates the levels of workload, sickness and presence. The overall findings revealed that multisensory feedback plays an important role in the overall perception and fidelity evaluations of VR systems and as such user experience needs to be included when investigating the effectiveness of sensory feedback signals. Throughout this thesis it was consistently shown that subjective measures of user perception in VR are directly comparable to the objective measures of performance and therefore both should be used in order to obtain a robust results when investigating the effectiveness of VR systems. This conceptual framework can provide an effective method to study human perception, which can in turn provide a deeper understanding of the environmental and cognitive factors that can influence the overall user experience, in terms of fidelity requirements, in virtual reality environments