Multimodality in VR: A survey

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer

Human factors in instructional augmented reality for intravehicular spaceflight activities and How gravity influences the setup of interfaces operated by direct object selection

In human spaceflight, advanced user interfaces are becoming an interesting mean to facilitate human-machine interaction, enhancing and guaranteeing the sequences of intravehicular space operations. The efforts made to ease such operations have shown strong interests in novel human-computer interaction like Augmented Reality (AR). The work presented in this thesis is directed towards a user-driven design for AR-assisted space operations, iteratively solving issues arisen from the problem space, which also includes the consideration of the effect of altered gravity on handling such interfaces.Auch in der bemannten Raumfahrt steigt das Interesse an neuartigen Benutzerschnittstellen, um nicht nur die Mensch-Maschine-Interaktion effektiver zu gestalten, sondern auch um einen korrekten Arbeitsablauf sicherzustellen. In der Vergangenheit wurden wiederholt Anstrengungen unternommen, Innenbordarbeiten mit Hilfe von Augmented Reality (AR) zu erleichtern. Diese Arbeit konzentriert sich auf einen nutzerorientierten AR-Ansatz, welcher zum Ziel hat, die Probleme schrittweise in einem iterativen Designprozess zu lösen. Dies erfordert auch die Berücksichtigung veränderter Schwerkraftbedingungen

Hybridizing 3-dimensional multiple object tracking with neurofeedback to enhance preparation, performance, and learning

Le vaste domaine de l’amélioration cognitive traverse les applications comportementales, biochimiques et physiques. Aussi nombreuses sont les techniques que les limites de ces premières : des études de pauvre méthodologie, des pratiques éthiquement ambiguës, de faibles effets positifs, des effets secondaires significatifs, des couts financiers importants, un investissement de temps significatif, une accessibilité inégale, et encore un manque de transfert. L’objectif de cette thèse est de proposer une méthode novatrice d’intégration de l’une de ces techniques, le neurofeedback, directement dans un paradigme d’apprentissage afin d’améliorer la performance cognitive et l’apprentissage. Cette thèse propose les modalités, les fondements empiriques et des données à l’appui de ce paradigme efficace d’apprentissage ‘bouclé’. En manipulant la difficulté dans une tâche en fonction de l’activité cérébrale en temps réel, il est démontré que dans un paradigme d’apprentissage traditionnel (3-dimentional multiple object tracking), la vitesse et le degré d’apprentissage peuvent être améliorés de manière significative lorsque comparés au paradigme traditionnel ou encore à un groupe de contrôle actif. La performance améliorée demeure observée même avec un retrait du signal de rétroaction, ce qui suggère que les effets de l’entrainement amélioré sont consolidés et ne dépendent pas d’une rétroaction continue. Ensuite, cette thèse révèle comment de tels effets se produisent, en examinant les corrélés neuronaux des états de préparation et de performance à travers les conditions d’état de base et pendant la tâche, de plus qu’en fonction du résultat (réussite/échec) et de la difficulté (basse/moyenne/haute vitesse). La préparation, la performance et la charge cognitive sont mesurées via des liens robustement établis dans un contexte d’activité cérébrale fonctionnelle mesurée par l’électroencéphalographie quantitative. Il est démontré que l’ajout d’une assistance- à-la-tâche apportée par la fréquence alpha dominante est non seulement appropriée aux conditions de ce paradigme, mais influence la charge cognitive afin de favoriser un maintien du sujet dans sa zone de développement proximale, ce qui facilite l’apprentissage et améliore la performance. Ce type de paradigme d’apprentissage peut contribuer à surmonter, au minimum, un des limites fondamentales du neurofeedback et des autres techniques d’amélioration cognitive : le manque de transfert, en utilisant une méthode pouvant être intégrée directement dans le contexte dans lequel l’amélioration de la performance est souhaitée.The domain of cognitive enhancement is vast, spanning behavioral, biochemical and physical applications. The techniques are as numerous as are the limitations: poorly conducted studies, ethically ambiguous practices, limited positive effects, significant side-effects, high financial costs, significant time investment, unequal accessibility, and lack of transfer. The purpose of this thesis is to propose a novel way of integrating one of these techniques, neurofeedback, directly into a learning context in order to enhance cognitive performance and learning. This thesis provides the framework, empirical foundations, and supporting evidence for a highly efficient ‘closed-loop’ learning paradigm. By manipulating task difficulty based on a measure of cognitive load within a classic learning scenario (3-dimentional multiple object tracking) using real-time brain activity, results demonstrate that over 10 sessions, speed and degree of learning can be substantially improved compared with a classic learning system or an active sham-control group. Superior performance persists even once the feedback signal is removed, which suggests that the effects of enhanced training are consolidated and do not rely on continued feedback. Next, this thesis examines how these effects occur, exploring the neural correlates of the states of preparedness and performance across baseline and task conditions, further examining correlates related to trial results (correct/incorrect) and task difficulty (slow/medium/fast speeds). Cognitive preparedness, performance and load are measured using well-established relationships between real-time quantified brain activity as measured by quantitative electroencephalography. It is shown that the addition of neurofeedback-based task assistance based on peak alpha frequency is appropriate to task conditions and manages to influence cognitive load, keeping the subject in the zone of proximal development more often, facilitating learning and improving performance. This type of learning paradigm could contribute to overcoming at least one of the fundamental limitations of neurofeedback and other cognitive enhancement techniques : a lack of observable transfer effects, by utilizing a method that can be directly integrated into the context in which improved performance is sought

Differentiation of the Causal Characteristics and Influences of Virtual Reality and the Effects on Learning at a Science Exhibit

Within the context of the informal science center, exhibits are the main interface for public learning. Essential to the success of a science center is how well exhibits model effective strategies for learning. Virtual Reality (VR) technology with its flexible, adaptive, multimedia, and immersive-learning capabilities is emerging for use by science centers in exhibits; however, research on learning in virtual environments at exhibits is scarce. To support the future development of VR science exhibits it is critical to investigate VR\u27s pedagogical value and effects on science learning. Research investigated the Smoke & Mirrors VR exhibit at the Reuben H. Fleet Science Center in San Diego, California. Inquiry focused on the interplay between elements of the exhibit\u27s design, assessing the separate and interactive effects of visual imagery, moving images, sound, narration, and interactive tools to differentiate the causal characteristics and influences that enhanced and detracted from learning. Case study methodology was employed utilizing visitor observations and interviews with 14 participants. Findings indicated that realistic visual elements with text were the primary sources of content learning; however, positive results were limited to only a few participants. High cognitive load due to interactive tools; instructional design; and movement of visual images were found to be significant detracting characteristics of participant learning. Other characteristics and influences of VR were also found that directly effected learning. Research results will inform the forthcoming design of a new VR exhibit at the Reuben H. Fleet Science Center and to the design and development of future VR exhibits at informal science centers. A prior brief mixed-methods evaluation of Smoke & Mirrors was conducted in 2003, contributing background to the study and its future implications and strategies

Visual Perception and Cognition in Image-Guided Intervention

Surgical image visualization and interaction systems can dramatically affect the efficacy and efficiency of surgical training, planning, and interventions. This is even more profound in the case of minimally-invasive surgery where restricted access to the operative field in conjunction with limited field of view necessitate a visualization medium to provide patient-specific information at any given moment. Unfortunately, little research has been devoted to studying human factors associated with medical image displays and the need for a robust, intuitive visualization and interaction interfaces has remained largely unfulfilled to this day. Failure to engineer efficient medical solutions and design intuitive visualization interfaces is argued to be one of the major barriers to the meaningful transfer of innovative technology to the operating room. This thesis was, therefore, motivated by the need to study various cognitive and perceptual aspects of human factors in surgical image visualization systems, to increase the efficiency and effectiveness of medical interfaces, and ultimately to improve patient outcomes. To this end, we chose four different minimally-invasive interventions in the realm of surgical training, planning, training for planning, and navigation: The first chapter involves the use of stereoendoscopes to reduce morbidity in endoscopic third ventriculostomy. The results of this study suggest that, compared with conventional endoscopes, the detection of the basilar artery on the surface of the third ventricle can be facilitated with the use of stereoendoscopes, increasing the safety of targeting in third ventriculostomy procedures. In the second chapter, a contour enhancement technique is described to improve preoperative planning of arteriovenous malformation interventions. The proposed method, particularly when combined with stereopsis, is shown to increase the speed and accuracy of understanding the spatial relationship between vascular structures. In the third chapter, an augmented-reality system is proposed to facilitate the training of planning brain tumour resection. The results of our user study indicate that the proposed system improves subjects\u27 performance, particularly novices\u27, in formulating the optimal point of entry and surgical path independent of the sensorimotor tasks performed. In the last chapter, the role of fully-immersive simulation environments on the surgeons\u27 non-technical skills to perform vertebroplasty procedure is investigated. Our results suggest that while training surgeons may increase their technical skills, the introduction of crisis scenarios significantly disturbs the performance, emphasizing the need of realistic simulation environments as part of training curriculum

Eyewear Computing \u2013 Augmenting the Human with Head-Mounted Wearable Assistants

The seminar was composed of workshops and tutorials on head-mounted eye tracking, egocentric vision, optics, and head-mounted displays. The seminar welcomed 30 academic and industry researchers from Europe, the US, and Asia with a diverse background, including wearable and ubiquitous computing, computer vision, developmental psychology, optics, and human-computer interaction. In contrast to several previous Dagstuhl seminars, we used an ignite talk format to reduce the time of talks to one half-day and to leave the rest of the week for hands-on sessions, group work, general discussions, and socialising. The key results of this seminar are 1) the identification of key research challenges and summaries of breakout groups on multimodal eyewear computing, egocentric vision, security and privacy issues, skill augmentation and task guidance, eyewear computing for gaming, as well as prototyping of VR applications, 2) a list of datasets and research tools for eyewear computing, 3) three small-scale datasets recorded during the seminar, 4) an article in ACM Interactions entitled \u201cEyewear Computers for Human-Computer Interaction\u201d, as well as 5) two follow-up workshops on \u201cEgocentric Perception, Interaction, and Computing\u201d at the European Conference on Computer Vision (ECCV) as well as \u201cEyewear Computing\u201d at the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)

Multimodality in {VR}: {A} Survey

Virtual reality has the potential to change the way we create and consume content in our everyday life. Entertainment, training, design and manufacturing, communication, or advertising are all applications that already benefit from this new medium reaching consumer level. VR is inherently different from traditional media: it offers a more immersive experience, and has the ability to elicit a sense of presence through the place and plausibility illusions. It also gives the user unprecedented capabilities to explore their environment, in contrast with traditional media. In VR, like in the real world, users integrate the multimodal sensory information they receive to create a unified perception of the virtual world. Therefore, the sensory cues that are available in a virtual environment can be leveraged to enhance the final experience. This may include increasing realism, or the sense of presence; predicting or guiding the attention of the user through the experience; or increasing their performance if the experience involves the completion of certain tasks. In this state-of-the-art report, we survey the body of work addressing multimodality in virtual reality, its role and benefits in the final user experience. The works here reviewed thus encompass several fields of research, including computer graphics, human computer interaction, or psychology and perception. Additionally, we give an overview of different applications that leverage multimodal input in areas such as medicine, training and education, or entertainment; we include works in which the integration of multiple sensory information yields significant improvements, demonstrating how multimodality can play a fundamental role in the way VR systems are designed, and VR experiences created and consumed