Multimodality in VR: A survey

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer

Natural Walking in Virtual Reality:A Review

Recent technological developments have finally brought virtual reality (VR) out of the laboratory and into the hands of developers and consumers. However, a number of challenges remain. Virtual travel is one of the most common and universal tasks performed inside virtual environments, yet enabling users to navigate virtual environments is not a trivial challenge—especially if the user is walking. In this article, we initially provide an overview of the numerous virtual travel techniques that have been proposed prior to the commercialization of VR. Then we turn to the mode of travel that is the most difficult to facilitate, that is, walking. The challenge of providing users with natural walking experiences in VR can be divided into two separate, albeit related, challenges: (1) enabling unconstrained walking in virtual worlds that are larger than the tracked physical space and (2) providing users with appropriate multisensory stimuli in response to their interaction with the virtual environment. In regard to the first challenge, we present walking techniques falling into three general categories: repositioning systems, locomotion based on proxy gestures, and redirected walking. With respect to multimodal stimuli, we focus on how to provide three types of information: external sensory information (visual, auditory, and cutaneous), internal sensory information (vestibular and kinesthetic/proprioceptive), and efferent information. Finally, we discuss how the different categories of walking techniques compare and discuss the challenges still facing the research community.</jats:p

Investigating Spatial Memory and Navigation in Developmental Amnesia: Evidence from a Google Street View Paradigm, Mental Navigation Tasks, and Route Descriptions

This dissertation examined the integrity of spatial representations of extensively travelled environments in developmental amnesia, thereby elucidating the role of the hippocampus in forming and retrieving spatial memories that enable flexible navigation. Previous research using mental navigation tasks found that developmental amnesic case H.C., an individual with atypical hippocampal development, could accurately estimate distance and direction between landmarks, but her representation of her environment was fragmented, inflexible, and lacked detail (Rosenbaum, Cassidy, & Herdman, 2015). Study 1 of this dissertation examined H.C.s spatial memory of her home environment using an ecologically valid virtual reality paradigm based on Google Street View. H.C. and control participants virtually navigated routes of varying familiarity within their home environment. To examine whether flexible navigation requires the hippocampus, participants also navigated familiar routes that had been mirror-reversed. H.C. performed similarly to control participants on all route conditions, suggesting that spatial learning of frequently travelled environments can occur despite compromised hippocampal system function. H.C.s unexpected ability to successfully navigate mirror-reversed routes might reflect the accumulation of spatial knowledge of her environment over the 6 years since she was first tested with mental navigation tasks. As such, Study 2 investigated how spatial representations of extensively travelled environments change over time in developmental amnesia by re-testing H.C. on mental navigation tasks 8 years later. H.C. continued to draw sketch maps that lacked cohesiveness and detail and had difficulty sequencing landmarks and generating detours on a blocked route task, suggesting that her overall representation of the environment did not improve over 8 years. Study 3 thoroughly examined the integrity of H.C.s detailed representation of the environment using a route description task. H.C. accurately described perceptual features of landmarks along a known route, but provided inaccurate information regarding the spatial relations of landmarks, resulting in a fragmented mental representation of the route. Taken together, these results contribute meaningfully to our current understanding of the integrity of spatial representations of extensively travelled environments in developmental amnesia. Non-spatial factors that could influence performance on navigation and spatial memory tasks are discussed, as is the impact of these results on theories of hippocampal function

Cycling as Reading a Cityscape: A Phenomenological Approach to Interface-Shaped Perception

This essay attempts to assess whether the perceptual issues posed by the contemporary interface culture, and the constant attitude shift demanded by the new media between the “natural” and the “as if” modes, might be considered a significant challenge for phenomenological aesthetics as understood in terms of Merleau-Ponty’s phenomenology of perception. To demonstrate how the use of a particular interface profoundly shapes the form and structure of an activity as well as enabling perception of a particular kind, the author does not focus directly on the state-of-the-art smart interfaces, but describes the experience of cycling in a large city, with the interface in the form of the bicycle upgraded with an imagined ride simulator. While the former enables a very particular entrance into the world of perception, shaped by its moderate speed and detachment from the ground, the latter enables techno-shaped perception in the “as if” screenic mode. The experience described raises questions concerning the kinaesthetic, proprioceptive and motor features contributing to the cyclist’s mobile perception, as well as pointing to issues related to the reading of the city’s network as a particular spatial configuration generated by the cyclist’s realtime activity. This is the space-time-event-ridescape maintained and modified by the corporeal act of cycling. The spatiality of such a ride does not presume the notion of a space that contains the cyclist, but builds on notions of being-in-the-ridescape (as a kind of cityscape), in terms not only of full corporeal and mental engagement, but also of bodily literacy. The reading of the cityscape enabled by the combination of two interfaces, the bicycle and the ride simulator, is discussed in relation to de Certeau’s account of pedestrian (walking) experience in a big city, his distinction between strategies and tactics, and the notion that each cyclist contributes a novel story to ridetext, which is viewed not as an aesthetic object but as the production of puzzles for the rider to solve. The paper cocludes by questioning the capacity of phenomenology to accommodate the contemporary phenomenon of a “mixed” or “augmented reality” either in concept or in relation to the demands of the phenomenological reduction and the ends of the epoché. Indo-Pacific Journal of Phenomenology, Volume 10, Edition 2, October 2010: 61-7

The Aha! Experience of Spatial Reorientation

The experience of spatial re-orientation is investigated as an instance of the wellknown phenomenon of the Aha! moment. The research question is: What are the visuospatial conditions that are most likely to trigger the spatial Aha! experience? The literature suggests that spatial re-orientation relies mainly on the geometry of the environment and a visibility graph analysis is used to quantify the visuospatial information. Theories from environmental psychology point towards two hypotheses. The Aha! experience may be triggered by a change in the amount of visual information, described by the isovist properties of area and revelation, or by a change in the complexity of the visual information associated with the isovist properties of clustering coefficient and visual control. Data from participants’ exploratory behaviour and EEG recordings are collected during wayfinding in virtual reality urban environments. Two types of events are of interest here: (a) sudden changes of the visuospatial information preceding subjects' response to investigate changes in EEG power; and (b) participants brain dynamics (Aha! effect) just before the response to examine differences in isovist values at this location. Research on insights, time-frequency analysis of the P3 component and findings from navigation and orientation studies suggest that the spatial Aha! experience may be reflected by: a parietal alpha power decrease associated with the switch of the representation and a frontocentral theta increase indexing spatial processing during decision-making. Single-trial time-frequency analysis is used to classify trials into two conditions based on the alpha/theta power differences between a 3s time-period before participants’ response and a time-period of equal duration before that. Behavioural results show that participants are more likely to respond at locations with low values of clustering coefficient and high values of visual control. The EEG analysis suggests that the alpha decrease/theta increase condition occurs at locations with significantly lower values of clustering coefficient and higher values of visual control. Small and large decreases in clustering coefficient, just before the response, are associated with significant differences in delta/theta power. The values of area and revelation do not show significant differences. Both behavioural and EEG results suggest that the Aha! experience of re-orientation is more likely to be triggered by a change in the complexity of the visual-spatial environment rather than a change in the amount, as measured by the relevant isovist properties

LoCoMoTe – a framework for classification of natural locomotion in VR by task, technique and modality

Virtual reality (VR) research has provided overviews of locomotion techniques, how they work, their strengths and overall user experience. Considerable research has investigated new methodologies, particularly machine learning to develop redirection algorithms. To best support the development of redirection algorithms through machine learning, we must understand how best to replicate human navigation and behaviour in VR, which can be supported by the accumulation of results produced through live-user experiments. However, it can be difficult to identify, select and compare relevant research without a pre-existing framework in an ever-growing research field. Therefore, this work aimed to facilitate the ongoing structuring and comparison of the VR-based natural walking literature by providing a standardised framework for researchers to utilise. We applied thematic analysis to study methodology descriptions from 140 VR-based papers that contained live-user experiments. From this analysis, we developed the LoCoMoTe framework with three themes: navigational decisions, technique implementation, and modalities. The LoCoMoTe framework provides a standardised approach to structuring and comparing experimental conditions. The framework should be continually updated to categorise and systematise knowledge and aid in identifying research gaps and discussions