An fMRI Study to Analyze Neural Correlates of Presence during Virtual Reality Experiences

[EN] In the field of virtual reality (VR), many efforts have been made to analyze presence, the sense of being in the virtual world. However, it is only recently that functional magnetic resonance imaging (fMRI) has been used to study presence during an automatic navigation through a virtual environment. In the present work, our aim was to use fMRI to study the sense of presence during a VR-free navigation task, in comparison with visualization of photographs and videos (automatic navigations through the same environment). The main goal was to analyze the usefulness of fMRI for this purpose, evaluating whether, in this context, the interaction between the subject and the environment is performed naturally, hiding the role of technology in the experience. We monitored 14 right-handed healthy females aged between 19 and 25 years. Frontal, parietal and occipital regions showed their involvement during free virtual navigation. Moreover, activation in the dorsolateral prefrontal cortex was also shown to be negatively correlated to sense of presence and the postcentral parietal cortex and insula showed a parametric increased activation according to the condition-related sense of presence, which suggests that stimulus attention and self-awareness processes related to the insula may be linked to the sense of presence.This study was funded by the Ministerio de Educación y Ciencia Spain, Project Game Teen (TIN2010-20187) and partially by projects Consolider-C (SEJ2006-14301/PSIC), ‘CIBER of Physiopathology of Obesity and Nutrition, an initiative of ISCIII’, the Excellence Research Program PROMETEO (Generalitat Valenciana. Conselleria de Educación, 2008-157) and the Consolider INGENIO program (CSD2007-00012). The work of Miriam Clemente was supported by the Generalitat Valenciana under a VALi+d Grant.Clemente Bellido, M.; Rey, B.; Rodríguez Pujadas, A.; Barros Loscertales, A.; Baños, RM.; Botella, C.; Alcañiz Raya, ML.... (2014). An fMRI Study to Analyze Neural Correlates of Presence during Virtual Reality Experiences. Interacting with Computers. 26(3):269-284. https://doi.org/10.1093/iwc/iwt037S269284263Aguirre, G. K., Detre, J. A., Alsop, D. C., & D’Esposito, M. (1996). The Parahippocampus Subserves Topographical Learning in Man. Cerebral Cortex, 6(6), 823-829. doi:10.1093/cercor/6.6.823Alcañiz, M., Rey, B., Tembl, J., & Parkhutik, V. (2009). A Neuroscience Approach to Virtual Reality Experience Using Transcranial Doppler Monitoring. Presence: Teleoperators and Virtual Environments, 18(2), 97-111. doi:10.1162/pres.18.2.97Amaro, E., & Barker, G. J. (2006). Study design in fMRI: Basic principles. Brain and Cognition, 60(3), 220-232. doi:10.1016/j.bandc.2005.11.009Astur, R. S., St. Germain, S. A., Baker, E. K., Calhoun, V., Pearlson, G. D., & Constable, R. T. (2005). fMRI Hippocampal Activity During a VirtualRadial Arm Maze. Applied Psychophysiology and Biofeedback, 30(3), 307-317. doi:10.1007/s10484-005-6385-zBaños, R. M., Botella, C., Garcia-Palacios, A., Villa, H., Perpiña, C., & Alcañiz, M. (2000). Presence and Reality Judgment in Virtual Environments: A Unitary Construct? CyberPsychology & Behavior, 3(3), 327-335. doi:10.1089/10949310050078760Baumann, S., Neff, C., Fetzick, S., Stangl, G., Basler, L., Vereneck, R., & Schneider, W. (2003). A Virtual Reality System for Neurobehavioral and Functional MRI Studies. CyberPsychology & Behavior, 6(3), 259-266. doi:10.1089/109493103322011542Maertens, M. (2008). Retinotopic activation in response to subjective contours in primary visual cortex. Frontiers in Human Neuroscience, 2, 1-7. doi:10.3389/neuro.09.002.2008Baumgartner, T., Valko, L., Esslen, M., & Jäncke, L. (2006). Neural Correlate of Spatial Presence in an Arousing and Noninteractive Virtual Reality: An EEG and Psychophysiology Study. CyberPsychology & Behavior, 9(1), 30-45. doi:10.1089/cpb.2006.9.30Belliveau, J., Kennedy, D., McKinstry, R., Buchbinder, B., Weisskoff, R., Cohen, M., … Rosen, B. (1991). Functional mapping of the human visual cortex by magnetic resonance imaging. Science, 254(5032), 716-719. doi:10.1126/science.1948051Born, R. T., & Bradley, D. C. (2005). STRUCTURE AND FUNCTION OF VISUAL AREA MT. Annual Review of Neuroscience, 28(1), 157-189. doi:10.1146/annurev.neuro.26.041002.131052Canli, T., Zhao, Z., Desmond, J. E., Kang, E., Gross, J., & Gabrieli, J. D. E. (2001). An fMRI study of personality influences on brain reactivity to emotional stimuli. Behavioral Neuroscience, 115(1), 33-42. doi:10.1037/0735-7044.115.1.33Clemente, M., Rodríguez, A., Rey, B., Rodríguez, A., Baños, R. M., Botella, C., … Ávila, C. (2011). Analyzing the Level of Presence While Navigating in a Virtual Environment during an fMRI Scan. Lecture Notes in Computer Science, 475-478. doi:10.1007/978-3-642-23768-3_61(Bud) Craig, A. D. (2009). How do you feel — now? The anterior insula and human awareness. Nature Reviews Neuroscience, 10(1), 59-70. doi:10.1038/nrn2555Dilger, S., Straube, T., Mentzel, H.-J., Fitzek, C., Reichenbach, J. R., Hecht, H., … Miltner, W. H. R. (2003). Brain activation to phobia-related pictures in spider phobic humans: an event-related functional magnetic resonance imaging study. Neuroscience Letters, 348(1), 29-32. doi:10.1016/s0304-3940(03)00647-5Dodds, C. M., Morein-Zamir, S., & Robbins, T. W. (2010). Dissociating Inhibition, Attention, and Response Control in the Frontoparietal Network Using Functional Magnetic Resonance Imaging. Cerebral Cortex, 21(5), 1155-1165. doi:10.1093/cercor/bhq187Epstein, R., & Kanwisher, N. (1998). A cortical representation of the local visual environment. Nature, 392(6676), 598-601. doi:10.1038/33402Flach, J. M., & Holden, J. G. (1998). The Reality of Experience: Gibson’s Way. Presence: Teleoperators and Virtual Environments, 7(1), 90-95. doi:10.1162/105474698565550Friston, K. J., Holmes, A. P., Poline, J.-B., Grasby, P. J., Williams, S. C. R., Frackowiak, R. S. J., & Turner, R. (1995). Analysis of fMRI Time-Series Revisited. NeuroImage, 2(1), 45-53. doi:10.1006/nimg.1995.1007GEAKE, J., & HANSEN, P. (2005). Neural correlates of intelligence as revealed by fMRI of fluid analogies. NeuroImage, 26(2), 555-564. doi:10.1016/j.neuroimage.2005.01.035Haldane, M., Cunningham, G., Androutsos, C., & Frangou, S. (2008). Structural brain correlates of response inhibition in Bipolar Disorder I. Journal of Psychopharmacology, 22(2), 138-143. doi:10.1177/0269881107082955Hartley, T., Maguire, E. A., Spiers, H. J., & Burgess, N. (2003). The Well-Worn Route and the Path Less Traveled. Neuron, 37(5), 877-888. doi:10.1016/s0896-6273(03)00095-3Heeter, C. (1992). Being There: The Subjective Experience of Presence. Presence: Teleoperators and Virtual Environments, 1(2), 262-271. doi:10.1162/pres.1992.1.2.262De Castro, F. (2009). Wiring olfaction: the cellular and molecular mechanisms that guide the development of synaptic connections from the nose to the cortex. Frontiers in Neuroscience. doi:10.3389/neuro.22.004.2009Johnson, P. B., Ferraina, S., Bianchi, L., & Caminiti, R. (1996). Cortical Networks for Visual Reaching: Physiological and Anatomical Organization of Frontal and Parietal Lobe Arm Regions. Cerebral Cortex, 6(2), 102-119. doi:10.1093/cercor/6.2.102Karnath, H.-O. (2005). Awareness of the Functioning of One’s Own Limbs Mediated by the Insular Cortex? Journal of Neuroscience, 25(31), 7134-7138. doi:10.1523/jneurosci.1590-05.2005Koechlin, E. (2003). The Architecture of Cognitive Control in the Human Prefrontal Cortex. Science, 302(5648), 1181-1185. doi:10.1126/science.1088545Lang, P. J., Bradley, M. M., Fitzsimmons, J. R., Cuthbert, B. N., Scott, J. D., Moulder, B., & Nangia, V. (1998). Emotional arousal and activation of the visual cortex: An fMRI analysis. Psychophysiology, 35(2), 199-210. doi:10.1017/s0048577298001991Le Bihan, D., Turner, R., Zeffiro, T. A., Cuenod, C. A., Jezzard, P., & Bonnerot, V. (1993). Activation of human primary visual cortex during visual recall: a magnetic resonance imaging study. Proceedings of the National Academy of Sciences, 90(24), 11802-11805. doi:10.1073/pnas.90.24.11802Lessiter, J., Freeman, J., Keogh, E., & Davidoff, J. (2001). A Cross-Media Presence Questionnaire: The ITC-Sense of Presence Inventory. Presence: Teleoperators and Virtual Environments, 10(3), 282-297. doi:10.1162/105474601300343612Loomis, J. M. (1992). Distal Attribution and Presence. Presence: Teleoperators and Virtual Environments, 1(1), 113-119. doi:10.1162/pres.1992.1.1.113Mellet, E., Laou, L., Petit, L., Zago, L., Mazoyer, B., & Tzourio-Mazoyer, N. (2009). Impact of the virtual reality on the neural representation of an environment. Human Brain Mapping, 31(7), 1065-1075. doi:10.1002/hbm.20917Mishkin, M., & Ungerleider, L. G. (1982). Contribution of striate inputs to the visuospatial functions of parieto-preoccipital cortex in monkeys. Behavioural Brain Research, 6(1), 57-77. doi:10.1016/0166-4328(82)90081-xMraz, R., Hong, J., Quintin, G., Staines, W. R., McIlroy, W. E., Zakzanis, K. K., & Graham, S. J. (2003). A Platform for Combining Virtual Reality Experiments with Functional Magnetic Resonance Imaging. CyberPsychology & Behavior, 6(4), 359-368. doi:10.1089/109493103322278736Ochsner, K. N., Bunge, S. A., Gross, J. J., & Gabrieli, J. D. E. (2002). Rethinking Feelings: An fMRI Study of the Cognitive Regulation of Emotion. Journal of Cognitive Neuroscience, 14(8), 1215-1229. doi:10.1162/089892902760807212Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9(1), 97-113. doi:10.1016/0028-3932(71)90067-4Owen, A. M., Downes, J. J., Sahakian, B. J., Polkey, C. E., & Robbins, T. W. (1990). Planning and spatial working memory following frontal lobe lesions in man. Neuropsychologia, 28(10), 1021-1034. doi:10.1016/0028-3932(90)90137-dPerani, D., Fazio, F., Borghese, N. A., Tettamanti, M., Ferrari, S., Decety, J., & Gilardi, M. C. (2001). Different Brain Correlates for Watching Real and Virtual Hand Actions. NeuroImage, 14(3), 749-758. doi:10.1006/nimg.2001.0872Petrides, M. (2000). The role of the mid-dorsolateral prefrontal cortex in working memory. Experimental Brain Research, 133(1), 44-54. doi:10.1007/s002210000399Pine, D. S., Grun, J., Maguire, E. A., Burgess, N., Zarahn, E., Koda, V., … Bilder, R. M. (2002). Neurodevelopmental Aspects of Spatial Navigation: A Virtual Reality fMRI Study. NeuroImage, 15(2), 396-406. doi:10.1006/nimg.2001.0988Riva, G., Waterworth, J. A., Waterworth, E. L., & Mantovani, F. (2011). From intention to action: The role of presence. New Ideas in Psychology, 29(1), 24-37. doi:10.1016/j.newideapsych.2009.11.002Rey, B., Alcañiz, M., Tembl, J., & Parkhutik, V. (2009). Brain activity and presence: a preliminary study in different immersive conditions using transcranial Doppler monitoring. Virtual Reality, 14(1), 55-65. doi:10.1007/s10055-009-0141-2Sanchez-Vives, M. V., & Slater, M. (2005). From presence to consciousness through virtual reality. Nature Reviews Neuroscience, 6(4), 332-339. doi:10.1038/nrn1651Scheibe, C., Wartenburger, I., Wüstenberg, T., Kathmann, N., Villringer, A., & Heekeren, H. R. (2006). Neural correlates of the interaction between transient and sustained processes: A mixed blocked/event-related fMRI study. Human Brain Mapping, 27(7), 545-551. doi:10.1002/hbm.20199Schuemie, M. J., van der Straaten, P., Krijn, M., & van der Mast, C. A. P. G. (2001). Research on Presence in Virtual Reality: A Survey. CyberPsychology & Behavior, 4(2), 183-201. doi:10.1089/109493101300117884Smith, S. M. (2004). Overview of fMRI analysis. The British Journal of Radiology, 77(suppl_2), S167-S175. doi:10.1259/bjr/33553595Usoh, M., Catena, E., Arman, S., & Slater, M. (2000). Using Presence Questionnaires in Reality. Presence: Teleoperators and Virtual Environments, 9(5), 497-503. doi:10.1162/105474600566989Vanni, S., Tanskanen, T., Seppa, M., Uutela, K., & Hari, R. (2001). Coinciding early activation of the human primary visual cortex and anteromedial cuneus. Proceedings of the National Academy of Sciences, 98(5), 2776-2780. doi:10.1073/pnas.041600898Wolf, U., Rapoport, M. J., & Schweizer, T. A. (2009). Evaluating the Affective Component of the Cerebellar Cognitive Affective Syndrome. Journal of Neuropsychiatry, 21(3), 245-253. doi:10.1176/appi.neuropsych.21.3.245Zahorik, P., & Jenison, R. L. (1998). Presence as Being-in-the-World. Presence: Teleoperators and Virtual Environments, 7(1), 78-89. doi:10.1162/10547469856554

Brain dynamic during landmark-based learning spatial navigation

In the current study, I investigated both human behavior and brain dynamics during spatial navigation to gain a better understanding of human navigational strategies and brain signals that underlie spatial cognition. To this end, a custom-built virtual reality task and a 64-channel scalp electroencephalogram (EEG) were utilized to study participants. At the first step, we presented a novel, straightforward, yet powerful tool to evaluate individual differences during navigation, comprising of a virtual radial-arm maze inspired to the animal experiments. The virtual maze is designed and furnished, similar to an art gallery, to provide a more realistic and exciting environment for subjects’ exploration. We investigated whether a different set of instructions (explicit or implicit) affects subjects’ navigational performance, and we assessed the effect of the set of instructions on exploration strategies during both place learning and recall. We tested 42 subjects and evaluated their way-finding ability. Individual differences were assessed through the analysis of the navigational paths, which permitted the isolation and definition of a few strategies adopted by both subjects who adopted a more explicit strategy, based on explicit instructions, and an implicit strategy, based on implicit instructions. The second step aimed to explore brain dynamics and neurophysiological activity during spatial navigation. More specifically, we aimed to figure out how navigational related brain regions are connected and how their interactions and electrical activity vary according to different navigational tasks and environment. This experiment was divided into two steps: learning phase and test phase. The same virtual maze (art gallery) as the behavioral part of the study was used so that subjects to perform landmark-based navigation. The main task of the experiment was finding and memorizing the position of some goals within the environment during the learning phase and retrieving the spatial information of the goals during the test phase. We recorded EEG signals of 20 subjects during the experiment, and both scalp-level and source-level analysis approaches were employed to figure out how the brain represents the spatial location of landmarks and targets and, more precisely, how different brain regions contribute to spatial orientation and landmark-based learning during navigation

Decision in space

Human navigation is generally believed to rely on two types of strategy adoption, route- based and map-based strategies. Both types of navigation require making spatial decisions along the traversed way. Nevertheless, formal computational and neural links between navigational strategies and mechanisms of value based decision making have so far been underexplored in humans. Here, we employed functional magnetic resonance imaging (fMRI) while subjects located different target objects in a virtual environment. We then modelled their paths using reinforcement learning (RL) algorithms, which successfully explain decision behaviour and its neural correlates. Our results show that subjects used a mixture of route and map-based navigation, and their paths could be well explained by the model-free and model-based RL algorithms. Furthermore, the value signals of model-free choices during route-based navigation modulated the BOLD signals in the ventro-medial prefrontal cortex (vmPFC). On the contrary, the BOLD signals in parahippocampal and medial temporal lobe (MTL) regions pertained to model- based value signals during map-based navigation. Our findings suggest that the brain might share computational mechanisms and neural substrates for navigation and value- based decisions, such that model-free choice guides route-based navigation and model- based choice directs map-based navigation. These findings open new avenues for computational modelling of wayfinding by directing attention to value-based decision, differing from common direction and distances approaches. The ability to find one’s way in a complex environment is crucial to everyday functioning. This navigational ability relies on the integrity of several cognitive functions and different strategies, route and map-based navigation, that individuals may adopt while navigating in the environment. As the integrity of these cognitive functions often decline with age, navigational abilities show marked changes in both normal aging and dementia. Combining a wayfinding task in a virtual reality (VR) environment and modeling technique based on reinforcement learning (RL) algorithms, we investigated the effects of cognitive aging on the selection and adoption of navigation strategies in human. The older participants performed the wayfinding task while undergoing functional Magnetic Resonance Imaging (fMRI), and the younger participants performed the same task outside the MRI machine. Compared with younger participants, older participants traversed a longer distance. They also exhibited a higher tendency to repeat previously established routes to locate the target objects. Despite these differences, the traversed paths in both groups could be well explained by the model-free and model-based RL algorithms. Furthermore, neuroimaging results from the older participants show that BOLD signal in the ventromedial prefrontal cortex (vmPFC) pertained to model-free value signals. This result provide evidence on the utility of the RL algorithms to explain how the aging brain computationally prefer to rely more on the route-based navigation

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

3-D Interfaces for Spatial Construction

It is becoming increasingly easy to bring the body directly to digital form via stereoscopic immersive displays and tracked input devices. Is this space a viable one in which to construct 3d objects? Interfaces built upon two-dimensional displays and 2d input devices are the current standard for spatial construction, yet 3d interfaces, where the dimensionality of the interactive space matches that of the design space, have something unique to offer. This work increases the richness of 3d interfaces by bringing several new tools into the picture: the hand is used directly to trace surfaces; tangible tongs grab, stretch, and rotate shapes; a handle becomes a lightsaber and a tool for dropping simple objects; and a raygun, analagous to the mouse, is used to select distant things. With these tools, a richer 3d interface is constructed in which a variety of objects are created by novice users with relative ease. What we see is a space, not exactly like the traditional 2d computer, but rather one in which a distinct and different set of operations is easy and natural. Design studies, complemented by user studies, explore the larger space of three-dimensional input possibilities. The target applications are spatial arrangement, freeform shape construction, and molecular design. New possibilities for spatial construction develop alongside particular nuances of input devices and the interactions they support. Task-specific tangible controllers provide a cultural affordance which links input devices to deep histories of tool use, enhancing intuition and affective connection within an interface. On a more practical, but still emotional level, these input devices frame kinesthetic space, resulting in high-bandwidth interactions where large amounts of data can be comfortably and quickly communicated. A crucial issue with this interface approach is the tension between specific and generic input devices. Generic devices are the tradition in computing -- versatile, remappable, frequently bereft of culture or relevance to the task at hand. Specific interfaces are an emerging trend -- customized, culturally rich, to date these systems have been tightly linked to a single application, limiting their widespread use. The theoretical heart of this thesis, and its chief contribution to interface research at large is an approach to customization. Instead of matching an application domain's data, each new input device supports a functional class. The spatial construction task is split into four types of manipulation: grabbing, pointing, holding, and rubbing. Each of these action classes spans the space of spatial construction, allowing a single tool to be used in many settings without losing the unique strengths of its specific form. Outside of 3d interface, outside of spatial construction, this approach strikes a balance between generic and specific suitable for many interface scenarios. In practice, these specific function groups are given versatility via a quick remapping technique which allows one physical tool to perform many digital tasks. For example, the handle can be quickly remapped from a lightsaber that cuts shapes to tools that place simple platonic solids, erase portions of objects, and draw double-helices in space. The contributions of this work lie both in a theoretical model of spatial interaction, and input devices (combined with new interactions) which illustrate the efficacy of this philosophy. This research brings the new results of Tangible User Interface to the field of Virtual Reality. We find a space, in and around the hand, where full-fledged haptics are not necessary for users physically connect with digital form.</p