MetaSpace II: Object and full-body tracking for interaction and navigation in social VR

MetaSpace II (MS2) is a social Virtual Reality (VR) system where multiple users can not only see and hear but also interact with each other, grasp and manipulate objects, walk around in space, and get tactile feedback. MS2 allows walking in physical space by tracking each user's skeleton in real-time and allows users to feel by employing passive haptics i.e., when users touch or manipulate an object in the virtual world, they simultaneously also touch or manipulate a corresponding object in the physical world. To enable these elements in VR, MS2 creates a correspondence in spatial layout and object placement by building the virtual world on top of a 3D scan of the real world. Through the association between the real and virtual world, users are able to walk freely while wearing a head-mounted device, avoid obstacles like walls and furniture, and interact with people and objects. Most current virtual reality (VR) environments are designed for a single user experience where interactions with virtual objects are mediated by hand-held input devices or hand gestures. Additionally, users are only shown a representation of their hands in VR floating in front of the camera as seen from a first person perspective. We believe, representing each user as a full-body avatar that is controlled by natural movements of the person in the real world (see Figure 1d), can greatly enhance believability and a user's sense immersion in VR.Comment: 10 pages, 9 figures. Video: http://living.media.mit.edu/projects/metaspace-ii

Towards Understanding and Expanding Locomotion in Physical and Virtual Realities

Among many virtual reality interactions, the locomotion dilemma remains a significant impediment to achieving an ideal immersive experience. The physical limitations of tracked space make it impossible to naturally explore theoretically boundless virtual environments with a one-to-one mapping. Synthetic techniques like teleportation and flying often induce simulator sickness and break the sense of presence. Therefore, natural walking is the most favored form of locomotion. Redirected walking offers a more natural and intuitive way for users to navigate vast virtual spaces efficiently. However, existing techniques either lead to simulator sickness due to visual and vestibular mismatch or detract users from the immersive experience that virtual reality aims to provide. This research presents innovative techniques and applications to enhance the user experience by expanding walkable, physical space in Virtual Reality. The thesis includes three main contributions. The first contribution proposes a mobile application that uses markerless Augmented Reality to allow users to explore a life-sized virtual library through a divide-and-rule approach. The second contribution presents a subtle redirected walking technique based on inattentional blindness, using dynamic foveated rendering and natural visual suppressions like blinks and saccades. Finally, the third contribution introduces a novel redirected walking solution that leverages a deep neural network, to predict saccades in real-time and eliminate the hardware requirements for eye-tracking. Overall, this thesis offers valuable contributions to human-computer interaction, investigating novel approaches to solving the locomotion dilemma. The proposed solutions were evaluated through extensive user studies, demonstrating their effectiveness and applicability in real-world scenarios like training simulations and entertainment

Feasibility of a walking virtual reality system for rehabilitation: objective and subjective parameters

[EN] Background: Even though virtual reality (VR) is increasingly used in rehabilitation, the implementation of walking navigation in VR still poses a technological challenge for current motion tracking systems. Different metaphors simulate locomotion without involving real gait kinematics, which can affect presence, orientation, spatial memory and cognition, and even performance. All these factors can dissuade their use in rehabilitation. We hypothesize that a marker-based head tracking solution would allow walking in VR with high sense of presence and without causing sickness. The objectives of this study were to determine the accuracy, the jitter, and the lag of the tracking system and its elicited sickness and presence in comparison of a CAVE system. Methods: The accuracy and the jitter around the working area at three different heights and the lag of the head tracking system were analyzed. In addition, 47 healthy subjects completed a search task that involved navigation in the walking VR system and in the CAVE system. Navigation was enabled by natural locomotion in the walking VR system and through a specific device in the CAVE system. An HMD was used as display in the walking VR system. After interacting with each system, subjects rated their sickness in a seven-point scale and their presence in the Slater-Usoh-Steed Questionnaire and a modified version of the Presence Questionnaire. Results: Better performance was registered at higher heights, where accuracy was less than 0.6 cm and the jitter was about 6 mm. The lag of the system was 120 ms. Participants reported that both systems caused similar low levels of sickness (about 2.4 over 7). However, ratings showed that the walking VR system elicited higher sense of presence than the CAVE system in both the Slater-Usoh-Steed Questionnaire (17.6 +/- 0.3 vs 14.6 +/- 0.6 over 21, respectively) and the modified Presence Questionnaire (107.4 +/- 2.0 vs 93.5 +/- 3.2 over 147, respectively). Conclusions: The marker-based solution provided accurate, robust, and fast head tracking to allow navigation in the VR system by walking without causing relevant sickness and promoting higher sense of presence than CAVE systems, thus enabling natural walking in full-scale environments, which can enhance the ecological validity of VR-based rehabilitation applications.The authors wish to thank the staff of LabHuman for their support in this project, especially José Miguel Martínez and José Roda for their assistance. This study was funded in part by Ministerio de Economia y Competitividad of Spain (Project NeuroVR, TIN2013-44741-R and Project REACT, TIN2014-61975-EXP), by Ministerio de Educacion y Ciencia of Spain (Project Consolider-C, SEJ2006-14301/PSIC), and by Universitat Politecnica de Valencia (Grant PAID-10-14).Borrego, A.; Latorre Grau, J.; Llorens Rodríguez, R.; Alcañiz Raya, ML.; Noé, E. (2016). Feasibility of a walking virtual reality system for rehabilitation: objective and subjective parameters. Journal of NeuroEngineering and Rehabilitation. 13:1-9. https://doi.org/10.1186/s12984-016-0174-1S1913Lee KM. Presence. Explicated Communication Theory. 2004;14(1):27–50.Riva G. Is presence a technology issue? Some insights from cognitive sciences. Virtual Reality. 2009;13(3):159–69.Banos RM, et al. Immersion and emotion: their impact on the sense of presence. Cyberpsychol Behav. 2004;7(6):734–41.Llorens R, et al. Tracking systems for virtual rehabilitation: objective performance vs. subjective experience. A practical scenario. Sensors (Basel). 2015;15(3):6586–606.Navarro MD, et al. Validation of a low-cost virtual reality system for training street-crossing. A comparative study in healthy, neglected and non-neglected stroke individuals. Neuropsychol Rehabil. 2013;23(4):597–618.Parsons TD. Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Front Hum Neurosci. 2015;9:660.Cameirao MS, et al. Neurorehabilitation using the virtual reality based Rehabilitation Gaming System: methodology, design, psychometrics, usability and validation. J Neuroeng Rehabil. 2010;7:48.Llorens R, et al. Improvement in balance using a virtual reality-based stepping exercise: a randomized controlled trial involving individuals with chronic stroke. Clin Rehabil. 2015;29(3):261–8.Llorens R, et al. Videogame-based group therapy to improve self-awareness and social skills after traumatic brain injury. J Neuroeng Rehabil. 2015;12:37.Fong KN, et al. Usability of a virtual reality environment simulating an automated teller machine for assessing and training persons with acquired brain injury. J Neuroeng Rehabil. 2010;7:19.Levin MF, Weiss PL, Keshner EA. Emergence of virtual reality as a tool for upper limb rehabilitation: incorporation of motor control and motor learning principles. Phys Ther. 2015;95(3):415–25.Llorens R, et al. Effectiveness, usability, and cost-benefit of a virtual reality-based telerehabilitation program for balance recovery after stroke: a randomized controlled trial. Arch Phys Med Rehabil. 2015;96(3):418–25. e2.Cruz-Neira C, et al. Scientists in wonderland: A report on visualization applications in the CAVE virtual reality environment. In: 1993. Proceedings IEEE 1993 Symposium on Research Frontiers in Virtual Reality. 1993.Juan MC, Perez D. Comparison of the levels of presence and anxiety in an acrophobic environment viewed via HMD or CAVE. Presence. 2009;18(3):232–48.Yang YR, et al. Virtual reality-based training improves community ambulation in individuals with stroke: a randomized controlled trial. Gait Posture. 2008;28(2):201–6.Cho KH, Lee WH. Virtual walking training program using a real-world video recording for patients with chronic stroke: a pilot study. Am J Phys Med Rehabil. 2013;92(5):371–84.Darter BJ, Wilken JM. Gait training with virtual reality-based real-time feedback: improving gait performance following transfemoral amputation. Phys Ther. 2011;91(9):1385–94.Yang S, et al. Improving balance skills in patients who had stroke through virtual reality treadmill training. Am J Phys Med Rehabil. 2011;90(12):969–78.Walker ML, et al. Virtual reality-enhanced partial body weight-supported treadmill training poststroke: feasibility and effectiveness in 6 subjects. Arch Phys Med Rehabil. 2010;91(1):115–22.Riley PO, et al. A kinematic and kinetic comparison of overground and treadmill walking in healthy subjects. Gait Posture. 2007;26(1):17–24.Alton F, et al. A kinematic comparison of overground and treadmill walking. Clin Biomech. 1998;13(6):434–40.Lee SJ, Hidler J. Biomechanics of overground vs. treadmill walking in healthy individuals. J Appl Physiol. 2008;104(3).Slater M. Measuring presence: a response to the witmer and Singer presence questionnaire. Presence. 1999;8(5):560–5.Viau A, et al. Reaching in reality and virtual reality: a comparison of movement kinematics in healthy subjects and in adults with hemiparesis. J Neuroeng Rehabil. 2004;1(1):11.Parsons TD, et al. The potential of function-led virtual environments for ecologically valid measures of executive function in experimental and clinical neuropsychology. Neuropsychol Rehabil. 2015;11:1–31. doi: 10.1080/09602011.2015.1109524 .Aravind G, Lamontagne A. Perceptual and locomotor factors affect obstacle avoidance in persons with visuospatial neglect. J Neuroeng Rehabil. 2014;11:38.Darekar A, Lamontagne A, Fung J. Dynamic clearance measure to evaluate locomotor and perceptuo-motor strategies used for obstacle circumvention in a virtual environment. Hum Mov Sci. 2015;40:359–71.Whittle MW. Chapter 4 - Methods of gait analysis. In: Whittle MW, editor. Gait analysis. Edinburgh: Butterworth-Heinemann; 2007. p. 137–75.Hodgson E, et al. WeaVR: a self-contained and wearable immersive virtual environment simulation system. Behav Res Methods. 2015;47(1):296–307.Akizuki H, et al. Effects of immersion in virtual reality on postural control. Neurosci Lett. 2005;379(1):23–6.Thies SB, et al. Comparison of linear accelerations from three measurement systems during "reach & grasp". Med Eng Phys. 2007;29(9):967–72.Fiala M. Designing highly reliable fiducial markers. IEEE Trans Pattern Anal Mach Intell. 2010;32(7):1317–24.Garrido-Jurado S, et al. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition. 2014;47(6):2280–92.Kim K, et al. Effects of virtual environment platforms on emotional responses. Comput Methods Programs Biomed. 2014;113(3):882–93.Slater M, Steed A. A virtual presence counter. Presence. 2000;9(5):413–34.Witmer BG, Singer MJ. Measuring presence in virtual environments: a presence questionnaire. Presence Teleop Virt. 1998;7(3):225–40.Martín-Gutiérrez J, et al. Design and validation of an augmented book for spatial abilities development in engineering students. Comput Graph. 2010;34(1):77–91.Lopez-Mir F, et al. Design and validation of an augmented reality system for laparoscopic surgery in a real environment. Biomed Res Int. 2013;2013:758491.Abawi DF, Bienwald J, Dorner R. Accuracy in optical tracking with fiducial markers: an accuracy function for ARToolKit. In: Third IEEE and ACM International symposium on mixed and augmented reality, ISMAR 2004. 2004.Malbezin P, Piekarski W, Thomas BH. Measuring ARTootKit accuracy in long distance tracking experiments. In: The first IEEE International workshop augmented reality toolkit. 2002.Paquette C, Paquet N, Fung J. Aging affects coordination of rapid head motions with trunk and pelvis movements during standing and walking. Gait Posture. 2006;24(1):62–9.Graham JE, et al. Walking speed threshold for classifying walking independence in hospitalized older adults. Phys Ther. 2010;90(11):1591–7.Gorea A. A refresher of the original Bloch’s Law paper (bloch, july 1885). i-Perception. 2015;6:4.Moss JD, Muth ER. Characteristics of head-mounted displays and their effects on Simulator sickness. Hum Factors. 2011;53(3):308–19.Draper MH, et al. Effects of image scale and system time delay on Simulator sickness within head-coupled virtual environments. Hum Factors. 2001;43(1):129–46.Fujisaki W. Effects of delayed visual feedback on grooved pegboard test performance. Front Psychol. 2012;3:61.Keshner EA, et al. Augmenting sensory-motor conflict promotes adaptation of postural behaviors in a virtual environment. Conf Proc IEEE Eng Med Biol Soc. 2011;2011:1379–82.Slaboda JC, Keshner EA. Reorientation to vertical modulated by combined support surface tilt and virtual visual flow in healthy elders and adults with stroke. J Neurol. 2012;259(12):2664–72.Tossavainen T. Comparison of CAVE and HMD for visual stimulation in postural control research. Stud Health Technol Inform. 2004;98:385–7.Akiduki H, et al. Visual-vestibular conflict induced by virtual reality in humans. Neurosci Lett. 2003;340(3):197–200.Duh HBL, et al. Effects of field of view on balance in an immersive environment. In: Virtual Reality, 2001. Proceedings. IEEE. 2001.Krijn M, et al. Treatment of acrophobia in virtual reality: the role of immersion and presence. Behav Res Ther. 2004;42(2):229–39.Mania K, Chalmers A. The effects of levels of immersion on memory and presence in virtual environments: a reality centered approach. Cyberpsychol Behav. 2001;4(2):247–64.Gorini A, et al. The role of immersion and narrative in mediated presence: the virtual hospital experience. Cyberpsychol Behav Soc Netw. 2011;14(3):99–105.Fromberger P, et al. Virtual viewing time: the relationship between presence and sexual interest in androphilic and gynephilic Men. PLoS One. 2015;10(5), e0127156.Slater M, et al. Visual realism enhances realistic response in an immersive virtual environment. IEEE Comput Graph Appl. 2009;29(3):76–84.Nir-Hadad SY, et al. A virtual shopping task for the assessment of executive functions: Validity for people with stroke. Neuropsychol Rehabil. 2015;11:1–26. doi: 10.1080/09602011.2015.1109523 .Vasilyeva M, Lourenco SF. Development of spatial cognition. Wiley Interdiscip Rev Cogn Sci. 2012;3(3):349–62.Banakou D, Groten R, Slater M. Illusory ownership of a virtual child body causes overestimation of object sizes and implicit attitude changes. Proc Natl Acad Sci U S A. 2013;110(31):12846–51.Yee N, Bailenson JN, Ducheneaut N. The proteus effect: implications of transformed digital self-representation on online and offline behavior. Commun Res. 2009;36(2):285–312.Baylor AL. Promoting motivation with virtual agents and avatars: role of visual presence and appearance. Philos Trans R Soc Lond B Biol Sci. 2009;364(1535):3559–65.Clemente M, et al. Assessment of the influence of navigation control and screen size on the sense of presence in virtual reality using EEG. Expert Sys App. 2014;41(4, Part 2):1584–92.Clemente M, et al. An fMRI study to analyze neural correlates of presence during virtual reality experiences. 2013. Interacting with Computers

Cognitive loading affects motor awareness and movement kinematics but not locomotor trajectories during goal-directed walking in a virtual reality environment.

The primary purpose of this study was to investigate the effects of cognitive loading on movement kinematics and trajectory formation during goal-directed walking in a virtual reality (VR) environment. The secondary objective was to measure how participants corrected their trajectories for perturbed feedback and how participants' awareness of such perturbations changed under cognitive loading. We asked 14 healthy young adults to walk towards four different target locations in a VR environment while their movements were tracked and played back in real-time on a large projection screen. In 75% of all trials we introduced angular deviations of ±5° to ±30° between the veridical walking trajectory and the visual feedback. Participants performed a second experimental block under cognitive load (serial-7 subtraction, counter-balanced across participants). We measured walking kinematics (joint-angles, velocity profiles) and motor performance (end-point-compensation, trajectory-deviations). Motor awareness was determined by asking participants to rate the veracity of the feedback after every trial. In-line with previous findings in natural settings, participants displayed stereotypical walking trajectories in a VR environment. Our results extend these findings as they demonstrate that taxing cognitive resources did not affect trajectory formation and deviations although it interfered with the participants' movement kinematics, in particular walking velocity. Additionally, we report that motor awareness was selectively impaired by the secondary task in trials with high perceptual uncertainty. Compared with data on eye and arm movements our findings lend support to the hypothesis that the central nervous system (CNS) uses common mechanisms to govern goal-directed movements, including locomotion. We discuss our results with respect to the use of VR methods in gait control and rehabilitation

Virtua Walker '87

Virtua Walker ’87 is an experimental VR installation that explores current nostalgia for period game technologies and consumer electronics. Virtua Walker ’87 is a playable critique of audience assumptions about VR and the walking simulator genre, informed by Hutcheon’s theory of parody (1989).In the game, players are tasked with walking along a virtual recreation of a real Scottish beach. They control their movement by physically walking on the spot, barefoot inside a custom, sand-filled controller. Player steps are captured by two force sensitive resistors that serve as inputs for the VR-based game. The game world is presented to the player in 2-bit colour (reminiscent of the Nintendo Game Boy), with 8-bit audio and purposefully constrained gameplay. Interaction is limited to walking and looking, with few points of interest along the 4-kilometer stretch of beach. As a practice-based research project, Virtua Walker ’87 sought to explore a number of ideas around virtual reality and walking as a mechanic, including: how contemporary VR can be understood as a form of technostalgia; why VR as a technology often falls short of user expectations in terms of affordances and player control; and how the act of walking is understood and appreciated by players within the wider context of a videogame. Importantly, Virtua Walker ’87 aimed to explore dissonant aesthetics in games, contrasting rich tactile experience (the act of walking barefoot on natural materials) with the disenchantment that results from antiquated sound and graphics and a monotonous ludic interface.<br/

Optimizing Natural Walking Usage in VR using Redirected Teleportation

Virtual Reality (VR) has come a long way since its inception and with the recent advancements in technology, high end VR headsets are now commercially available. Although these headsets offer full motion tracking capabilities, locomotion in VR is yet to be fully solved due to space constraints, potential VR sickness and problems with retaining immersion. Teleportation is the most popular locomotion technique in VR as it allows users to safely navigate beyond the confines of the available positional tracking space without inducing VR sickness. It has been argued that the use of teleportation doesn’t facilitate the use of natural walking input which is considered to have a higher presence because teleportation is faster, requires little physical effort and uses limited available tracking space. When a user walks to the edge of the tracking space, he/she must switch to teleportation. When navigating in the same direction, available walking space does not increase, which forces users to remain stationary and continue using teleportation. We present redirected teleportation, a novel locomotion method that increases tracking space usage and natural walking input by subtle reorientation and repositioning of the user. We first analyzed the positional tendencies of the users as they played popular games implementing teleportation and found the utilization of the tracking space to be limited. We then compared redirected teleportation with regular teleportation using a navigation task in three different environments. Analysis of our data show that although redirected walking takes more time, users used significantly fewer teleports and more natural walking input while using more of the available tracking space. The increase in time is largely due to users walking more, which takes more time than using teleportation. Our results provide evidence that redirected teleportation may be a viable approach to increase the usage of natural walking input while decreasing the dependency on teleportation

Short-Term Trajectory Prediction for Full-Immersive Multiuser Virtual Reality with Redirected Walking

Full-immersive multiuser Virtual Reality (VR) envisions supporting unconstrained mobility of the users in the virtual worlds, while at the same time constraining their physical movements inside VR setups through redirected walking. For enabling delivery of high data rate video content in real-time, the supporting wireless networks will leverage highly directional communication links that will "track" the users for maintaining the Line-of-Sight (LoS) connectivity. Recurrent Neural Networks (RNNs) and in particular Long Short-Term Memory (LSTM) networks have historically presented themselves as a suitable candidate for near-term movement trajectory prediction for natural human mobility, and have also recently been shown as applicable in predicting VR users' mobility under the constraints of redirected walking. In this work, we extend these initial findings by showing that Gated Recurrent Unit (GRU) networks, another candidate from the RNN family, generally outperform the traditionally utilized LSTMs. Second, we show that context from a virtual world can enhance the accuracy of the prediction if used as an additional input feature in comparison to the more traditional utilization of solely the historical physical movements of the VR users. Finally, we show that the prediction system trained on a static number of coexisting VR users be scaled to a multi-user system without significant accuracy degradation.Comment: 7 pages, 9 figures, 2 table