Multimodality with Eye tracking and Haptics: A New Horizon for Serious Games?

The goal of this review is to illustrate the emerging use of multimodal virtual reality that can benefit learning-based games. The review begins with an introduction to multimodal virtual reality in serious games and we provide a brief discussion of why cognitive processes involved in learning and training are enhanced under immersive virtual environments. We initially outline studies that have used eye tracking and haptic feedback independently in serious games, and then review some innovative applications that have already combined eye tracking and haptic devices in order to provide applicable multimodal frameworks for learning-based games. Finally, some general conclusions are identified and clarified in order to advance current understanding in multimodal serious game production as well as exploring possible areas for new applications

Evaluating Direct Pointing and Indirect Cursor Interactions with Fitts' Law in Stereoscopic Environments

The development of virtual environment research has reached the stage of human interaction with three-dimensional (3D) objects. In this study, Fitts' method was used to such interaction techniques in virtual environment, and the Fitts' law applicability in 3D virtual environment was also considered. The experiment included two modes of interaction: direct interaction and indirect interaction that utilize different techniques depending on how users interact with 3D objects. Both interaction techniques were conducted in three indexes of difficulties and three egocentric target distances (a distance from participant to target). Movement time and throughput were measured for each interaction technique. The results show that the direct pointing technique is more efficient for interaction with the targets close to the participant, while the indirect cursor technique may be a viable option for targets further away from participant. Throughputs were found to be significantly higher for the direct pointing technique compared to the indirect cursor technique. The results of the mean movement time were highly correlated with the targets' index of difficulty for all interaction techniques, supporting evidence that Fitts' law can be applied to the interactions in 3D virtual environment. Based on the results, developers of VE application may relate to these findings in designing proper users' interactions

Evaluating Direct Pointing and Indirect Cursor Interactions with Fitts' Law in Stereoscopic Environments

The development of virtual environment research has reached the stage of human interaction with three-dimensional (3D) objects. In this study, Fitts' method was used to such interaction techniques in virtual environment, and the Fitts' law applicability in 3D virtual environment was also considered. The experiment included two modes of interaction: direct interaction and indirect interaction that utilize different techniques depending on how users interact with 3D objects. Both interaction techniques were conducted in three indexes of difficulties and three egocentric target distances (a distance from participant to target). Movement time and throughput were measured for each interaction technique. The results show that the direct pointing technique is more efficient for interaction with the targets close to the participant, while the indirect cursor technique may be a viable option for targets further away from participant. Throughputs were found to be significantly higher for the direct pointing technique compared to the indirect cursor technique. The results of the mean movement time were highly correlated with the targets' index of difficulty for all interaction techniques, supporting evidence that Fitts' law can be applied to the interactions in 3D virtual environment. Based on the results, developers of VE application may relate to these findings in designing proper users' interactions

Effects of feedback, mobility and index of difficulty on deictic spatial audio target acquisition in the horizontal plane

We present the results of an empirical study investigating the effect of feedback, mobility and index of difficulty on a deictic spatial audio target acquisition task in the horizontal plane in front of a user. With audio feedback, spatial audio display elements are found to enable usable deictic interac-tion that can be described using Fitts law. Feedback does not affect perceived workload or preferred walking speed compared to interaction without feedback. Mobility is found to degrade interaction speed and accuracy by 20%. Participants were able to perform deictic spatial audio target acquisition when mobile while walking at 73% of their pre-ferred walking speed. The proposed feedback design is ex-amined in detail and the effects of variable target widths are quantified. Deictic interaction with a spatial audio display is found to be a feasible solution for future interface designs

Sonic interactions in virtual environments

This book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Evaluating the Influence of Haptic Force-Feedback on 3D Selection Tasks using Natural Egocentric Gestures

Immersive Virtual Environments (IVEs) allow participants to interact with their 3D surroundings using natural hand gestures. Previous work shows that the addition of haptic feedback cues improves performance on certain 3D tasks. However, we believe this is not true for all situations. Depending on the difficulty of the task, we suggest that we should expect differences in the ballistic movement of our hands when presented with different types of haptic force-feedback conditions. We investigated how hard, soft and no haptic force-feedback responses, experienced when in contact with the surface of an object, affected user performance on a task involving selection of multiple targets. To do this, we implemented a natural egocentric selection interaction technique by integrating a two-handed large-scale force-feedback device in to a CAVE (TM)-like IVE system. With this, we performed a user study where we show that participants perform selection tasks best when interacting with targets that exert soft haptic force-feedback cues. For targets that have hard and no force-feedback properties, we highlight certain associated hand movement that participants make under these conditions, that we hypothesise reduce their performance

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments