Responsive audio feedback for hand gesture interaction to enhance immersion in audio-only games

This thesis investigates different audio feedback designs for touch-less hand gesture interaction to enhance immersion and encourage intuitive user participation in audio-only environments. In particular, we aim to assess whether replacing explicit audio instructions for hand positions and movements with responsive audio feedback using environmental story-related audio cues that are suggestive of interaction methods leads to measurably higher immersion. The potential of using audio cues as feedback to improve the responsiveness of touch-less gesture interaction has been an active topic of study. We focus on methods for effectively informing players about gesture interaction methods through audio cues without breaking game immersion. As part of our research, we developed a framework for hand gesture detection based on the Leap Motion hardware and software. We also present a design approach for audio feedback that considers three stages, before, during and after a gesture sequence, and proposes corresponding design principles. In order to test different audio designs and support our hypothesis, two game versions using non-diegetic or diegetic audio feedback were created as testbeds for a between-subjects study. The study found that the version using environmental feedback led to higher immersion in terms of story and was consistently rated higher. In particular, compared to explicit vocal instructions, environmental story-related audio cues were found to be more helpful in guiding gesture interaction and in enhancing how real the game feels.M.S., Digital Media -- Drexel University, 201

Dialectical Polyptych: an interactive movie installation

Most of the known video games developed by important software companies usually establish an approach to the cinematic language in an attempt to create a perfect combination of narrative, visual technique and interaction. Unlike most video games, interactive film narratives normally involve an interruption in time whenever the spectator has to make choices. “Dialectical Polyptych” is an interactive movie included in a project called “Characters looking for a spectactor”, which aims to give the spectator on-the-fly control over film editing, thus exploiting the role of the spectator as an active subject in the presented narrative. This paper presents an installation based on a mobile device, which allows seamless real-time interactivity with the movie. Different finger touches in the screen allow the spectator to alternate between two parallel narratives, both producing a complementary narrative, and change the angle or shot within each narrative.info:eu-repo/semantics/publishedVersio

GEMINI: A Generic Multi-Modal Natural Interface Framework for Videogames

In recent years videogame companies have recognized the role of player engagement as a major factor in user experience and enjoyment. This encouraged a greater investment in new types of game controllers such as the WiiMote, Rock Band instruments and the Kinect. However, the native software of these controllers was not originally designed to be used in other game applications. This work addresses this issue by building a middleware framework, which maps body poses or voice commands to actions in any game. This not only warrants a more natural and customized user-experience but it also defines an interoperable virtual controller. In this version of the framework, body poses and voice commands are respectively recognized through the Kinect's built-in cameras and microphones. The acquired data is then translated into the native interaction scheme in real time using a lightweight method based on spatial restrictions. The system is also prepared to use Nintendo's Wiimote as an auxiliary and unobtrusive gamepad for physically or verbally impractical commands. System validation was performed by analyzing the performance of certain tasks and examining user reports. Both confirmed this approach as a practical and alluring alternative to the game's native interaction scheme. In sum, this framework provides a game-controlling tool that is totally customizable and very flexible, thus expanding the market of game consumers.Comment: WorldCIST'13 Internacional Conferenc

The Forking Paths revisited: experimenting on interactive film

Based on the triad film-interactivity-experimentation, the applied research project The Forking Paths, developed at the Centre for Research in Arts and Communication (CIAC), endeavours to find alternative narrative forms in the field of Cinema and, more specifically, in the subfield of Interactive Cinema. The films in the project The Forking Paths invest in the interconnectivity between the film narrative and the viewer, who is given the possibility to be more active and engaged. At same time, the films undertake a research on the development of audio-visual language. The project is available at an online platform, which aims to foster the creation and web hosting of other Interactive Cinema projects in its different variables. This article focusses on the three films completed up to the moment: Haze, The Book of the Dead, and Waltz.info:eu-repo/semantics/publishedVersio

The role of participatory workshops in investigating narrative and sound ecologies in the design of an ambient intelligence audio display

We describe two participatory workshops conducted to support design decisions in the making of the audio display for an ambient intelligent game platform. The workshops discussed here explore specific issues of players’ interactions with sound and auditory display design. The workshops helped move our design process forward by specifying the role of narrative and sound ecologies in our design. They clarified the role of sound in creating narrative coherence, guiding player actions, and supporting group interaction. We describe the workshops, the auditory display issues we addressed, discuss how the workshops helped inform our subsequent design, and extend recommendations on how participatory workshops can be used by other designers of auditory displays