Immersion and interaction: Creating virtual 3d worlds for stage performances

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.This thesis formulates an approach towards the creation of a gesture activated and body movement controlled real time virtual 3d world in a dance performance context. It investigates immersion and navigation techniques derived from modern video games and methodologies and proposes how they can be used to further involve a performer into a virtual space as well as simultaneously offer a stimulating visual spectacle for an audience. The argument presented develops through practice-based methodology and artistic production strategies in interdisciplinary and collaborative contexts. Two choreographic performance/installations are used as cases studies to demonstrate in practice the proposed methodologies. First, the interactive dance work Suna No Onna, created in collaboration with Birringer/Danjoux and the Dap Lab, investigates the use of interactive pre-rendered animations in a real time setting and in real time by incorporating wearable sensors in the performance. Secondly, the potentials offered by the sensor technology and real time rendering engines led to the “creation scene", a key scene in the choreographic installation UKIYO (Moveable Worlds). This thesis investigates the design, creation and interaction qualities of virtual 3d spaces by exploring the potentialities offered by a shared space, between an intelligent space and a dancer in a hybrid world. The methodology applied uses as a theoretical base the phenomenological approach of Merleau-Ponty and Mark Hansen‟s mixed reality paradigm proposing the concept of the “space schema", a system which replicates and embeds proprioception, perception and motility into the space fabric offering a world which “lives”, functions and interacts with the performer. The outcome of the research is the generation of an interactive, non-linear, randomized 3d virtual space that collaborates with a technologically embedded performer in creating a 3d world which evolves and transforms, driven by the performer‟s intention and agency. This research contributes to the field of interactive performance art by making transparent the methodology, the instruments and the code used, in a non-technical terminology, making it accessible for both team members with less technological expertise as well as artists aspiring to engage interactive 3d media promoting further experimentation and conceptual discussions

Haptic Media Scenes

The aim of this thesis is to apply new media phenomenological and enactive embodied cognition approaches to explain the role of haptic sensitivity and communication in personal computer environments for productivity. Prior theory has given little attention to the role of haptic senses in influencing cognitive processes, and do not frame the richness of haptic communication in interaction design—as haptic interactivity in HCI has historically tended to be designed and analyzed from a perspective on communication as transmissions, sending and receiving haptic signals. The haptic sense may not only mediate contact confirmation and affirmation, but also rich semiotic and affective messages—yet this is a strong contrast between this inherent ability of haptic perception, and current day support for such haptic communication interfaces. I therefore ask: How do the haptic senses (touch and proprioception) impact our cognitive faculty when mediated through digital and sensor technologies? How may these insights be employed in interface design to facilitate rich haptic communication? To answer these questions, I use theoretical close readings that embrace two research fields, new media phenomenology and enactive embodied cognition. The theoretical discussion is supported by neuroscientific evidence, and tested empirically through case studies centered on digital art. I use these insights to develop the concept of the haptic figura, an analytical tool to frame the communicative qualities of haptic media. The concept gauges rich machine- mediated haptic interactivity and communication in systems with a material solution supporting active haptic perception, and the mediation of semiotic and affective messages that are understood and felt. As such the concept may function as a design tool for developers, but also for media critics evaluating haptic media. The tool is used to frame a discussion on opportunities and shortcomings of haptic interfaces for productivity, differentiating between media systems for the hand and the full body. The significance of this investigation is demonstrating that haptic communication is an underutilized element in personal computer environments for productivity and providing an analytical framework for a more nuanced understanding of haptic communication as enabling the mediation of a range of semiotic and affective messages, beyond notification and confirmation interactivity

Defining Interaction within Immersive Virtual Environments

PhDThis thesis is concerned with the design of Virtual Environments (YEs) - in particular with the tools and techniques used to describe interesting and useful environments. This concern is not only with respect to the appearance of objects in the VE but also with their behaviours and their reactions to actions of the participants. The main research hypothesis is that there are several advantages to constructing these interactions and behaviours whilst remaining immersed within the VE which they describe. These advantages include the fact that editing is done interactively with immediate effect and without having to resort to the usual edit-compile-test cycle. This means that the participant doesn't have to leave the VE and lose their sense of presence within it, and editing tasks can take advantage of the enhanced spatial cognition and naturalistic interaction metaphors a VE provides. To this end a data flow dialogue architecture with an immersive virtual environment presentation system was designed and built. The data flow consists of streams of data that originate at sensors that register the body state of the participant, flowing through filters that modify the streams and affect the yE. The requirements for such a system and the filters it should contain are derived from two pieces of work on interaction metaphors, one based on a desktop system using a novel input device and the second a navigation technique for an immersive system. The analysis of these metaphors highlighted particular tasks that such a virtual environment dialogue architecture (VEDA) system might be used to solve, and illustrate the scope of interactions that should be accommodated. Initial evaluation of the VEDA system is provided by moderately sized demonstration environments and tools constructed by the author. Further evaluation is provided by an in-depth study where three novice VE designers were invited to construct VEs with the VEDA system. This highlighted the flexibility that the VEDA approach provides and the utility of the immersive presentation over traditional techniques in that it allows the participant to use more natural and expressive techniques in the construction process. In other words the evaluation shows how the immersive facilities of VEs can be exploited in the process of constructing further VEs

Virtual worlds for education: methodology, interaction and evaluation

2011 - 2012When students arrive in the classroom they expect to be involved in immersive, fun and challenging learning experiences. There is a high risk that they become quickly bored by the traditional instructional methods. The technological evolution offers a great variety of sophisticated interactive devices and applications that can be combined with innovative learning approaches to enhance study efficiency during the learning process. 3D immersive multi-user Virtual Worlds (VWs) are increasingly becoming popular and accessible to wide public due to the advances in computational power graphics and network bandwidth also connected with reduced costs. As a consequence, it is possible to offer more engaging user experiences. This is particularly true in the learning sector, where an increasing interest is worldwide rising towards three-dimensional (3D) VWs and new interaction modalities to which young digital native people are accustomed to. Researches on the educational value of VWs have revealed their potential as learning platforms. However, further studies are always needed in order to assess their effectiveness, satisfactorily and social engagement not only in the general didactic use of the environment, but also for each specific learning subjects, activities and modality. The main challenge is to well exploit VW features and determine learning approaches and interaction modalities in which the didactic actions present added value with respect to traditional education. Indeed, educational VW activities are evolving from the early ones based only on information displaying towards simulated laboratories and new interaction modalities. The main objective of this thesis is to propose new learning methodologies in Virtual Worlds, also experimenting new interaction modalities and evaluating the effectiveness of the support provided. To this aim we first investigate how effectively a 3D city-building game supports the learning of the waste disposal practice and promotes behavior change. The game is one of the results of a research project funded by Regione Campania and is addressed to primary school children. A deep analysis of the didactic methodologies adopted worldwide has been performed to propose a reputation-based learning approach based on collaborative, competitive and individual activities. Also, the effectiveness of the proposed approach has been evaluated. The didactic opportunities offered by VWs when considering new interaction approaches are also investigated. Indeed, if for the last four decades keyboard and mouse have been the primary means for interacting with computers, recently, the availability of greater processing power, wider memories, cameras, and sensors make it possible to introduce new interaction modalities in commonly used software. Gestural interfaces offer new interaction modalities that the primary school children known well and may result accepted also for higher students. To assess the potentiality of this new interaction approach during learning activities we selected Geography as subject, since there is a decreasing interest of the students towards this topic. To this aim the GeoFly system supporting the Geography learning based on a Virtual Globe and on the interaction modalities offered by Microsoft Kinect has been developed. GeoFly is designed for elementary school level Geography students. It enables the exploration of the World by flying, adopting the bird (or aeroplane) metaphor. It also enables the teacher to create learning trips by associating to specific places images, text and videos, to develop learning activities concerning geographically situated scenarios. The proposed approach has been evaluated through a controlled experiment aiming at assessing the effect of the adoption of GeoFly on both the students' attitude towards learning Geography and also on their knowledge. [edited by author]XI n.s

Immersion and interaction : creating virtual 3D worlds for stage performances

This thesis formulates an approach towards the creation of a gesture activated and body movement controlled real time virtual 3d world in a dance performance context. It investigates immersion and navigation techniques derived from modern video games and methodologies and proposes how they can be used to further involve a performer into a virtual space as well as simultaneously offer a stimulating visual spectacle for an audience. The argument presented develops through practice-based methodology and artistic production strategies in interdisciplinary and collaborative contexts. Two choreographic performance/installations are used as cases studies to demonstrate in practice the proposed methodologies. First, the interactive dance work Suna No Onna, created in collaboration with Birringer/Danjoux and the Dap Lab, investigates the use of interactive pre-rendered animations in a real time setting and in real time by incorporating wearable sensors in the performance. Secondly, the potentials offered by the sensor technology and real time rendering engines led to the “creation scene", a key scene in the choreographic installation UKIYO (Moveable Worlds). This thesis investigates the design, creation and interaction qualities of virtual 3d spaces by exploring the potentialities offered by a shared space, between an intelligent space and a dancer in a hybrid world. The methodology applied uses as a theoretical base the phenomenological approach of Merleau-Ponty and Mark Hansen‟s mixed reality paradigm proposing the concept of the “space schema", a system which replicates and embeds proprioception, perception and motility into the space fabric offering a world which “lives”, functions and interacts with the performer. The outcome of the research is the generation of an interactive, non-linear, randomized 3d virtual space that collaborates with a technologically embedded performer in creating a 3d world which evolves and transforms, driven by the performer‟s intention and agency. This research contributes to the field of interactive performance art by making transparent the methodology, the instruments and the code used, in a non-technical terminology, making it accessible for both team members with less technological expertise as well as artists aspiring to engage interactive 3d media promoting further experimentation and conceptual discussions.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Toward multimodality: gesture and vibrotactile feedback in natural human computer interaction

In the present work, users’ interaction with advanced systems has been investigated in different application domains and with respect to different interfaces. The methods employed were carefully devised to respond to the peculiarities of the interfaces under examination. We could extract a set of recommendations for developers. The first application domain examined regards the home. In particular, we addressed the design of a gestural interface for controlling a lighting system embedded into a piece of furniture in the kitchen. A sample of end users was observed while interacting with the virtual simulation of the interface. Based on the videoanalysis of users’ spontaneous behaviors, we could derive a set of significant interaction trends The second application domain involved the exploration of an urban environment in mobility. In a comparative study, a haptic-audio interface and an audio-visual interface were employed for guiding users towards landmarks and for providing them with information. We showed that the two systems were equally efficient in supporting the users and they were both well- received by them. In a navigational task we compared two tactile displays each embedded in a different wearable device, i.e., a glove and a vest. Despite the differences in the shape and size, both systems successfully directed users to the target. The strengths and the flaws of the two devices were pointed out and commented by users. In a similar context, two devices supported Augmented Reality technology, i.e., a pair of smartglasses and a smartphone, were compared. The experiment allowed us to identify the circumstances favoring the use of smartglasses or the smartphone. Considered altogether, our findings suggest a set of recommendations for developers of advanced systems. First, we outline the importance of properly involving end users for unveiling intuitive interaction modalities with gestural interfaces. We also highlight the importance of providing the user the chance to choose the interaction mode better fitting the contextual characteristics and to adjust the features of every interaction mode. Finally, we outline the potential of wearable devices to support interactions on the move and the importance of finding a proper balance between the amount of information conveyed to the user and the size of the device

Direct Animation Interfaces : an Interaction Approach to Computer Animation

Creativity tools for digital media have been largely democratised, offering a range from beginner to expert tools. Yet computer animation, the art of instilling life into believable characters and fantastic worlds, is still a highly sophisticated process restricted to the spheres of expert users. This is largely due to the methods employed: in keyframe animation dynamics are indirectly specified over abstract descriptions, while performance animation suffers from inflexibility due to a high technological overhead. The reverse trend in human-computer interaction to make interfaces more direct, intuitive, and natural to use has so far hardly touched the animation world: decades of interaction research have scarcely been linked to research and development of animation techniques. The hypothesis of this work is that an interaction approach to computer animation can inform the design and development of novel animation techniques. Three goals are formulated to illustrate the validity of this thesis. Computer animation methods and interfaces must be embedded in an interaction context. The insights this brings for designing next generation animation tools must be examined and formalised. The practical consequences for the development of motion creation and editing tools must be demonstrated with prototypes that are more direct, efficient, easy-to-learn, and flexible to use. The foundation of the procedure is a conceptual framework in the form of a comprehensive discussion of the state of the art, a design space of interfaces for time-based visual media, and a taxonomy for mappings between user and medium space-time. Based on this, an interaction-centred analysis of computer animation culminates in the concept of direct animation interfaces and guidelines for their design. These guidelines are tested in two point designs for direct input devices. The design, implementation and test of a surface-based performance animation tool takes a system approach, addressing interaction design issues as well as challenges in extending current software architectures to support novel forms of animation control. The second, a performance timing technique, shows how concepts from video browsing can be applied to motion editing for more direct and efficient animation timing

Virtual reality and body rotation: 2 flight experiences in comparison

Embodied interfaces, represented by devices that incorporate bodily motion and proprioceptive stimulation, are promising for Virtual Reality (VR) because they can improve immersion and user experience while at the same time reducing simulator sickness compared to more traditional handheld interfaces (e.g.,gamepads). The aim of the study is to evaluate a novel embodied interface called VitruvianVR. The machine is composed of two separate rings that allow its users to bodily rotate onto three different axes. The suitability of the VitruvianVR was tested in a Virtual Reality flight scenario. In order to reach the goal we compared the VitruvianVR to a gamepad using perfomance measures (i.e., accuracy, fails), head movements and position of the body. Furthermore, a series of data coming from questionnaires about sense of presence, user experience, cognitive load, usability and cybersickness was retrieved.Embodied interfaces, represented by devices that incorporate bodily motion and proprioceptive stimulation, are promising for Virtual Reality (VR) because they can improve immersion and user experience while at the same time reducing simulator sickness compared to more traditional handheld interfaces (e.g.,gamepads). The aim of the study is to evaluate a novel embodied interface called VitruvianVR. The machine is composed of two separate rings that allow its users to bodily rotate onto three different axes. The suitability of the VitruvianVR was tested in a Virtual Reality flight scenario. In order to reach the goal we compared the VitruvianVR to a gamepad using perfomance measures (i.e., accuracy, fails), head movements and position of the body. Furthermore, a series of data coming from questionnaires about sense of presence, user experience, cognitive load, usability and cybersickness was retrieved