Exploitation of Novel Multiplayer Gesture-based Interaction and Virtual Puppetry for Digital Storytelling to Develop Children’s Narrative Skills

In recent years, digital storytelling has demonstrated powerful pedagogical functions by improving creativity, collaboration and intimacy among young children. Saturated with digital media technologies in their daily lives, the young generation demands natural interactive learning environments which offer multimodalities of feedback and meaningful immersive learning experiences. Virtual puppetry assisted storytelling system for young children, which utilises depth motion sensing technology and gesture control as the Human-Computer Interaction (HCI) method, has been proved to provide natural interactive learning experience for single player. In this paper, we designed and developed a novel system that allows multiple players to narrate, and most importantly, to interact with other characters and interactive virtual items in the virtual environment. We have conducted one user experiment with four young children for pedagogical evaluation and another user experiment with five postgraduate students for system evaluation. Our user study shows this novel digital storytelling system has great potential to stimulate learning abilities of young children through collaboration tasks

Exploitation of multiplayer interaction and development of virtual puppetry storytelling using gesture control and stereoscopic devices

With the rapid development of human-computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head-mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head-mounted display outperformed the traditional monoscopic display in a comparison between the two

Hand gesture-based interactive puppetry system to assist storytelling for children

© 2016 The Author(s)Digital techniques have been used to assist narrative and storytelling, especially in many pedagogical practices. With the rapid development of HCI techniques, saturated with digital media in their daily lives, young children, demands more interactive learning methods and meaningful immersive learning experiences. In this paper, we propose a novel hand gesture-based puppetry storytelling system which provides a more intuitive and natural human computer interaction method for young children to develop narrative ability in virtual story world. Depth motion sensing and hand gestures control technology is utilized in the implementation of user-friendly interaction. Young players could intuitively use hand gestures to manipulate virtual puppet to perform story and interact with different items in virtual environment to assist narration. Based on the result of the evaluation, this novel digital storytelling system shows positive pedagogical functions on children’s narrating ability as well as the competencies of cognitive and motor coordination. The usability of the system is preliminary examined in our test, and the results which showed that young children can benefit from playing with Puppet Narrator

Shadows, touch and digital puppeteering: a media archaeological approach

Aims The practical aim of this research project is to create a multi-touch digital puppetry system that simulates shadow theatre environments and translates gestural acts of touch into live and expressive control of virtual shadow figures. The research is focussed on the qualities of movement achievable through the haptics of single and multi-touch control of the digital puppets in the simulation. An associated aim is to create a collaborative environment where multiple performers can control dynamic animation and scenography, and create novel visualisations and narratives. The conceptual aim is to link traditional and new forms of puppetry seeking cultural significance in the ‘remediation’ of old forms that avail themselves of new haptic resources and collaborative interfaces. The thesis evaluates related prior art where traditional worlds of shadow performance meet new media, digital projection and 3D simulation, in order to investigate how changing technical contexts transform the potential of shadows as an expressive medium. Methodology The thesis uses cultural analysis of relevant documentary material to contextualise the practical work by relating the media archaeology of 2D puppetry—shadows, shadowgraphs and silhouettes—to landmark work in real-time computer graphics and performance animation. The survey considers the work of puppeteers, animators, computer graphics specialists and media artists. Through practice and an experimental approach to critical digital creativity, the study provides practical evidence of multiple iterations of controllable physics-based animation delivering expressive puppet motion through touch and multiuser interaction. Video sequences of puppet movement and written observational analysis document the intangible aspects of animation in performance. Through re-animation of archival shadow puppets, the study presents an emerging artistic media archaeological method. The major element of this method has been the restoration of a collection of Turkish Karagöz Shadow puppets from the Institut International de la Marionnette (Charleville, France) into a playable digital form. Results The thesis presents a developing creative and analytical framework for digital shadow puppetry. It proposes a media archaeological method for working creatively with puppet archives that unlock the kinetic and expressive potential of restored figures. The interaction design introduces novel approaches to puppetry control systems—using spring networks—with objects under physics-simulation that demonstrate emergent expressive qualities. The system facilitates a dance of agency¹ between puppeteer and digital instrument. The practical elements have produced several software iterations and a tool-kit for generating elegant, nuanced multi-touch shadow puppetry. The study presents accidental discoveries—serendipitous benefits of open-ended practical exploration. For instance: the extensible nature of the control system means novel input—other than touch—can provide exciting potential for accessible user interaction, e.g. with gaze duration and eye direction. The study also identifies limitations including the rate of software change and obsolescence, the scope of physics-based animation and failures of simulation. Originality/value The work has historical value in that it documents and begins a media archaeology of digital puppetry, an animated phenomenon of increasing academic and commercial interest. The work is of artistic value providing an interactive approach to making digital performance from archival material in the domain of shadow theatre. The work contributes to the electronic heritage of existing puppetry collections. The study establishes a survey of digital puppetry, setting a research agenda for future studies. Work may proceed to digitise, rig and create collaborative and web-mediated touch-based motion control systems for 2D and 3D puppets. The present study thus provides a solid platform to restore past performances and create new work from old, near forgotten-forms. ¹ Following Andrew Pickering, puppetry is ‘a temporally extended back-and-forth dance of human and non-human agency in which activity and passivity on both sides are reciprocally intertwined’ PICKERING, A. 2010. Material Culture and the Dance of Agency. In: BEAUDRY, M. C. & HICKS, D. (eds.) Oxford Handbook of Material Culture Studies. Oxford University Press.

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion