Tangible meets gestural: Comparing and blending post-WIMP interaction paradigms

More and more objects of our everyday environment are becoming smart and connected, offering us new interaction possibilities. Tangible interaction and gestural interaction are promising communication means with these objects in this post-WIMP interaction era. Although based on different principles, they both exploit our body awareness and our skills to provide a richer and more intuitive interaction. Occasionally, when user gestures involve physical artifacts, tangible interaction and gestural interaction can blend into a new paradigm, i.e., tangible gesture interaction [5]. This workshop fosters the comparison among these different interaction paradigms and offers a unique opportunity to discuss their analogies and differences, as well as the definitions, boundaries, strengths, application domains and perspectives of tangible gesture interaction. Participants from different backgrounds are invited

Exploitation of multiplayer interaction and development of virtual puppetry storytelling using gesture control and stereoscopic devices

With the rapid development of human-computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head-mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head-mounted display outperformed the traditional monoscopic display in a comparison between the two

Exploring How Interactive Technology Enhances Gesture-Based Expression and Engagement: A Design Study

The interpretation and understanding of physical gestures play a significant role in various forms of art. Interactive technology and digital devices offer a plethora of opportunities for personal gesture-based experience and they assist in the creation of collaborative artwork. In this study, three prototypes for use with different digital devices (digital camera, PC camera, and Kinect) were designed. Subsequently, a series of workshops were conducted and in-depth interviews with participants from different cultural and occupational backgrounds. The latter were designed to explore how to specifically design personalised gesture-based expressions and how to engage the creativity of the participants in their gesture-based experiences. The findings indicated that, in terms of gesture-based interaction, the participants preferred to engage with the visual traces that were displayed at specific timings in multi-experience spaces. Their gesture-based interactions could effectively support non-verbal emotional expression. In addition, the participants were shown to be strongly inclined to combine their personal stories and emotions into their own gesture-based artworks. Based on the participants’ different cultural and occupational backgrounds, their artistic creation could be spontaneously formed

User interfaces for tangible characters: can children connect remotely through toy perspectives

ABSTRACT What if children's make-believe characters could keep in touch when the children were apart? We propose a novel concept for children's use of technology through imagination play: user interfaces designed to be used by children's character toys rather than directly by the children ("doll-computer interfaces"). We apply this model to the challenge of remote communication for children with an enhanced dollhouse containing small-scale interfaces for the dolls with a variety of fully functional multimodal communication functions. Using this interface as a technology probe, we explore a variety of design decisions with remote pairs of children. Our preliminary results suggest that toy-perspective and manipulable toy elements are particularly helpful in supporting play and successful use of communication technologies, while the "true-to-life" toy aspects are sensitive to individual frames of reference and more flexible interfaces that still fit within the toy context lead to creative communication strategies. We found that different communication channels offered interesting tradeoffs between uninterrupted play and rich verbal description. We also learned that the concept appeals to a wide age range but that the youngest children may need additional scaffolding for successful remote play

Move, hold and touch: A framework for Tangible gesture interactive systems

© 2015 by the authors. Technology is spreading in our everyday world, and digital interaction beyond the screen, with real objects, allows taking advantage of our natural manipulative and communicative skills. Tangible gesture interaction takes advantage of these skills by bridging two popular domains in Human-Computer Interaction, tangible interaction and gestural interaction. In this paper, we present the Tangible Gesture Interaction Framework (TGIF) for classifying and guiding works in this field. We propose a classification of gestures according to three relationships with objects: move, hold and touch. Following this classification, we analyzed previous work in the literature to obtain guidelines and common practices for designing and building new tangible gesture interactive systems. We describe four interactive systems as application examples of the TGIF guidelines and we discuss the descriptive, evaluative and generative power of TGIF

Tangible user interfaces and social interaction in children with autism

Tangible User Interfaces (TUIs) offer the potential for new modes of social interaction for children with Autism Spectrum Conditions (ASC). Familiar objects that are embedded with digital technology may help children with autism understand the actions of others by providing feedback that is logical and predictable. Objects that move, playback sound or create sound – thus repeating programmed effects – offer an exciting way for children to investigate objects and their effects. This thesis presents three studies of children with autism interacting with objects augmented with digital technology. Study one looked at Topobo, a construction toy augmented with kinetic memory. Children played with Topobo in groups of three of either Typically Developing (TD) or ASC children. The children were given a construction task, and were also allowed to play with the construction sets with no task. Topobo in the task condition showed an overall significant effect for more onlooker, cooperative, parallel, and less solitary behaviour. For ASC children significantly less solitary and more parallel behaviour was recorded than other play states. In study two, an Augmented Knights Castle (AKC) playset was presented to children with ASC. The task condition was extended to allow children to configure the playset with sound. A significant effect in a small sample was found for configuration of the AKC, leading to less solitary behaviour, and more cooperative behaviour. Compared to non-digital play, the AKC showed reduction of solitary behaviour because of augmentation. Qualitative analysis showed further differences in learning phase, user content, behaviour oriented to other children, and system responsiveness. Tangible musical blocks (‘d-touch’) in study three focused on the task. TD and ASC children were presented with a guided/non-guided task in pairs, to isolate effects of augmentation. Significant effects were found for an increase in cooperative symbolic play in the guided condition, and more solitary functional play was found in the unguided condition. Qualitative analysis highlighted differences in understanding blocks and block representation, exploratory and expressive play, understanding of shared space and understanding of the system. These studies suggest that the structure of the task conducted with TUIs may be an important factor for children’s use. When the task is undefined, play tends to lose structure and the benefits of TUIs decline. Tangible technology needs to be used in an appropriately structured manner with close coupling (the distance between digital housing and digital effect), and works best when objects are presented in familiar form

Language use between preschoolers, their families, and a social robot while sharing virtual toys

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 66-68).This thesis proposes an approach to language learning for preschool aged children using social robots as conversation partners within a shared play context for children and their families. It addresses an underserved age for language learning, where early learning can greatly impact later educational success, but that cannot benefit from text-based interventions. With the goal of establishing a shared physical context between multiple participants without absorbing all of the children's focus onto digital content, a hybrid physical and digital interface was iteratively designed and play-tested. This interface took the form of a "café table" on which the child and robot could share food. A robot was programmed to introduce itself and name foods in French, eat some foods and express dislike towards others, respond with distress to a new object, show its focus of attention through gaze, and in one experimental condition, express feedback about its comprehension when spoken to in French or English. The study found that some children as young as 3 years old would treat a social robot as an agent capable of understanding them and of perceiving a shared physical context, and would spontaneously modify their use of language and gesture in order to communicate with it - particularly when the robot communicated confusion. The study also found that parents tended to frame their scaffolding of the children's behavior with the robot in a social context, and without prompting aligned their guidance and reinforcement with language learning goals. After one exposure to the robot and new French vocabulary, children did not retain the robot's utterances, but engaged in communicative and social behaviors and language mimicry throughout the interaction. The system appeared to support multi-user social participation, including both caretakers and siblings of the participants.by Natalie Anne Freed.S.M

Supporting improvised games for young people in public spaces

PhD ThesisResearchers looking at technologically mediated play and games have explored how games can be taken away from the computer screen and played in outdoor spaces. This has resulted in new pervasive games that benefit from the opportunities for rich social and physical interaction in new mobile contexts. However, we have only just begun to explore these opportunities; game designs should bring young people together in these new contexts in play that is appropriate, meaningful, and can be enjoyed on their own mobile devices. The research in this thesis explores how game designers and interaction designers can design more playful mobile games for young people that can be played together in public spaces. This work draws upon a research through design approach that has been informed by the researcher’s own practice of game design and working co-creatively with custodians of public spaces. The contributions are based on the analysis of empirical data collected from two case studies in a community library and a country house, while additionally drawing upon three further game designs made in collaborations with other partners. This work contributes a game design framework that provides an approach, a step by step method, guidelines and a software library for making mobile games with more open, spontaneous, and improvised styles of play. The mobile games are designed with and based on a simplistic game system that presents digital playing cards to provide the game structure and bound play, while the mobile device is also used to configure the play space and sustain play. The intention is to provide designers with a practical and evidence-based approach to designing digital games for new mobile contexts. This work will appeal to game designers who are motivated by an interest in play and playfulness that will resonate with our childhood memories of play.UK AHRC KE Hub for the Creative Economy (ref: AH/J005150/1 Creative Exchange