Content rendering and interaction technologies for digital heritage systems

Existing digital heritage systems accommodate a huge amount of digital repository information; however their content rendering and interaction components generally lack the more interesting functionality that allows better interaction with heritage contents. Many digital heritage libraries are simply collections of 2D images with associated metadata and textual content, i.e. little more than museum catalogues presented online. However, over the last few years, largely as a result of EU framework projects, some 3D representation of digital heritage objects are beginning to appear in a digital library context. In the cultural heritage domain, where researchers and museum visitors like to observe cultural objects as closely as possible and to feel their existence and use in the past, giving the user only 2D images along with textual descriptions significantly limits interaction and hence understanding of their heritage. The availability of powerful content rendering technologies, such as 3D authoring tools to create 3D objects and heritage scenes, grid tools for rendering complex 3D scenes, gaming engines to display 3D interactively, and recent advances in motion capture technologies for embodied immersion, allow the development of unique solutions for enhancing user experience and interaction with digital heritage resources and objects giving a higher level of understanding and greater benefit to the community. This thesis describes DISPLAYS (Digital Library Services for Playing with Shared Heritage Resources), which is a novel conceptual framework where five unique services are proposed for digital content: creation, archival, exposition, presentation and interaction services. These services or tools are designed to allow the heritage community to create, interpret, use and explore digital heritage resources organised as an online exhibition (or virtual museum). This thesis presents innovative solutions for two of these services or tools: content creation where a cost effective render grid is proposed; and an interaction service, where a heritage scenario is presented online using a real-time motion capture and digital puppeteer solution for the user to explore through embodied immersive interaction their digital heritage

What do Collaborations with the Arts Have to Say About Human-Robot Interaction?

This is a collection of papers presented at the workshop What Do Collaborations with the Arts Have to Say About HRI , held at the 2010 Human-Robot Interaction Conference, in Osaka, Japan

Shadows, touch and digital puppeteering: a media archaeological approach

Aims The practical aim of this research project is to create a multi-touch digital puppetry system that simulates shadow theatre environments and translates gestural acts of touch into live and expressive control of virtual shadow figures. The research is focussed on the qualities of movement achievable through the haptics of single and multi-touch control of the digital puppets in the simulation. An associated aim is to create a collaborative environment where multiple performers can control dynamic animation and scenography, and create novel visualisations and narratives. The conceptual aim is to link traditional and new forms of puppetry seeking cultural significance in the ‘remediation’ of old forms that avail themselves of new haptic resources and collaborative interfaces. The thesis evaluates related prior art where traditional worlds of shadow performance meet new media, digital projection and 3D simulation, in order to investigate how changing technical contexts transform the potential of shadows as an expressive medium. Methodology The thesis uses cultural analysis of relevant documentary material to contextualise the practical work by relating the media archaeology of 2D puppetry—shadows, shadowgraphs and silhouettes—to landmark work in real-time computer graphics and performance animation. The survey considers the work of puppeteers, animators, computer graphics specialists and media artists. Through practice and an experimental approach to critical digital creativity, the study provides practical evidence of multiple iterations of controllable physics-based animation delivering expressive puppet motion through touch and multiuser interaction. Video sequences of puppet movement and written observational analysis document the intangible aspects of animation in performance. Through re-animation of archival shadow puppets, the study presents an emerging artistic media archaeological method. The major element of this method has been the restoration of a collection of Turkish Karagöz Shadow puppets from the Institut International de la Marionnette (Charleville, France) into a playable digital form. Results The thesis presents a developing creative and analytical framework for digital shadow puppetry. It proposes a media archaeological method for working creatively with puppet archives that unlock the kinetic and expressive potential of restored figures. The interaction design introduces novel approaches to puppetry control systems—using spring networks—with objects under physics-simulation that demonstrate emergent expressive qualities. The system facilitates a dance of agency¹ between puppeteer and digital instrument. The practical elements have produced several software iterations and a tool-kit for generating elegant, nuanced multi-touch shadow puppetry. The study presents accidental discoveries—serendipitous benefits of open-ended practical exploration. For instance: the extensible nature of the control system means novel input—other than touch—can provide exciting potential for accessible user interaction, e.g. with gaze duration and eye direction. The study also identifies limitations including the rate of software change and obsolescence, the scope of physics-based animation and failures of simulation. Originality/value The work has historical value in that it documents and begins a media archaeology of digital puppetry, an animated phenomenon of increasing academic and commercial interest. The work is of artistic value providing an interactive approach to making digital performance from archival material in the domain of shadow theatre. The work contributes to the electronic heritage of existing puppetry collections. The study establishes a survey of digital puppetry, setting a research agenda for future studies. Work may proceed to digitise, rig and create collaborative and web-mediated touch-based motion control systems for 2D and 3D puppets. The present study thus provides a solid platform to restore past performances and create new work from old, near forgotten-forms. ¹ Following Andrew Pickering, puppetry is ‘a temporally extended back-and-forth dance of human and non-human agency in which activity and passivity on both sides are reciprocally intertwined’ PICKERING, A. 2010. Material Culture and the Dance of Agency. In: BEAUDRY, M. C. & HICKS, D. (eds.) Oxford Handbook of Material Culture Studies. Oxford University Press.

Supplementing Frequency Domain Interpolation Methods for Character Animation

The animation of human characters entails difficulties exceeding those met simulating objects, machines or plants. A person's gait is a product of nature affected by mood and physical condition. Small deviations from natural movement are perceived with ease by an unforgiving audience. Motion capture technology is frequently employed to record human movement. Subsequent playback on a skeleton underlying the character being animated conveys many of the subtleties of the original motion. Played-back recordings are of limited value, however, when integration in a virtual environment requires movements beyond those in the motion library, creating a need for the synthesis of new motion from pre-recorded sequences. An existing approach involves interpolation between motions in the frequency domain, with a blending space defined by a triangle network whose vertices represent input motions. It is this branch of character animation which is supplemented by the methods presented in this thesis, with work undertaken in three distinct areas. The first is a streamlined approach to previous work. It provides benefits including an efficiency gain in certain contexts, and a very different perspective on triangle network construction in which they become adjustable and intuitive user-interface devices with an increased flexibility allowing a greater range of motions to be blended than was possible with previous networks. Interpolation-based synthesis can never exhibit the same motion variety as can animation methods based on the playback of rearranged frame sequences. Limitations such as this were addressed by the second phase of work, with the creation of hybrid networks. These novel structures use properties of frequency domain triangle blending networks to seamlessly integrate playback-based animation within them. The third area focussed on was distortion found in both frequency- and time-domain blending. A new technique, single-source harmonic switching, was devised which greatly reduces it, and adds to the benefits of blending in the frequency domain

Wayang Authoring: A Web-based Authoring Tool for Visual Storytelling for Children

This research focuses on the development of the Wayang Authoring tool as it aims to assist children in creating and performing stories, developing an appreciation for cultural artifacts, and enhancing intercultural empathy while building a young storyteller community in a virtual world. This study seeks a framework of interaction design of an authoring media which is appropriate to support children s narrative development. The concept of the tool is based on the narrative element of the ancient Indonesian art form wayang, a traditional two dimensional shadow puppet theater. To understand the user s requirements and the cultural dimension, children and professional story performers who use wayang have been involved in the design process. In order to evaluate the tool, several workshops have been conducted with children from different cultural backgrounds as well as with their teachers. Wayang Authoring is composed of three elements: the imagination-building element, the creative acting element and the social interaction element. Children take existing materials as an inspiration tool, imagine what they themselves want to tell, create a story based on their own ideas, play with their creations, share their stories and creations with others, and reflect on their experiences at the end. This virtual creative production tool is expected to provide a space for young people to change their role from a simple user to a (co-)creator in the virtual and narrative worlds. The core contributions are in the field of web technology for storytelling. The uses of web-based authoring media enable children to put themselves into the process of developing stories. When they are connecting stories, they are connected and immersed with other children as well. They have to act and play by themselves or with others within the stories in order to experience the narratives. They train to have the skills to interact, to share their ideas and to collaborate constructively. This makes it possible for them to participate in today s media-driven culture. This research found that a better understanding of how stories are crafted and brought to life in a performance tradition offers a better design of interaction of an authoring media. The handling of cultural artifacts supports the ability to understand different cultural codes and to pursue the learning process surrounding the original culture behind these artifacts

Investigating User Experience Using Gesture-based and Immersive-based Interfaces on Animation Learners

Creating animation is a very exciting activity. However, the long and laborious process can be extremely challenging. Keyframe animation is a complex technique that takes a long time to complete, as the procedure involves changing the poses of characters through modifying the time and space of an action, called frame-by-frame animation. This involves the laborious, repetitive process of constantly reviewing results of the animation in order to make sure the movement-timing is accurate. A new approach to animation is required in order to provide a more intuitive animating experience. With the evolution of interaction design and the Natural User Interface (NUI) becoming widespread in recent years, a NUI-based animation system is expected to allow better usability and efficiency that would benefit animation. This thesis investigates the effectiveness of gesture-based and immersive-based interfaces as part of animation systems. A practice-based element of this research is a prototype of the hand gesture interface, which was created based on experiences from reflective practices. An experimental design is employed to investigate the usability and efficiency of gesture-based and immersive-based interfaces in comparison to the conventional GUI/WIMP interface application. The findings showed that gesture-based and immersive-based interfaces are able to attract animators in terms of the efficiency of the system. However, there was no difference in their preference for usability with the two interfaces. Most of our participants are pleasant with NUI interfaces and new technologies used in the animation process, but for detailed work and taking control of the application, the conventional GUI/WIMP is preferable. Despite the awkwardness of devising gesture-based and immersive-based interfaces for animation, the concept of the system showed potential for a faster animation process, an enjoyable learning system, and stimulating interest in a kinaesthetic learning experience

Tangent-space optimization for interactive animation control

Character animation tools are based on a keyframing metaphor where artists pose characters at selected keyframes and the software automatically interpolates the frames inbetween. Although the quality of the interpolation is critical for achieving a fluid and engaging animation, the tools available to adjust the result of the automatic inbetweening are rudimentary and typically require manual editing of spline parameters. As a result, artists spend a tremendous amount of time posing and setting more keyframes. In this pose-centric workflow, animators use combinations of forward and inverse kinematics. While forward kinematics leads to intuitive interpolations, it does not naturally support positional constraints such as fixed contact points. Inverse kinematics can be used to fix certain points in space at keyframes, but can lead to inferior interpolations, is slow to compute, and does not allow for positional contraints at non-keyframe frames. In this paper, we address these problems by formulating the control of interpolations with positional constraints over time as a space-time optimization problem in the tangent space of the animation curves driving the controls. Our method has the key properties that it (1) allows the manipulation of positions and orientations over time, extending inverse kinematics, (2) does not add new keyframes that might conflict with an artist's preferred keyframe style, and (3) works in the space of artist editable animation curves and hence integrates seamlessly with current pipelines. We demonstrate the utility of the technique in practice via various examples and use cases.</jats:p

Roots Reloaded. Culture, Identity and Social Development in the Digital Age

This edited volume is designed to explore different perspectives of culture, identity and social development using the impact of the digital age as a common thread, aiming at interdisciplinary audiences. Cases of communities and individuals using new technology as a tool to preserve and explore their cultural heritage alongside new media as a source for social orientation ranging from language acquisition to health-related issues will be covered. Therefore, aspects such as Art and Cultural Studies, Media and Communication, Behavioral Science, Psychology, Philosophy and innovative approaches used by creative individuals are included. From the Aboriginal tribes of Australia, to the Maoris of New Zealand, to the mystical teachings of Sufi brotherhoods, the significance of the oral and written traditions and their current relation to online activities shall be discussed in the opening article. The book continues with a closer look at obesity awareness support groups and their impact on social media, Facebook usage in language learning context, smartphone addiction and internet dependency, as well as online media reporting of controversial ethical issues. The Digital progress has already left its dominating mark as the world entered the 21st century. Without a doubt, as technology continues its ascent, society will be faced with new and altering values in an effort to catch-up with this extraordinary Digitization, adapt satisfactorily in order to utilize these strong developments in everyday life

Performative Authoring: Nurturing Children’s Creativity and Creative Self-Efficacy through Digitally-Augmented Enactment-Based Storytelling

Psychological research, especially by Paul Torrance, has shown that the child’s engagement in creative activities tends to drop precipitously at around the 3rd- to 4th-grade period (8 to 11 years old). This phenomenon, called the ‘Fourth-Grade Slump’, occurs possibly because of an increase in social awareness and critical self-evaluation of competence in the child during this period. Increasing awareness of the complexity of the world’s problems, new paradigms of design focusing on the user, and advances in technology has led to rapid developments in the design and development of tools to support children’s creativity. Research in creativity support tools has generally focused on augmenting creative performance within specific tasks, using strategies such as facilitating access to information, or exposing the user to a space of possible combinations. Much less studied however, is how tools may help to develop positive attitudes towards creativity in individuals. This is important, especially in systems designed for children where the focus on the development of the person, during critical periods of growth such as the period of the Fourth-Grade Slump, may be said to be of equivalent or greater importance than the support of process or the generation of product. In the domain of storytelling or narrative construction, work in child development, educational pedagogy, social psychology, and performance studies have looked at how to tap into the power of children’s imagination during pretend play to nurture their storytelling abilities and their sense of self-efficacy or confidence. These interventions typically take the form of drama workshops or classroom roleplaying exercises. While results appear to provide good evidence that drama interventions and theater-based methods have some positive effects on children’s development of narratives, studies have shown mixed results in terms of the effects on children’s self-efficacy. I refer here to self-efficacy in the sense of a child’s perception of her creative abilities, in other words, her belief that she can produce creative outcomes. This creativity-oriented sense of self-efficacy has been called ‘creative self-efficacy’. This dissertation investigates how pretend play can be harnessed into the design of an interface to support children’s creativity in storytelling and their sense of creative self-efficacy. This overarching question was explored through four phases of research: Exploration, Design, Evaluation, and Integration. The Exploration phase consisted of two studies: a) a set of interviews with elementary school teachers, and b) an experimental study of how the interface or medium may affect children’s creative storytelling process; The Design phase consisted of two experimental studies, and design and development: a) the first study investigated how the physicality of props may support children’s enactment-based storytelling, and b) the second study explored the influence of the presentation of digital contextual/environmental cues on children’s enactment-based storytelling, c) design and development consisted of an exercise using the NEVO methodology to embody design knowledge gained from the Design phase into a concrete usable system, called DiME; The Evaluation phase consisted of two studies: a) the first was a pilot study that tested the usability of DiME and protocol of use with children, and b) the second was an experimental study across two school districts with different profiles investigating the effects of digitally-augmented enactment-based storytelling using DiME, on children’s creativity, story writing and creative self-efficacy; The Integration phase of the research consisted of a workshop with elementary school teachers, which initiated an exploration into how such a story authoring approach may be used in an elementary school curriculum and setting. The body of work that this dissertation presents elucidates (i) a physical enactment-based method for the authoring of stories by children, and (ii) how a digitally-augmented space may move beyond simple drama methods to positively influence the child’s creativity and imagination during storytelling, as well as her self-belief and motivation to engage in creation. The digitally-augmented enactment-based storytelling environment, that I term performative authoring, allows the child to collaboratively create a story through pretend play with a partner, while her enactments are reflected in real-time in the form of animated cartoon characters and objects on a large screen display through the use of motion tracking technologies. I have found that performative authoring has positive effects not only on the child’s creative self-efficacy, especially for the less extraverted children, but also on the richness of the child’s retelling or written narrative of her story. The significance of the results of the studies is with respect to the various domains and subareas represented (child-computer interaction, interactive storytelling, education and educational psychology, creativity and cognition). There is great potential to extend the concept of exploiting digitally-augmented enactment to support and scaffold higher-level cognition, beyond physical enactment. Extensions of this work include making use of more epistemic forms of enactment, instead of full-blown enactment, to support children’s creative story brainstorming, or to make use of digitally-augmented enactment to support other forms of higher thought apart from creativity and imagination. In the domain of storytelling or narrative construction, work in child development, educational pedagogy, social psychology, and performance studies have looked at how to tap into the power of children’s imagination during pretend play to nurture their storytelling abilities and their sense of self-efficacy or confidence. These interventions typically take the form of drama workshops or classroom roleplaying exercises. While results appear to provide good evidence that drama interventions and theater-based methods have some positive effects on children’s development of narratives, studies have shown mixed results in terms of the effects on children’s self-efficacy. I refer here to self-efficacy in the sense of a child’s perception of her creative abilities, in other words, her belief that she can produce creative outcomes. This creativity-oriented sense of self-efficacy has been called ‘creative self-efficacy’ (Beghetto, 2006, 2007). This dissertation investigates how pretend play can be harnessed into the design of an interface to support children’s creativity in storytelling and their sense of creative self-efficacy. This overarching question was explored through four phases of research: I. Exploration, II. Design, III. Evaluation, and IV. Integration. Phase 1 Exploration consisted of two studies: 1A) a set of interviews with elementary school teachers, and 1B) an experimental study of how the interface or medium may affect children’s creative storytelling process; Phase 2 Design consisted of two experimental studies, and design and development: 2A) the first study investigates how the physicality of props may support children’s enactment-based storytelling, and 2B) the second study explores the influence of the presentation of digital contextual/environmental cues on children’s enactment-based storytelling, 2C) design and development consisted of an exercise using the NEVO methodology to embody design knowledge gained from Phase 2 into a concrete usable system, called DiME; Phase 3 Evaluation consisted of two studies: 3A) the first was a pilot study that tested the usability of DiME and protocol of use with children, and 3B) the second was an experimental study across two school districts with different profiles investigating the effects of digitally-augmented enactment-based storytelling using DiME, on children’s creativity, story writing and creative self-efficacy; and Phase 4 Integration consisted of a workshop with elementary school teachers, which initiated an exploration into how such a story authoring approach may be used in an elementary school curriculum and setting. The body of work that this dissertation presents elucidates (i) a physical enactment-based method for the authoring of stories by children, and (ii) how a digitally-augmented space may move beyond simple drama methods to positively influence the child’s creativity and imagination during storytelling, as well as her self-belief and motivation to engage in creation. The digitally-augmented enactment-based storytelling environment, termed performative authoring in this document, allows the child to collaboratively create a story through pretend play with a partner, while her enactments are reflected in real-time in the form of animated cartoon characters and objects on a large screen display through the use of motion tracking technologies. I have found that performative authoring has positive effects not only on the child’s creative self- efficacy, especially for the less extraverted children, but also on the richness of the child’s retelling or written narrative of her story. This dissertation concludes by discussing the significance of the results of our studies with respect to the various domains and subareas represented (child-computer interaction, interactive storytelling, education and educational psychology, creativity and cognition) and extends the concept of exploiting digitally-augmented enactment to support and scaffold higher-level cognition, beyond physical enactment. Extensions of this work include making use of more epistemic forms of enactment, instead of full-blown enactment, to support children’s creative story brainstorming, or to make use of digitally-augmented enactment to support other forms of higher thought apart from creativity and imagination

Designing Digital Art and Communication Tools Inspired by Traditional Craft

Ph.DDOCTOR OF PHILOSOPH