Shadows, touch and digital puppeteering: a media archaeological approach

Aims The practical aim of this research project is to create a multi-touch digital puppetry system that simulates shadow theatre environments and translates gestural acts of touch into live and expressive control of virtual shadow figures. The research is focussed on the qualities of movement achievable through the haptics of single and multi-touch control of the digital puppets in the simulation. An associated aim is to create a collaborative environment where multiple performers can control dynamic animation and scenography, and create novel visualisations and narratives. The conceptual aim is to link traditional and new forms of puppetry seeking cultural significance in the ‘remediation’ of old forms that avail themselves of new haptic resources and collaborative interfaces. The thesis evaluates related prior art where traditional worlds of shadow performance meet new media, digital projection and 3D simulation, in order to investigate how changing technical contexts transform the potential of shadows as an expressive medium. Methodology The thesis uses cultural analysis of relevant documentary material to contextualise the practical work by relating the media archaeology of 2D puppetry—shadows, shadowgraphs and silhouettes—to landmark work in real-time computer graphics and performance animation. The survey considers the work of puppeteers, animators, computer graphics specialists and media artists. Through practice and an experimental approach to critical digital creativity, the study provides practical evidence of multiple iterations of controllable physics-based animation delivering expressive puppet motion through touch and multiuser interaction. Video sequences of puppet movement and written observational analysis document the intangible aspects of animation in performance. Through re-animation of archival shadow puppets, the study presents an emerging artistic media archaeological method. The major element of this method has been the restoration of a collection of Turkish Karagöz Shadow puppets from the Institut International de la Marionnette (Charleville, France) into a playable digital form. Results The thesis presents a developing creative and analytical framework for digital shadow puppetry. It proposes a media archaeological method for working creatively with puppet archives that unlock the kinetic and expressive potential of restored figures. The interaction design introduces novel approaches to puppetry control systems—using spring networks—with objects under physics-simulation that demonstrate emergent expressive qualities. The system facilitates a dance of agency¹ between puppeteer and digital instrument. The practical elements have produced several software iterations and a tool-kit for generating elegant, nuanced multi-touch shadow puppetry. The study presents accidental discoveries—serendipitous benefits of open-ended practical exploration. For instance: the extensible nature of the control system means novel input—other than touch—can provide exciting potential for accessible user interaction, e.g. with gaze duration and eye direction. The study also identifies limitations including the rate of software change and obsolescence, the scope of physics-based animation and failures of simulation. Originality/value The work has historical value in that it documents and begins a media archaeology of digital puppetry, an animated phenomenon of increasing academic and commercial interest. The work is of artistic value providing an interactive approach to making digital performance from archival material in the domain of shadow theatre. The work contributes to the electronic heritage of existing puppetry collections. The study establishes a survey of digital puppetry, setting a research agenda for future studies. Work may proceed to digitise, rig and create collaborative and web-mediated touch-based motion control systems for 2D and 3D puppets. The present study thus provides a solid platform to restore past performances and create new work from old, near forgotten-forms. ¹ Following Andrew Pickering, puppetry is ‘a temporally extended back-and-forth dance of human and non-human agency in which activity and passivity on both sides are reciprocally intertwined’ PICKERING, A. 2010. Material Culture and the Dance of Agency. In: BEAUDRY, M. C. & HICKS, D. (eds.) Oxford Handbook of Material Culture Studies. Oxford University Press.

Deus Ex Machinima: A Rhetorical Analysis of User-Generated Machinima

Beginning with corporate demonstrations and continuously evolving into today, machinima has become a major expressive art form for the gamer generation. Machinima is the user-centered production of video presentations using pre-rendered animated content, as generated from video games. The term \u27machinima\u27 is a combination of \u27machine\u27 (from which the video content is derived) and \u27cinema\u27 (the ultimate end product). According to Paul Marino and other members of the machinima community, Hugh Hancock, the creator of Machinima.com, first coined the term in 2000. Video productions of this kind have been used in various capacities for the past several years, including instruction or marketing, as well as rapid prototyping of large-scale cinema projects (Marino). In this thesis, I will briefly outline the current research on machinima. I will then build a methodology for my own rhetorical analysis of machinima as they formulate the promotion of their arguments. This methodology will include examples from major rhetorical theorists, including Lloyd Bitzer, Kenneth Burke, and Gunther Kress and Theo VanLeeuwan, among others. I will then apply my analytical tools to modern user-generated machinima from a variety of sources as a series of case studies. These cases include non-profit and for-profit examples, as well as educational and entertainment examples. Finally, I will explain how this framework may be used as a guideline for rhetorically sound and effective machinima

The Quest for Life and Intelligence in Digital Puppets.

Performers and artists wishing to make collaborative improvisations using three-dimensional computer graphics will encounter the following difficulty: the animation process lacks the functionality required for spontaneous, serendipitous, real-time interaction. While human motion capture makes such real-time interaction and the corresponding spontaneity possible, it lacks the expressivity required for non-realistic characterisation. My practice-based research study proposes digital puppetry as a solution to this problem. My focus is on credibility as opposed to realism: the expectation is that the characters will behave in a puppet-like manner as opposed to manifesting the aesthetics – in terms of both movement and visual design – found in human actor-based motion capture and three- dimensional animated cartoons. The practical element is motivated by three imperatives: to improvise, to collaborate and to perform live. The primary question this study sets out to answer is: Is it possible to achieve the same spontaneity and animation[1] in digital puppets as it is with tangible puppets and, if so, what are the mechanisms involved? Unable to find a three-dimensional computer graphics digital puppetry software and hardware solution, I have devised what I call the GLOPPID[2] method, where GLOPPID is an acronym for Goniometric[3] Live Organic Performance Puppetry Improvisation Digitalia. The GLOPPID method comprises an artistic approach and a practical solution in the form of a Human Digital Puppetry Interface. It uses off-the-shelf three-dimensional computer animation software, which I have incorporated into a pipeline customised to suit my creative process. This pipeline is configured to transform ready-made computer graphics models into digital puppets that can be used as collaborators, thereby allowing the performer to experience the same kind of spontaneity as is possible in physical puppet performance. My thesis asserts that it is possible to improvise with digital puppets, and I have devised my own solution in order to do this. I argue that the real-time, improvised manipulation of digital puppets offers creatively advantageous opportunities for spontaneity and expressivity. My research presents the technique of digital puppetry as an expansion of what I call the pro- puppetry thesis – the idea that puppets have dramatic advantages over human actors. It also contributes to the ontological discourses surrounding the Human-Machine Interface (HMI), trans-embodiment, the post-human, the illusion of life, and cybernetics. In addition, it explores how algorithms can be used in the arts, particularly in performance (see Kleber & Trojanowska 2019, p.101). It makes a timely contribution to the pool of knowledge, because I see digital puppets as zeitgeists – apt vehicles for human hopes and fears surrounding the digital and existential angst that is part of the fabric of 21st-century life. The professional digital puppetry practice undertaken and discussed in this study requires both general, transferable human-machine interacting skills, and the specific digital puppetry skills necessary for project phases such as rigging and manipulation. This practical approach prioritises the physical, as opposed to the psycho-physical. Informed by the theories and practice of human dramatic technique practitioners such as Decroux and Lecoq, it takes the basic building blocks of movement identifiable in the segmented anatomy and rotational articulations of the actor-as-puppet, rod, glove and string puppets, and configures them in their digital counterpart in order to accentuate the odd by means of atypical combinations, economy of motion, and asymmetry. My approach is underpinned by the idea that a puppet is uninhibited or influenced by its own ego, backstory, or emotions. Nevertheless, these properties are present in a channeled, and therefore changed, form that emanates from the Human in the Loop, namely, the digital puppeteer. In digital puppetry, the protagonist is a digital puppet operated by a human who is embedded at the core of the activity, and who simultaneously witnesses their own emotional responses as they are acted out in front of them. My work demonstrates that non-realistic, expressive approaches to movement performance derived from human physical theatre techniques (including theatre clowning), combined with the use of algorithm-assisted techniques of rigging and manipulation, mean that the puppets are not under the complete control of their puppeteers. Instead, their rogue nature and irrationality enables digital puppets to satirise and subvert notions surrounding computer-generated imagery and artificial general intelligence, while avoiding exile in the “uncanny valley” (Mori 1970). In this study the term puppet is used both literally and metaphorically. Deployed literally, puppet refers to a figure or object that is manipulated in real time in the presence of a puppeteer, in both tangible and intangible material formats. The term is used metaphorically in the sense that a puppet can be seen as an analogue of a human being that acts as a mirror, reflecting aspects of the human condition or predicament. My work expands upon an understanding of the quest for ‘life’ in physical puppets and the corresponding development of their ur-narrative – as described by Kohler & Jones (2009, p.346) – and extends this to include digital puppets. By combining the concept of this quest with Rokeby’s idea of technology as a “prosthetic of philosophy” (2019, p.107) and with the notion of art as a mirror, I explore how distinctive features present in digital puppetry practice can be used to express truths about being human. The work employs a practice-as-research methodology that provides moments for reflection during the creative process, and reflection on the creative outcomes: reflection-in-action and reflection-on-action. The thesis can be expressed in the form of the following formula: Extemporising with puppets + the ur-narrative of puppets + art as a mirror + technology as a mirror = a pro-digital-puppetry thesis on HMI, AGI + what it is to be human [1] The term animation used here refers not to the techniques of incremental, frame-by-frame posing of characters, but to the act of bringing them to life. [2] ‘Gloppid’ is also the name of a glove-puppet character invented for an ecologically-based travelling show performed during the 1980s (Childs 1988). [3] Goniometric refers to the measurement of the range of motion in a joint

Puppets between human, animal and machine: towards the modes of movement contesting the anthropocentric view of life in animation

In this PhD thesis, I challenge animation studies’ conventional notion that animation can bring something inanimate to “life”. This emphasis on animation’s capacity to make a figure appear to move on screen has led to the problematic notion that movement has a synonymous relationship with life. Contesting these discourses, I show in this thesis that not every animated figure suggests the impression of life. In order to prove this, I put forward as a critical focus the puppet-as-puppet figure, that is, the figure of a puppet depicted as a puppet per se in the film diegesis, which problematises the impression of life even if appearing to move on screen. A related focus in my thesis is the mode of movement which functions as a visual and physical parameter in order to analyse what an animated (or static) figure is intended to look like, instead of reducing it to a question of life. Through case studies of these puppet-as-puppet figures, which I classify into four groups, I examine the varying ways in which they are depicted as inanimate or sub/nonhuman, even when in human form, in contrast to human or (anthropomorphic) animal figures, both in terms of their mode of movement as well as their appearance. Examining how these depictions demonstrate anthropocentric views of puppets, I consider religio-philosophical, scientific and aesthetic discourses on puppets and human/animal simulacra. Further, I explore a selection of puppet-as-puppet figures as alternatives to these anthropocentric conventions, examining their defamiliarisation of the animating human subject’s mastery over the animated non/subhuman object, and the non-anthropocentric sensations which their movements arouse on screen in the relationship between humanity and materiality

Computer-Assisted Interactive Documentary and Performance Arts in Illimitable Space

This major component of the research described in this thesis is 3D computer graphics, specifically the realistic physics-based softbody simulation and haptic responsive environments. Minor components include advanced human-computer interaction environments, non-linear documentary storytelling, and theatre performance. The journey of this research has been unusual because it requires a researcher with solid knowledge and background in multiple disciplines; who also has to be creative and sensitive in order to combine the possible areas into a new research direction. [...] It focuses on the advanced computer graphics and emerges from experimental cinematic works and theatrical artistic practices. Some development content and installations are completed to prove and evaluate the described concepts and to be convincing. [...] To summarize, the resulting work involves not only artistic creativity, but solving or combining technological hurdles in motion tracking, pattern recognition, force feedback control, etc., with the available documentary footage on film, video, or images, and text via a variety of devices [....] and programming, and installing all the needed interfaces such that it all works in real-time. Thus, the contribution to the knowledge advancement is in solving these interfacing problems and the real-time aspects of the interaction that have uses in film industry, fashion industry, new age interactive theatre, computer games, and web-based technologies and services for entertainment and education. It also includes building up on this experience to integrate Kinect- and haptic-based interaction, artistic scenery rendering, and other forms of control. This research work connects all the research disciplines, seemingly disjoint fields of research, such as computer graphics, documentary film, interactive media, and theatre performance together.Comment: PhD thesis copy; 272 pages, 83 figures, 6 algorithm

Direct Animation Interfaces : an Interaction Approach to Computer Animation

Creativity tools for digital media have been largely democratised, offering a range from beginner to expert tools. Yet computer animation, the art of instilling life into believable characters and fantastic worlds, is still a highly sophisticated process restricted to the spheres of expert users. This is largely due to the methods employed: in keyframe animation dynamics are indirectly specified over abstract descriptions, while performance animation suffers from inflexibility due to a high technological overhead. The reverse trend in human-computer interaction to make interfaces more direct, intuitive, and natural to use has so far hardly touched the animation world: decades of interaction research have scarcely been linked to research and development of animation techniques. The hypothesis of this work is that an interaction approach to computer animation can inform the design and development of novel animation techniques. Three goals are formulated to illustrate the validity of this thesis. Computer animation methods and interfaces must be embedded in an interaction context. The insights this brings for designing next generation animation tools must be examined and formalised. The practical consequences for the development of motion creation and editing tools must be demonstrated with prototypes that are more direct, efficient, easy-to-learn, and flexible to use. The foundation of the procedure is a conceptual framework in the form of a comprehensive discussion of the state of the art, a design space of interfaces for time-based visual media, and a taxonomy for mappings between user and medium space-time. Based on this, an interaction-centred analysis of computer animation culminates in the concept of direct animation interfaces and guidelines for their design. These guidelines are tested in two point designs for direct input devices. The design, implementation and test of a surface-based performance animation tool takes a system approach, addressing interaction design issues as well as challenges in extending current software architectures to support novel forms of animation control. The second, a performance timing technique, shows how concepts from video browsing can be applied to motion editing for more direct and efficient animation timing

Muscle activation mapping of skeletal hand motion: an evolutionary approach.

Creating controlled dynamic character animation consists of mathe- matical modelling of muscles and solving the activation dynamics that form the key to coordination. But biomechanical simulation and control is com- putationally expensive involving complex di erential equations and is not suitable for real-time platforms like games. Performing such computations at every time-step reduces frame rate. Modern games use generic soft- ware packages called physics engines to perform a wide variety of in-game physical e ects. The physics engines are optimized for gaming platforms. Therefore, a physics engine compatible model of anatomical muscles and an alternative control architecture is essential to create biomechanical charac- ters in games. This thesis presents a system that generates muscle activations from captured motion by borrowing principles from biomechanics and neural con- trol. A generic physics engine compliant muscle model primitive is also de- veloped. The muscle model primitive forms the motion actuator and is an integral part of the physical model used in the simulation. This thesis investigates a stochastic solution to create a controller that mimics the neural control system employed in the human body. The control system uses evolutionary neural networks that evolve its weights using genetic algorithms. Examples and guidance often act as templates in muscle training during all stages of human life. Similarly, the neural con- troller attempts to learn muscle coordination through input motion samples. The thesis also explores the objective functions developed that aids in the genetic evolution of the neural network. Character interaction with the game world is still a pre-animated behaviour in most current games. Physically-based procedural hand ani- mation is a step towards autonomous interaction of game characters with the game world. The neural controller and the muscle primitive developed are used to animate a dynamic model of a human hand within a real-time physics engine environment

Design-led approach for transferring the embodied skills of puppet stop-motion animators into haptic workspaces

This design-led research investigates the transfer of puppet stop-motion animators’ embodied skills from the physical workspace into a digital environment. The approach is to create a digital workspace that evokes an embodied animating experience and allows puppet stop-motion animators to work in it unencumbered. The insights and outcomes of the practical explorations are discussed from the perspective of embodied cognition. The digital workspace employs haptic technology, an advanced multi-modal interface technology capable of invoking the tactile, kinaesthetic and proprioceptive senses. The overall aim of this research is to contribute, to the Human-Computer Interaction design community, design considerations and strategies for developing haptic workspaces that can seamlessly transfer and accommodate the rich embodied knowledge of non-digital skillful practitioners. Following an experiential design methodology, a series of design studies in collaboration with puppet stop-motion animators led to the development of a haptic workspace prototype for producing stop-motion animations. Each design study practically explored the transfer of different aspects of the puppet stop-motion animation practice into the haptic workspace. Beginning with an initial haptic workspace prototype, its design was refined in each study with the addition of new functionalities and new interaction metaphors which were always developed with the aim to create and maintain an embodied animating experience. The method of multiple streams of reflection was proposed as an important design tool for identifying, understanding and articulating design insights, empirical results and contextual considerations throughout the design studies. This thesis documents the development of the haptic workspace prototype and discusses the collected design insights and empirical results from the perspective of embodied cognition. In addition, it describes and reviews the design methodology that was adopted as an appropriate approach towards the design of the haptic workspace prototype

3D Virtual Worlds and the Metaverse: Current Status and Future Possibilities

Moving from a set of independent virtual worlds to an integrated network of 3D virtual worlds or Metaverse rests on progress in four areas: immersive realism, ubiquity of access and identity, interoperability, and scalability. For each area, the current status and needed developments in order to achieve a functional Metaverse are described. Factors that support the formation of a viable Metaverse, such as institutional and popular interest and ongoing improvements in hardware performance, and factors that constrain the achievement of this goal, including limits in computational methods and unrealized collaboration among virtual world stakeholders and developers, are also considered