3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance

This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil

Synthetic worlds, synthetic strategies: attaining creativity in the metaverse

This text will attempt to delineate the underlying theoretical premises and the definition of the output of an immersive learning approach pertaining to the visual arts to be implemented in online, three dimensional synthetic worlds. Deviating from the prevalent practice of the replication of physical art studio teaching strategies within a virtual environment, the author proposes instead to apply the fundamental tenets of Roy Ascott’s “Groundcourse”, in combination with recent educational approaches such as “Transformative Learning” and “Constructionism”. In an amalgamation of these educational approaches with findings drawn from the fields of Metanomics, Ludology, Cyberpsychology and Presence Studies, as well as an examination of creative practices manifest in the metaverse today, the formulation of a learning strategy for creative enablement unique to online, three dimensional synthetic worlds; one which will focus upon “Play” as well as Role Play, virtual Assemblage and the visual identity of the avatar within the pursuits, is being proposed in this chapter

Synthetic worlds, synthetic strategies: attaining creativity in the metaverse

This text will attempt to delineate the underlying theoretical premises and the definition of the output of an immersive learning approach pertaining to the visual arts to be implemented in online, three dimensional synthetic worlds. Deviating from the prevalent practice of the replication of physical art studio teaching strategies within a virtual environment, the author proposes instead to apply the fundamental tenets of Roy Ascott’s “Groundcourse”, in combination with recent educational approaches such as “Transformative Learning” and “Constructionism”. In an amalgamation of these educational approaches with findings drawn from the fields of Metanomics, Ludology, Cyberpsychology and Presence Studies, as well as an examination of creative practices manifest in the metaverse today, the formulation of a learning strategy for creative enablement unique to online, three dimensional synthetic worlds; one which will focus upon “Play” as well as Role Play, virtual Assemblage and the visual identity of the avatar within the pursuits, is being proposed in this chapter

From ‘hands up’ to ‘hands on’: harnessing the kinaesthetic potential of educational gaming

Traditional approaches to distance learning and the student learning journey have focused on closing the gap between the experience of off-campus students and their on-campus peers. While many initiatives have sought to embed a sense of community, create virtual learning environments and even build collaborative spaces for team-based assessment and presentations, they are limited by technological innovation in terms of the types of learning styles they support and develop. Mainstream gaming development – such as with the Xbox Kinect and Nintendo Wii – have a strong element of kinaesthetic learning from early attempts to simulate impact, recoil, velocity and other environmental factors to the more sophisticated movement-based games which create a sense of almost total immersion and allow untethered (in a technical sense) interaction with the games’ objects, characters and other players. Likewise, gamification of learning has become a critical focus for the engagement of learners and its commercialisation, especially through products such as the Wii Fit. As this technology matures, there are strong opportunities for universities to utilise gaming consoles to embed levels of kinaesthetic learning into the student experience – a learning style which has been largely neglected in the distance education sector. This paper will explore the potential impact of these technologies, to broadly imagine the possibilities for future innovation in higher education

A multi-modal dance corpus for research into real-time interaction between humans in online virtual environments

We present a new, freely available, multimodal corpus for research into, amongst other areas, real-time realistic interaction between humans in online virtual environments. The specific corpus scenario focuses on an online dance class application scenario where students, with avatars driven by whatever 3D capture technology are locally available to them, can learn choerographies with teacher guidance in an online virtual ballet studio. As the data corpus is focused on this scenario, it consists of student/teacher dance choreographies concurrently captured at two different sites using a variety of media modalities, including synchronised audio rigs, multiple cameras, wearable inertial measurement devices and depth sensors. In the corpus, each of the several dancers perform a number of fixed choreographies, which are both graded according to a number of specific evaluation criteria. In addition, ground-truth dance choreography annotations are provided. Furthermore, for unsynchronised sensor modalities, the corpus also includes distinctive events for data stream synchronisation. Although the data corpus is tailored specifically for an online dance class application scenario, the data is free to download and used for any research and development purposes

Body parts

In this project, the artist wishes to examine corporeality in the virtual realm, through the usage of the (non)-physical body of the avatar. An art installation created in the virtual world of Second Life®, which is meant to be accessed with site specific avatars, will provide the creative platform whereby this investigation is undertaken. Thus, “body parts” seeks to challenge the residents of virtual environments into connecting with the virtual manifestations, i.e., avatars of others in an emotionally expressive/intimate manner

Weaving Lighthouses and Stitching Stories: Blind and Visually Impaired People Designing E-textiles

We describe our experience of working with blind and visually impaired people to create interactive art objects that are personal to them, through a participatory making process using electronic textiles (e-textiles) and hands-on crafting techniques. The research addresses both the practical considerations about how to structure hands-on making workshops in a way which is accessible to participants of varying experience and abilities, and how effective the approach was in enabling participants to tell their own stories and feel in control of the design and making process. The results of our analysis is the offering of insights in how to run e-textile making sessions in such a way for them to be more accessible and inclusive to a wider community of participants

Transitioning from Transmedia to Transreality Storyboarding to Improve the Co-Creation of the Experience Space

Transmedia storytelling is a digital based marketing approach in present day consumer markets. Typically applied to spanning or segueing stories or experiences across media such as film, books, comics and video-games to reach broader target audiences, often triggering a narrative, into which customers can participate and co-create the narrative. Common aims at customer engagement have been through shared stories on present day social media. However, for the creative-consumer, sharing on social media falls short of fully immersive storytelling ecology. Creatives (traditional designers and consumers) would benefit through tools and processes for incrementally expanding dimensions, mediums, fidelity, and shared interactions and senses across multiple media and interactive realities. This paper presents use cases of Transreality Storyboarding Framework (TSF), a design framework that affords creation of experience spaces for consumer-product engagement. Further, we propose a TSF app, to allow non-expert designers/everyday-consumers to contribute to storytelling, participation and production of product experience spaces