A dramaturgy of intermediality: composing with integrative design

The thesis investigates and develops a compositional system on intermediality in theatre and performance as a dramaturgical practice through integrative design. The position of the visual/sonic media in theatre and performance has been altered by the digitalisation and networking of media technologies, which enables enhanced dynamic variables in the intermedial processes. The emergent intermediality sites are made accessible by developments in media technologies and form part of broader changes towards a mediatised society: a simultaneous shift in cultural contexts, theatre practice and audience perception. The practice-led research is situated within a postdramatic context and develops a system of compositional perspectives and procedures to enhance the knowledge of a dramaturgy on intermediality. The intermediality forms seem to re-situate the actual/virtual relations in theatre and re-construct the processes of theatricalisation in the composition of the stage narrative. The integration of media and performers produces a compositional environment of semiosis, where the theatre becomes a site of narration, and the designed integration in-between medialities emerges as intermediality sites in the performance event. A selection of performances and theatre directors is identified, who each in distinct ways integrate mediating technologies as a core element in their compositional design. These directors and performances constitute a source of reflection on compositional strategies from the perspective of practice, and enable comparative discussions on dramaturgical design and the consistency of intermediality sites. The practice-led research realised a series of prototyping processes situated in performance laboratories in 2004-5. The laboratories staged investigations into the relation between integrative design procedures and parameters for composition of intermediality sites, particularly the relative presence in-between the actual and the virtual, and the relative duration and distance in-between timeness and placeness. The integration of performer activities and media operations into dramaturgical structures were developed as a design process of identifying the mapping and experiencing the landscape through iterative prototyping. The developed compositional concepts and strategies were realised in the prototype performance Still I Know Who I Am, performed October 2006. This final research performance was a full-scale professional production, which explored the developed dramaturgical designs through creative practice. The performance was realised as a public event, and composed of a series of scenes, each presenting a specific composite of the developed integrative design strategies, and generating a particular intermediality site. The research processes in the performance laboratories and the prototype performance developed on characteristics, parameters and procedures of compositional strategies, investigating the viability of a dramaturgy of intermediality through integrative design. The practice undertaken constitutes raw material from which the concepts are drawn and underpins the premises for the theoretical reflections

Computational Multimedia for Video Self Modeling

Video self modeling (VSM) is a behavioral intervention technique in which a learner models a target behavior by watching a video of oneself. This is the idea behind the psychological theory of self-efficacy - you can learn or model to perform certain tasks because you see yourself doing it, which provides the most ideal form of behavior modeling. The effectiveness of VSM has been demonstrated for many different types of disabilities and behavioral problems ranging from stuttering, inappropriate social behaviors, autism, selective mutism to sports training. However, there is an inherent difficulty associated with the production of VSM material. Prolonged and persistent video recording is required to capture the rare, if not existed at all, snippets that can be used to string together in forming novel video sequences of the target skill. To solve this problem, in this dissertation, we use computational multimedia techniques to facilitate the creation of synthetic visual content for self-modeling that can be used by a learner and his/her therapist with a minimum amount of training data. There are three major technical contributions in my research. First, I developed an Adaptive Video Re-sampling algorithm to synthesize realistic lip-synchronized video with minimal motion jitter. Second, to denoise and complete the depth map captured by structure-light sensing systems, I introduced a layer based probabilistic model to account for various types of uncertainties in the depth measurement. Third, I developed a simple and robust bundle-adjustment based framework for calibrating a network of multiple wide baseline RGB and depth cameras

Augmented Reality to Engage Visitors of Science Museums through Interactive Experiences

In the last years, interactive exhibitions based on digital technologies have become widely common, thanks to their flexibility and effectiveness in engaging visitors and creating memorable experiences. One of the topics in which digital technologies can be particularly effective is the communication of abstract concepts that are difficult for the human mind to imagine. An emblematic example is the astronomy discipline, which requires us to imagine and understand phenomena far away from our everyday life. In this paper, the authors present a research project, MARSS, in which digital technologies are used effectively to enhance the Users' Experience of the Museo Astronomico di Brera located in Milan. Specifically, the MARSS project aims at designing and developing a new digital journey inside the museum to allow different categories of visitors to enjoy the exhibition in an engaging and interactive way. The paper presents the design and development phases of the experience and its evaluation with users. The results of the evaluation indicate that the digital interactive experience is appreciated by users and is successful in translating the content of high scientific value into more engaging and easily understandable elements

How to Build an Embodiment Lab: Achieving Body Representation Illusions in Virtual Reality

Advances in computer graphics algorithms and virtual reality (VR) systems, together with the reduction in cost of associated equipment, have led scientists to consider VR as a useful tool for conducting experimental studies in fields such as neuroscience and experimental psychology. In particular virtual body ownership, where the feeling of ownership over a virtual body is elicited in the participant, has become a useful tool in the study of body representation, in cognitive neuroscience and psychology, concerned with how the brain represents the body. Although VR has been shown to be a useful tool for exploring body ownership illusions, integrating the various technologies necessary for such a system can be daunting. In this paper we discuss the technical infrastructure necessary to achieve virtual embodiment. We describe a basic VR system and how it may be used for this purpose, and then extend this system with the introduction of real-time motion capture, a simple haptics system and the integration of physiological and brain electrical activity recordings

Design and Analysis of a Single-Camera Omnistereo Sensor for Quadrotor Micro Aerial Vehicles (MAVs)

We describe the design and 3D sensing performance of an omnidirectional stereo (omnistereo) vision system applied to Micro Aerial Vehicles (MAVs). The proposed omnistereo sensor employs a monocular camera that is co-axially aligned with a pair of hyperboloidal mirrors (a vertically-folded catadioptric configuration). We show that this arrangement provides a compact solution for omnidirectional 3D perception while mounted on top of propeller-based MAVs (not capable of large payloads). The theoretical single viewpoint (SVP) constraint helps us derive analytical solutions for the sensor’s projective geometry and generate SVP-compliant panoramic images to compute 3D information from stereo correspondences (in a truly synchronous fashion). We perform an extensive analysis on various system characteristics such as its size, catadioptric spatial resolution, field-of-view. In addition, we pose a probabilistic model for the uncertainty estimation of 3D information from triangulation of back-projected rays. We validate the projection error of the design using both synthetic and real-life images against ground-truth data. Qualitatively, we show 3D point clouds (dense and sparse) resulting out of a single image captured from a real-life experiment. We expect the reproducibility of our sensor as its model parameters can be optimized to satisfy other catadioptric-based omnistereo vision under different circumstances

A Digital Twin City Model for Age-Friendly Communities: Capturing Environmental Distress from Multimodal Sensory Data

As the worldwide population is aging, the demands of aging-in-place are also increasing and require smarter and more connected cities to keep mobility independence of older adults. However, today’s aging built environment often poses great environmental demands to older adults’ mobility and causes their distresses. To better understand and help mitigating older adults’ distress in their daily trips, this paper proposes constructing the digital twin city (DTC) model that integrates multimodal data (i.e., physiological sensing, visual sensing) on environmental demands in urban communities, so that such environmental demands can be considered in mobility planning of older adults. Specifically, this paper examines how data acquired from various modalities (i.e., electrodermal activity, gait patterns, visual sensing) can portray environmental demands associated with older adults’ mobility. In addition, it discusses the challenges and opportunities of multimodal data fusion in capturing environmental distresses in urban communities