3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance

This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil

MotionDesigner: a tool for creating interactive performances using RGB-D cameras

In the last two decades the use of technology in art projects has proliferated, as is the case of the interactive projections based on movement used in art performances and installations. However, the artists responsible for creating this work typically have to rely on computer experts to implement this type of interactive systems. The tool herein presented, MotionDesigner, intends to assist the role of the artistic creator in the design of these systems, allowing them to have autonomy and efficiency during the creative process of their own works. The proposed tool has a design oriented to these users so that it stimulates and proliferates their work, having an extensible nature, in the way that more content may be added further in the future. The developed software was tested with dancers, choreographers and architects, revealing itself as an aid and catalyst of the creative process.Desde há duas décadas que o uso da tecnologia em projetos artísticos tem proliferado cada vez mais, como é o caso das projeções interativas baseadas em movimento utilizadas em performances e instalações artísticas. No entanto, os artistas responsáveis pela criação destes trabalhos têm, tipicamente, de recorrer a especialistas em computadores para implementar este tipo de sistemas interativos. A ferramenta apresentada nesta dissertação, o MotionDesigner, pretende auxiliar o papel do criador artístico na conceção destes sistemas, permitindo que haja autonomia e eficiência no processo criativo das suas próprias obras. A ferramenta proposta possui um design orientado para estes utilizadores de modo a estimular e agilizar a criação autónoma deste tipo de obras e tem uma natureza extensível, na medida em que mais conteúdo pode ser adicionado no futuro. O software desenvolvido foi testado com bailarinos, coreógrafos e arquitetos, revelando-se como uma ajuda e um catalisador do processo criativo da suas obras interativas

Aesthetic potential of human-computer interaction in performing arts

Human-computer interaction (HCI) is a multidisciplinary area that studies the communication between users and computers. In this thesis, we want to examine if and how HCI when incorporated into staged performances can generate new possibilities for artistic expression on stage. We define and study four areas of technology-enhanced performance that were strongly influenced by HCI techniques: multimedia expression, body representation, body augmentation and interactive environments. We trace relevant artistic practices that contributed to the exploration of these topics and then present new forms of creative expression that emerged after the incorporation of HCI techniques. We present and discuss novel practices like: performer and the media as one responsive entity, real-time control of virtual characters, on-body projections, body augmentation through humanmachine systems and interactive stage design. The thesis concludes by showing some concrete examples of these novel practices implemented in performance pieces. We present and discuss technologyaugmented dance pieces developed during this master’s degree. We also present a software tool for aesthetic visualisation of movement data and discuss its application in video creation, staged performances and interactive installations

Networks of Liveness in Singer-Songwriting: A practice-based enquiry into developing audio-visual interactive systems and creative strategies for composition and performance.

This enquiry explores the creation and use of computer-based, real-time interactive audio-visual systems for the composition and performance of popular music by solo artists. Using a practice-based methodology, research questions are identified that relate to the impact of incorporating interactive systems into the songwriting process and the liveness of the performances with them. Four approaches to the creation of interactive systems are identified: creating explorative-generative tools, multiple tools for guitar/vocal pieces, typing systems and audio-visual metaphors. A portfolio of ten pieces that use these approaches was developed for live performance. A model of the songwriting process is presented that incorporates system-building and strategies are identified for reconciling the indeterminate, electronic audio output of the system with composed popular music features and instrumental/vocal output. The four system approaches and ten pieces are compared in terms of four aspects of liveness, derived from current theories. It was found that, in terms of overall liveness, a unity to system design facilitated both technological and aesthetic connections between the composition, the system processes and the audio and visual outputs. However, there was considerable variation between the four system approaches in terms of the different aspects of liveness. The enquiry concludes by identifying strategies for maximising liveness in the different system approaches and discussing the connections between liveness and the songwriting process

ZATLAB : recognizing gestures for artistic performance interaction

Most artistic performances rely on human gestures, ultimately resulting in an elaborate interaction between the performer and the audience. Humans, even without any kind of formal analysis background in music, dance or gesture are typically able to extract, almost unconsciously, a great amount of relevant information from a gesture. In fact, a gesture contains so much information, why not use it to further enhance a performance? Gestures and expressive communication are intrinsically connected, and being intimately attached to our own daily existence, both have a central position in our (nowadays) technological society. However, the use of technology to understand gestures is still somehow vaguely explored, it has moved beyond its first steps but the way towards systems fully capable of analyzing gestures is still long and difficult (Volpe, 2005). Probably because, if on one hand, the recognition of gestures is somehow a trivial task for humans, on the other hand, the endeavor of translating gestures to the virtual world, with a digital encoding is a difficult and illdefined task. It is necessary to somehow bridge this gap, stimulating a constructive interaction between gestures and technology, culture and science, performance and communication. Opening thus, new and unexplored frontiers in the design of a novel generation of multimodal interactive systems. This work proposes an interactive, real time, gesture recognition framework called the Zatlab System (ZtS). This framework is flexible and extensible. Thus, it is in permanent evolution, keeping up with the different technologies and algorithms that emerge at a fast pace nowadays. The basis of the proposed approach is to partition a temporal stream of captured movement into perceptually motivated descriptive features and transmit them for further processing in Machine Learning algorithms. The framework described will take the view that perception primarily depends on the previous knowledge or learning. Just like humans do, the framework will have to learn gestures and their main features so that later it can identify them. It is however planned to be flexible enough to allow learning gestures on the fly. This dissertation also presents a qualitative and quantitative experimental validation of the framework. The qualitative analysis provides the results concerning the users acceptability of the framework. The quantitative validation provides the results about the gesture recognizing algorithms. The use of Machine Learning algorithms in these tasks allows the achievement of final results that compare or outperform typical and state-of-the-art systems. In addition, there are also presented two artistic implementations of the framework, thus assessing its usability amongst the artistic performance domain. Although a specific implementation of the proposed framework is presented in this dissertation and made available as open source software, the proposed approach is flexible enough to be used in other case scenarios, paving the way to applications that can benefit not only the performative arts domain, but also, probably in the near future, helping other types of communication, such as the gestural sign language for the hearing impaired.Grande parte das apresentações artísticas são baseadas em gestos humanos, ultimamente resultando numa intricada interação entre o performer e o público. Os seres humanos, mesmo sem qualquer tipo de formação em música, dança ou gesto são capazes de extrair, quase inconscientemente, uma grande quantidade de informações relevantes a partir de um gesto. Na verdade, um gesto contém imensa informação, porque não usá-la para enriquecer ainda mais uma performance? Os gestos e a comunicação expressiva estão intrinsecamente ligados e estando ambos intimamente ligados à nossa própria existência quotidiana, têm uma posicão central nesta sociedade tecnológica actual. No entanto, o uso da tecnologia para entender o gesto está ainda, de alguma forma, vagamente explorado. Existem já alguns desenvolvimentos, mas o objetivo de sistemas totalmente capazes de analisar os gestos ainda está longe (Volpe, 2005). Provavelmente porque, se por um lado, o reconhecimento de gestos é de certo modo uma tarefa trivial para os seres humanos, por outro lado, o esforço de traduzir os gestos para o mundo virtual, com uma codificação digital é uma tarefa difícil e ainda mal definida. É necessário preencher esta lacuna de alguma forma, estimulando uma interação construtiva entre gestos e tecnologia, cultura e ciência, desempenho e comunicação. Abrindo assim, novas e inexploradas fronteiras na concepção de uma nova geração de sistemas interativos multimodais . Este trabalho propõe uma framework interativa de reconhecimento de gestos, em tempo real, chamada Sistema Zatlab (ZtS). Esta framework é flexível e extensível. Assim, está em permanente evolução, mantendo-se a par das diferentes tecnologias e algoritmos que surgem num ritmo acelerado hoje em dia. A abordagem proposta baseia-se em dividir a sequência temporal do movimento humano nas suas características descritivas e transmiti-las para posterior processamento, em algoritmos de Machine Learning. A framework descrita baseia-se no facto de que a percepção depende, principalmente, do conhecimento ou aprendizagem prévia. Assim, tal como os humanos, a framework terá que aprender os gestos e as suas principais características para que depois possa identificá-los. No entanto, esta está prevista para ser flexível o suficiente de forma a permitir a aprendizagem de gestos de forma dinâmica. Esta dissertação apresenta também uma validação experimental qualitativa e quantitativa da framework. A análise qualitativa fornece os resultados referentes à aceitabilidade da framework. A validação quantitativa fornece os resultados sobre os algoritmos de reconhecimento de gestos. O uso de algoritmos de Machine Learning no reconhecimento de gestos, permite a obtençãoc¸ ˜ao de resultados finais que s˜ao comparaveis ou superam outras implementac¸ ˜oes do mesmo g´enero. Al ´em disso, s˜ao tamb´em apresentadas duas implementac¸ ˜oes art´ısticas da framework, avaliando assim a sua usabilidade no dom´ınio da performance art´ıstica. Apesar duma implementac¸ ˜ao espec´ıfica da framework ser apresentada nesta dissertac¸ ˜ao e disponibilizada como software open-source, a abordagem proposta ´e suficientemente flex´ıvel para que esta seja usada noutros cen´ arios. Abrindo assim, o caminho para aplicac¸ ˜oes que poder˜ao beneficiar n˜ao s´o o dom´ınio das artes performativas, mas tamb´em, provavelmente num futuro pr ´oximo, outros tipos de comunicac¸ ˜ao, como por exemplo, a linguagem gestual usada em casos de deficiˆencia auditiva

The big screen centre stage: a director's analysis of intermedia theatre production

A thesis submitted to the University of Bedfordshire in partial fulfillment of the requirements for the degree of Masters by ResearchThe proposed research investigates the analysis of the use of the moving image within contemporary theatre performance making, with specific reference to the director's role. Moving from the realisation that to date there are no academically accepted models used to assess the impact of the moving image on various aspects of theatre making, the research investigate the history of the problem, relating methodologies and key literature associated with the topic. Two key aspects of the theory of 'liveness' are considered and three case studies of the author's own intermedia production are inititally assessed. One case study, The Master Builder, in then analysed in greater depth using the theory of liveness as a tool. Conclusions reveal that the analysis of practice alongside theory facilitated the director in identifying causes to the problems encountered during production which had impacted craft and had been highlighted in the initial assessment. the work has been written by an intermedia theatre director, and as such it aims to inform her own practice and understanding on future productions

Actor and technology : appropriating digital media into the creative process

Réfléchissant à l'utilisation croissante de la technologie sur la scène contemporaine, ce mémoire-création a pour but d'analyser l'incorporation des technologies numériques dans le jeu de l'acteur-créateur. La recherche est divisée en trois parties examinant les aspects historiques, théoriques et pratiques, ce qui permet un regard plus global sur le sujet. D'abord, le mémoire présente les histoires parallèles de la technologie sur scène et du jeu d'acteur au XXe siècle. Il est constaté que leurs chemins sont restés assez indépendants au début du siècle, se croisant de plus en plus lors des années 1960s et 1970s. Vers la fin du XXe siècle, par contre, les recherches sur le jeu d'acteur ont diminué, alors que l'intérêt est resté fort pour les laboratoires technologiques. Suite à cet aperçu historique, le mémoire examine le travail d'un praticien et théoricien américain, David Saltz, qui explore l'utilisation de technologies variées sur scène. Saltz a suggéré une taxonomie de 12 rôles technologiques pour classifier les différentes utilisations des technologies au théâtre, ce qui aide à mieux cerner les recherches portant sur la technologie. La taxonomie de ce mémoire a été utilisée pour encadrer une exploration pratique en utilisant un seul élément technologique : la projection. Adoptant une approche basée sur un théâtre corporel - notamment par les techniques du masque neutre et du mime - le laboratoire a exploré le rôle des costumes interactifs par le biais des projections. Les résultats du travail pratique ont été présentés lors d'une conférence-démonstration, dans un contexte de répétition pour mettre l'accent sur le processus de création. Les techniques présentées portèrent sur l'installation de l'équipement, sur la sélection des projections, ainsi que sur le mouvement de l'acteur. Le mémoire-création conclut en reliant les trois parties de la recherche, suggérant d'autres possibilités d'exploration et affirmant l'intérêt de relier les recherches du jeu d'acteur avec celles de la technologie.\ud ______________________________________________________________________________ \ud MOTS-CLÉS DE L’AUTEUR : Technologies numériques, acteur-créateur, jeu d'acteur, projections, costumes interactifs, masque neutre, mime, interdisciplinarité, intermédialit

Designing performance systems for audience inclusion

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 154-168).We define the concept of the Hyperaudience and a unique approach towards designing real-time interactive performance systems: the design of these systems encourages audience participation and augments the experience of audience members through interconnected networks. In doing so, it embraces concepts found in ubiquitous computing, affective computing, interactive arts, music, theatrical tradition, and pervasive gaming. In addition, five new systems are demonstrated to develop a framework for thinking about audience participation and orchestrating social co-presence in and beyond the performance space. Finally, the principles and challenges that shaped the design of these five systems are defined by measuring, comparing, and evaluating their expressiveness and communicability.by Akito Van Troyer.S.M