Pathway to Future Symbiotic Creativity

This report presents a comprehensive view of our vision on the development path of the human-machine symbiotic art creation. We propose a classification of the creative system with a hierarchy of 5 classes, showing the pathway of creativity evolving from a mimic-human artist (Turing Artists) to a Machine artist in its own right. We begin with an overview of the limitations of the Turing Artists then focus on the top two-level systems, Machine Artists, emphasizing machine-human communication in art creation. In art creation, it is necessary for machines to understand humans' mental states, including desires, appreciation, and emotions, humans also need to understand machines' creative capabilities and limitations. The rapid development of immersive environment and further evolution into the new concept of metaverse enable symbiotic art creation through unprecedented flexibility of bi-directional communication between artists and art manifestation environments. By examining the latest sensor and XR technologies, we illustrate the novel way for art data collection to constitute the base of a new form of human-machine bidirectional communication and understanding in art creation. Based on such communication and understanding mechanisms, we propose a novel framework for building future Machine artists, which comes with the philosophy that a human-compatible AI system should be based on the "human-in-the-loop" principle rather than the traditional "end-to-end" dogma. By proposing a new form of inverse reinforcement learning model, we outline the platform design of machine artists, demonstrate its functions and showcase some examples of technologies we have developed. We also provide a systematic exposition of the ecosystem for AI-based symbiotic art form and community with an economic model built on NFT technology. Ethical issues for the development of machine artists are also discussed

Almost Human: The Study of Physical Processes and the Performance of a Prosthetic Digital Spine

Almost Human is an investigation of interdisciplinary performance through music that looks to the self to try to further understand subjective performance practices in expression, gesture and sonic output. This text presents experimental methods of examining and creating music through kinaesthetic and electronic-assisted means within instrumental, dance and interactive works. The extraction of affective, performative and sonic properties from these works aids in unlocking the relationship between the choreographic, physical and conceptual object. The first part of the text explores and illustrates multimodal approaches to analysing, capturing, measuring and archiving the moving musician and dancer in an assortment of performative settings. It focuses on a series of works for solo cello, as well as interdisciplinary pieces which positions movement and embodied expressivity at the forefront of the discussion. The second part is dedicated to the aesthetic, conceptual and utilitarian content of a new interactive work for cellist/mover, and a prosthetic digital spine. Here, relationships are combined to showcase the permeability of the body, as well as its expressive content. The conceptual object, The Spine, serves as a generator to help expand musical and artistic possibilities. Its inclusion in the work aids in refocusing my relationship to movement and sound for creation and performance, but also aesthetically, it adds to the growing canon of experimental ventures in conceptualising expressivity. Beyond the text, the portfolio of Almost Human includes an auditory and visual chronicle of the process between the years 2012-14, which is used to assist the reader in further understanding the performative practice and findings

ZATLAB : recognizing gestures for artistic performance interaction

Most artistic performances rely on human gestures, ultimately resulting in an elaborate interaction between the performer and the audience. Humans, even without any kind of formal analysis background in music, dance or gesture are typically able to extract, almost unconsciously, a great amount of relevant information from a gesture. In fact, a gesture contains so much information, why not use it to further enhance a performance? Gestures and expressive communication are intrinsically connected, and being intimately attached to our own daily existence, both have a central position in our (nowadays) technological society. However, the use of technology to understand gestures is still somehow vaguely explored, it has moved beyond its first steps but the way towards systems fully capable of analyzing gestures is still long and difficult (Volpe, 2005). Probably because, if on one hand, the recognition of gestures is somehow a trivial task for humans, on the other hand, the endeavor of translating gestures to the virtual world, with a digital encoding is a difficult and illdefined task. It is necessary to somehow bridge this gap, stimulating a constructive interaction between gestures and technology, culture and science, performance and communication. Opening thus, new and unexplored frontiers in the design of a novel generation of multimodal interactive systems. This work proposes an interactive, real time, gesture recognition framework called the Zatlab System (ZtS). This framework is flexible and extensible. Thus, it is in permanent evolution, keeping up with the different technologies and algorithms that emerge at a fast pace nowadays. The basis of the proposed approach is to partition a temporal stream of captured movement into perceptually motivated descriptive features and transmit them for further processing in Machine Learning algorithms. The framework described will take the view that perception primarily depends on the previous knowledge or learning. Just like humans do, the framework will have to learn gestures and their main features so that later it can identify them. It is however planned to be flexible enough to allow learning gestures on the fly. This dissertation also presents a qualitative and quantitative experimental validation of the framework. The qualitative analysis provides the results concerning the users acceptability of the framework. The quantitative validation provides the results about the gesture recognizing algorithms. The use of Machine Learning algorithms in these tasks allows the achievement of final results that compare or outperform typical and state-of-the-art systems. In addition, there are also presented two artistic implementations of the framework, thus assessing its usability amongst the artistic performance domain. Although a specific implementation of the proposed framework is presented in this dissertation and made available as open source software, the proposed approach is flexible enough to be used in other case scenarios, paving the way to applications that can benefit not only the performative arts domain, but also, probably in the near future, helping other types of communication, such as the gestural sign language for the hearing impaired.Grande parte das apresentações artísticas são baseadas em gestos humanos, ultimamente resultando numa intricada interação entre o performer e o público. Os seres humanos, mesmo sem qualquer tipo de formação em música, dança ou gesto são capazes de extrair, quase inconscientemente, uma grande quantidade de informações relevantes a partir de um gesto. Na verdade, um gesto contém imensa informação, porque não usá-la para enriquecer ainda mais uma performance? Os gestos e a comunicação expressiva estão intrinsecamente ligados e estando ambos intimamente ligados à nossa própria existência quotidiana, têm uma posicão central nesta sociedade tecnológica actual. No entanto, o uso da tecnologia para entender o gesto está ainda, de alguma forma, vagamente explorado. Existem já alguns desenvolvimentos, mas o objetivo de sistemas totalmente capazes de analisar os gestos ainda está longe (Volpe, 2005). Provavelmente porque, se por um lado, o reconhecimento de gestos é de certo modo uma tarefa trivial para os seres humanos, por outro lado, o esforço de traduzir os gestos para o mundo virtual, com uma codificação digital é uma tarefa difícil e ainda mal definida. É necessário preencher esta lacuna de alguma forma, estimulando uma interação construtiva entre gestos e tecnologia, cultura e ciência, desempenho e comunicação. Abrindo assim, novas e inexploradas fronteiras na concepção de uma nova geração de sistemas interativos multimodais . Este trabalho propõe uma framework interativa de reconhecimento de gestos, em tempo real, chamada Sistema Zatlab (ZtS). Esta framework é flexível e extensível. Assim, está em permanente evolução, mantendo-se a par das diferentes tecnologias e algoritmos que surgem num ritmo acelerado hoje em dia. A abordagem proposta baseia-se em dividir a sequência temporal do movimento humano nas suas características descritivas e transmiti-las para posterior processamento, em algoritmos de Machine Learning. A framework descrita baseia-se no facto de que a percepção depende, principalmente, do conhecimento ou aprendizagem prévia. Assim, tal como os humanos, a framework terá que aprender os gestos e as suas principais características para que depois possa identificá-los. No entanto, esta está prevista para ser flexível o suficiente de forma a permitir a aprendizagem de gestos de forma dinâmica. Esta dissertação apresenta também uma validação experimental qualitativa e quantitativa da framework. A análise qualitativa fornece os resultados referentes à aceitabilidade da framework. A validação quantitativa fornece os resultados sobre os algoritmos de reconhecimento de gestos. O uso de algoritmos de Machine Learning no reconhecimento de gestos, permite a obtençãoc¸ ˜ao de resultados finais que s˜ao comparaveis ou superam outras implementac¸ ˜oes do mesmo g´enero. Al ´em disso, s˜ao tamb´em apresentadas duas implementac¸ ˜oes art´ısticas da framework, avaliando assim a sua usabilidade no dom´ınio da performance art´ıstica. Apesar duma implementac¸ ˜ao espec´ıfica da framework ser apresentada nesta dissertac¸ ˜ao e disponibilizada como software open-source, a abordagem proposta ´e suficientemente flex´ıvel para que esta seja usada noutros cen´ arios. Abrindo assim, o caminho para aplicac¸ ˜oes que poder˜ao beneficiar n˜ao s´o o dom´ınio das artes performativas, mas tamb´em, provavelmente num futuro pr ´oximo, outros tipos de comunicac¸ ˜ao, como por exemplo, a linguagem gestual usada em casos de deficiˆencia auditiva

HIVE Tracker: A tiny, low-cost, and scalable device for sub-millimetric 3D positioning

Positional tracking systems could hugely benefit a number of niches, including performance art, athletics, neuroscience, and medicine. Commercial solutions can precisely track a human inside a room with sub-millimetric precision. However, these systems can track only a few objects at a time; are too expensive to be easily accessible; and their controllers or trackers are too large and inaccurate for research or clinical use. We present a light and small wireless device that piggybacks on current commercial solutions to provide affordable, scalable, and highly accurate positional tracking. This device can be used to track small and precise human movements, to easily embed custom objects inside of a VR system, or to track freely moving subjects for research purposes

Reconstructing the Present Through Kinesthetic History: An Investigation into Modes of Preserving, Transmitting, and Restaging Contemporary Dance

Methods of dance preservation have evolved alongside conceptual themes that have framed dance’s historical narrative. The tradition of written dance notation developed in accordance with notions that prioritized logocentricity, and placed historical legitimacy on tangible artifacts and irrefutable archives; whereas the technical revolution of the late twentieth century saw dance preservation practices shift to embrace film and video documentation because they provided more accessible, and more convenient records. Since the 1970s video recordings have generally been considered to provide authentic visual representations of dance works, and the tradition of score writing has begun to wane. However, scholarly criticism has unveiled both philosophical and practical challenges posed by these two modes of documentation, thus illuminating a gap between theories of embodiment and the practice of dance preservation. In alignment with contemporary discourse, which legitimizes the body as a site of generating and storing knowledge, this dissertation suggests ‘kinesthetic history’ as a valid mode of dance preservation. Operating as a counterpart to oral history, and borrowing theoretical concepts from contemporary historiography, existential phenomenology and ethnography, the term ‘kinesthetic history’ suggests a mode of corporeal inscription and transmission that relies on the reciprocal interaction of bodies in space. The use of ‘kinesthetic history’ as a methodological approach to the preservation, translation, and reconstruction of movement material reflects the elements of fluidity, plurality and subjectivity that are often characteristic of contemporary choreographic practices. This theory is interrogated through a case study, which explores the ways in which both a written and digitized score, video recordings, and the ‘kinesthetic history’ of an original cast member operated as modes of transmission in a 2013 restaging of William Forsythe’s One Flat Thing, reproduced (2000) at The Juilliard School. Conclusions drawn from the case study challenge the traditional notions of reconstruction and restaging and suggest ‘regeneration’ as an alternative term to describe the process of preserving and transmitting contemporary dance works

Graphical marionette : a modern-day Pinocchio

Thesis (M.S.V.S.)--Massachusetts Institute of Technology, Dept. of Architecture, 1983.MICROFICHE COPY AVAILABLE IN ARCHIVES AND ROTCHIncludes bibliographical references (leaves 104-108).Representing the form and movement of human beings naturally, expressively, and computationally is a challenging endeavor. While the skeletal framework can be conveniently described, conventional modeling techniques are generally inadequate for rendering the flexible and irregular surface features of the body. Detail that is sacrificed in articulating the geometry of a life-like character can be compensated for by realistically depicting the character's motion. It is theorized that the most effective method of capturing the subtle dynamics of human motion is to track that motion directly. With the goal of complete body-tracking in view, a prototype system for designing "graphical marionettes" animated by diverse inputs has been developed. The evolution of body modeling as both an artistic and scientific concern are reviewed as precedents to current modeling systems. The development of data structures, animation programs, and some singular rendering techniques are discussed within the context of the project. several difficulties in handling missing or occluded motion data are presented. Some interesting animation scenarios are envisioned as future applications of the system. Improvements to the present version - are suggested in the concluding chapter.by Delle Rae Maxwell.M.S.V.S

Patterning culture: developing a system for the visual notation of greetings

This practice-based thesis examines ways in which the cultural patterning of greetings can be understood through their visualisation. The graphic design practice uses a range of digital tools and software to develop an animated graphic notation system for analysing nonverbal aspects of greetings both within and across cultures. The thesis explicates the process of this development and outlines its context and significance. Previous systems of visualisation devised by anthropologists for the study of greetings have stopped short of using contemporary digital technology. Further, although intercultural contact has increased, most existing greetings studies still focus on intracultural greetings

ECLAP 2012 Conference on Information Technologies for Performing Arts, Media Access and Entertainment

It has been a long history of Information Technology innovations within the Cultural Heritage areas. The Performing arts has also been enforced with a number of new innovations which unveil a range of synergies and possibilities. Most of the technologies and innovations produced for digital libraries, media entertainment and education can be exploited in the field of performing arts, with adaptation and repurposing. Performing arts offer many interesting challenges and opportunities for research and innovations and exploitation of cutting edge research results from interdisciplinary areas. For these reasons, the ECLAP 2012 can be regarded as a continuation of past conferences such as AXMEDIS and WEDELMUSIC (both pressed by IEEE and FUP). ECLAP is an European Commission project to create a social network and media access service for performing arts institutions in Europe, to create the e-library of performing arts, exploiting innovative solutions coming from the ICT