Analysis of the qualities of human movement in individual action

The project was organized as a preliminary study for Use Case #1 of the Horizon 2020 Research Project \u201cDance in the Dark\u201d (H2020 ICT Project n.645553 - http://dance.dibris.unige.it). The main objective of the DANCE project is to study and develop novel techniques and algorithms for the automated measuring of non-verbal bodily expression and the emotional qualities conveyed by human movement, in order to enable the perception of nonverbal artistic whole-body experiences to visual impaired people. In the framework of the eNTERFACE \u201915 Workshop we investigated methods for analyzing human movements in terms of expressive qualities. When analyzing an individual action we were mainly concentrating on the quality of motion and on elements suggesting different emotions. We developed a system to automatically extract several movement features and transfer them to the auditory domain through interactive sonification. We performed an experiment with 26 participants and collected a dataset made of video and audio recordings plus accelerometer data. Finally, we performed a perception study through questionnaires, in order to evaluate and validate the system. As real time application of our system we developed a game named \u201dMove in the Dark\u201d, which has been presented in the Mundaneum Museum of Mons, Belgium and Festival della Scienza, Genova, Italy (27 November 2015)

Audiovisual granular synthesis: creating synergistic relationships between sound and image

The aims of this research were to investigate how an audio processing technique known as granular synthesis can be translated to a visual processing equivalent, and to develop software that fuses audiovisual relationships for the creation of real-time audiovisual art. In order to carry out this project, two main research questions were posed. The first question was: how can audio processing techniques such as granular synthesis be adapted and applied to influence new visual performance techniques, and the second question was: how can computer software synergistically integrate audio and visuals to enable the real-time creation and performance of audiovisual art. The project at the centre of my research was the creation of a real-time audiovisual granular synthesis instrument named Kortex. The research project involved a practice-based methodology and used an iterative performance cycle to evaluate and develop the Kortex prototype. These included performing iterations of the Kortex prototype at a number of local, interstate and international events. Kortex facilitates the identification of shared characteristics found between sound and image at the micro and macro level. The micro level addresses individual audiovisual segments, or grains, while the macro level addresses post-processing effects applied to the stream of audiovisual grains. Audiovisual characteristics are paired together by the user at each level, enabling composition with both media simultaneously. This provides the audiovisual artist with a dynamic approach for the creation of new works. Creating relationships between image and sound is highly subjective, yet an artist may use a mathematical, metaphorical/intuitive or intrinsic approach to create a convincing correlation between the two media. The mathematical approach expresses the relationship between sound and image as an equation. Metaphorical/intuitive relationships are formed when the two media share similar emotional or perceptual characteristics, while intrinsic relationships occur when audio and visual media are synthesised from the same source. Performers need powerful control strategies to manipulate large collections of variables in real-time. I found that pattern-generating modulation sources created overlapping phrases that evolved the behaviour of audiovisual relationships. Furthermore, saving interesting aesthetics that emerged into banks of presets, along with the ability to slide from one to the next, facilitated powerful transformations during a performance. The project has contributed to the field of audiovisual art, specifically to the performance work of DJs and VJs. Kortex provides a single audiovisual composition and performance environment that can be used by DJs and VJs for creative collaboration. Kortex has enormous potential for adoption by the DJ/VJ community to assist in the production of tightly synchronised real-time audiovisual performances

Networks of Liveness in Singer-Songwriting: A practice-based enquiry into developing audio-visual interactive systems and creative strategies for composition and performance.

This enquiry explores the creation and use of computer-based, real-time interactive audio-visual systems for the composition and performance of popular music by solo artists. Using a practice-based methodology, research questions are identified that relate to the impact of incorporating interactive systems into the songwriting process and the liveness of the performances with them. Four approaches to the creation of interactive systems are identified: creating explorative-generative tools, multiple tools for guitar/vocal pieces, typing systems and audio-visual metaphors. A portfolio of ten pieces that use these approaches was developed for live performance. A model of the songwriting process is presented that incorporates system-building and strategies are identified for reconciling the indeterminate, electronic audio output of the system with composed popular music features and instrumental/vocal output. The four system approaches and ten pieces are compared in terms of four aspects of liveness, derived from current theories. It was found that, in terms of overall liveness, a unity to system design facilitated both technological and aesthetic connections between the composition, the system processes and the audio and visual outputs. However, there was considerable variation between the four system approaches in terms of the different aspects of liveness. The enquiry concludes by identifying strategies for maximising liveness in the different system approaches and discussing the connections between liveness and the songwriting process

Vibrating instruments in virtual reality: A cohesive approach to the design of virtual reality musical instruments

This thesis presents the design, implementation and findings of a Virtual Reality Musical Instrument (VRMI). The project was done under the direction of the Sound and Physical Interaction (SOPI) research group. The project was made following an iterative design methodology and the metaphors and design patterns used in Ubiquitous Music Systems. In contrast with the fast adoption of Virtual Reality as a platform for new entertainment productions, it is noticeable that the area of new interfaces for musical expression (NIME) has been disbelieving towards this technology. At the same time, previous projects under the category of VRMI have made a clear distinction between the instrument, an external 3D model, and the user. Thereby, this thesis presents a project that focuses on how VR can enhance individual musical interaction? In order to do so, this project is directed to blurry the lines between performer, instrument and environment by creating immersion through 3D audio, audiovisual feedback, bodily and spatial interaction, the performer and the system's autonomous responses. As a final result, this thesis reaches to provide the NIME community with a purposeful use of Virtual Reality as an interactive musical platform

Call for Participation

numediart is a long-term research programme centered o

Modelling the live-electronics in electroacoustic music using particle systems

Contemporary music is largely influenced by technology. Empowered by the current available tools and resources, composers have the possibility to not only compose with sounds, but also to compose the sounds themselves. Personal computers powered with intuitive and interactive audio applications and development tools allow the creation of a vast range of real-time manipulation of live instrumental input and also real-time generation of sound through synthesis techniques. Consequently, achieving a desired sonority and interaction between the electronic and acoustic sounds in real-time, deeply rely on the choice and technical implementation of the audio processes and logical structures that will perform the electronic part of the composition. Due to the artistic and technical complexity of the development and implementation of such a complex artistic work, a very common strategy historically adopted by composers is to develop the composition in collaboration with a technology expert, which in this context is known as a musical assistant. In this perspective, the work of the musical assistant can be considered as one of translating musical, artistic and aesthetic concepts into mathematical algorithms and audio processes. The work presented in this dissertation addresses the problem of choosing, combining and manipulating the audio processes and logical structures that take place on the liveelectronics (i.e the electronic part of a mixed music composition) of a contemporary electroacoustic music composition, by using particle systems to model and simulate the dynamic behaviors that reflect the conceptual and aesthetic principles envisaged by the composer for a determined musical piece. The presented research work initiates with a thorough identification and analysis of the agents, processes and structures that are present in the live-electronics system of a mixed music composition. From this analysis a logical formalization of a typical live-electronics system is proposed, and then adapted to integrate a particle-based modelling strategy. From the formalization, a theoretical and practical framework for developing and implementing live-electronics systems for mixed music compositions using particle systems is proposed. The framework is experimented and validated in the development of distinct mixed music compositions by distinct composers, in real professional context. From the analysis of the case studies and the logical formalization, and the feedback given by the composers, it is possible to conclude that the proposed particle systems modelling method proves to be effective in the task of assisting the conceptual translation of musical and aesthetic ideas into implementable audio processing software.A música contemporânea é amplamente influenciada pela tecnologia. Os recursos tecnológicos atualmente disponíveis permitem que os compositores criem com sons e ao mesmo tempo criem os sons em si próprios. Os atuais aplicativos e ferramentas de software focados no desenvolvimento, controle e manipulação de processamentos de áudio, permitem a elaboração de diversos tipos de tratamentos e sínteses de som com a capacidade de serem executados e manipulados em tempo real. Consequentemente, a escolha dos algoritmos de processamento de áudio e suas respectivas implementações técnicas em forma de software, são determinantes para que a sonoridade desejada seja atingida, e para que o resultado sonoro satisfaça os objetivos estéticos e conceituais da relação entre as fontes sonoras acústicas e os sons eletrônicos presentes em uma composição eletroacústica de caráter misto. Devido à complexidade artística e técnica do desenvolvimento e implementação do sistema de eletrônica em tempo real de uma composição eletroacústica mista, uma estratégia historicamente adotada por compositores é a de desenvolver a composição em colaboração com um especialista em tecnologia, que neste contexto é usualmente referido como assistente musical. Nesta perspectiva, o trabalho do assistente musical pode ser interpretado como o de traduzir conceitos musicais, artísticos e estéticos em algoritmos matemáticos e processamento de áudio. O trabalho apresentado nesta dissertação aborda a problemática da escolha, combinação e manipulação dos processamentos de áudio e estruturas lógicas presentes no sistema de eletrônica em tempo real de uma composição de música eletroacústica contemporânea, e propõem o uso de sistemas de partículas para modelar e simular os comportamentos dinâmicos e morfológicos que refletem os princípios conceituais e estéticos previstos pelo compositor para uma determinada composição. A parte inicial do trabalho apresentado consiste na identificação e análise detalhada dos agentes, estruturas e processos envolvidos na realização e execução do sistema de eletrônica em tempo real. A partir desta análise é proposta uma formalização lógica e genérica de um sistema de eletrônica em tempo real. Em seguida, esta formalização é modificada e adaptada para integrar uma estratégia de modelagem através de sistemas de partículas. Em sequencia da formalização lógica, um método teórico e prático para o desenvolvimento de sistemas de eletrônica em tempo real para composições de música mista é proposto. O teste e consequente validação do método se dá através de sua utilização na realização da eletrônica em tempo real para obras de diferentes compositores. A análise dos casos de estudo e da formalização lógica, e também o parecer e opinião dos compositores, permitem concluir que o método proposto é de fato eficaz na tarefa de auxiliar o processo de tradução dos conceitos musicais e estéticos propostos pelos compositores em forma de algoritmos e processamentos de som implementados em software

Developing a hybrid video synthesiser for audiovisual performance and composition

This thesis presents the development process and analysis of a hybrid video synthesiser that was designed towards use in audiovisual performance and composition. Though there are various instruments, platforms, and approaches towards audiovisual performance and composition, this practice-based thesis, through the development of an instrument, addresses a very specific gap – the lack of a hybrid, i.e., a software-based physical modular video synthesiser. This thesis will trace the historical origins of the video synthesiser, contextualise audiovisual performance and composition, and discuss the development of this hybrid video synthesiser. This thesis will document the iterative development process, provide key takeaways from performances and finally, provide a discussion, certain recommendations, and contributions that this instrument makes within the field of video synthesisers and audiovisual performance