FEATUR.UX: exploiting multitrack information for artistic visualization

FEATUR.UX (Feature - ous) is an audio visualization tool, currently in the process of development, which proposes to introduce a new approach to sound visualization using pre-mixed, independent multitracks and audio feature extraction. Sound visualization is usually performed using a final mix, mono or stereo track of audio. Audio feature extraction is commonly used in the field of music information retrieval to create search and recommendation systems for large music databases rather than generating live visualizations. Visualizing multitrack audio circumvents problems related to the source separation of mixed audio signals and presents an opportunity to examine interdependent relationships within and between separate streams of music. This novel approach to sound visualization aims to provide an enhanced accession to the listening experience corresponding to this use case that employs non-tonal, non-notated forms of electronic music. Findings from prior research studies focused on live performance and preliminary quantitative results from a user survey have provided the basis from which to develop a prototype that will be used throughout an iterative design study to examine the impact of using multitrack audio and audio feature extraction on sound visualization practice

Interactive Machine Learning for Generative Models

Effective control of generative media models remains a challenge for specialised generation tasks, including where no suitable dataset to train a contrastive language model exists. We describe a new approach that enables users to interactively create bespoke text-to-media mappings for arbitrary media generation models, using a small number of examples. This approach - very distinct from contrastive language pretraining approaches - facilitates new strategies for using language to drive media creation in creative contexts not well served by existing methods

The role of live visuals in audience understanding of electronic music Performances

There is an identified lack of visual feedback in electronic music performances. Live visuals have been used to fill in this gap. However, there is a scarcity of studies that analyze the effectiveness of live visuals in conveying feedback. In this paper, we aim to study the contribution of live visuals to the understanding of electronic music performances, from the perspective of the audience. We present related work in the fields of audience studies in performing arts, electronic music and audiovisuals. For this purpose, we organized two live events, where 10 audiovisual performances took place. We used questionnaires to conduct an audience study in these events. Results point to a better audience understanding in two of the four design patterns we used as analytical framework. In our discussion, we suggest best practices for the design of audiovisual performance systems that can lead to improved audience understanding

Bountiful Data: Leveraging Multitrack Audio and Content-Based for Audiovisual Performance.

PhD Thesis.Artists and researchers have long theorized and evaluated connections between sound and image in the context of musical performance. In our investigation, we introduce music information retrieval (MIR) techniques to the practice of live sound visualization and utilize computer-generated graphics to create aesthetic representations of music signals in real time. This thesis presents research that assesses design requirements for live audiovisual practice and evaluates different sound and image interaction systems. We propose a visualization method based on automated music analysis and multitrack audio to provide fine controls for audio-to-visual mapping and to support creative practice. We adopted a user-centered design approach informed by a meta-analysis of user studies exploring contemporary methods of live visual performance. We then conducted online surveys collecting general and specialist knowledge about audiovisual practices, multitrack audio, and audio feature extraction from over 50 practitioners. We performed research through design (RtD) and developed four audiovisual artifacts to test different audiovisual paradigms according to user interaction with audio data, mapping strategies, expression, and affordances. This helped us identify features and limitations of audiovisual models for live performance. Our final prototype (FEATUR.UX.AV) enables users to compose live visuals driven by audio features extracted on multiple instrumental audio stems. We conducted an experiment with 22 audiovisual performers to assess visualization methods in different audio input (multitrack, final mix) and audio feature (raw audio, content-based audio features) conditions. We used Human Computer Interaction (HCI) frameworks to assess usability, hedonic experience, preference, and value as a creativity support tool. In addition to established frameworks, we used qualitative methods to analyze reflective feedback from open answer questions related to aspects of user experience. This evaluation helped us gain insight into the nuances of user experience and highlight advantages and drawbacks of multitrack audio and audio content-analysis for live audiovisual practice

Surveying the Compositional and Performance Practices of Audiovisual Practitioners

This paper presents a brief overview of an online survey conducted with the objective of gaining insight into compositional and performance practices of contemporary audiovisual practitioners. The survey gathered information regarding how practitioners relate aural and visual media in their work, and how compositional and performance practices involving multiple modalities might differ from other practices. Discussed here are three themes: compositional approaches, transparency and audience knowledge, and error and risk, which emerged from participants’ responses. We believe these themes contribute to a discussion within the NIME community regarding unique challenges and objectives presented when working with multiple media