Cross-talk: A Shared Parameter Space for Gesturally Extended Human/Machine Improvisation

This paper describes Cross-talk, a piece of music and performance system for two instruments augmented with infrared motion-tracking capability, and an artificial software improviser. Cross-talk was commissioned by the Ammerman Center for Arts and Technology at Connecticut College, for the 13th Biennial Symposium on Arts and Technology. The work is part of an ongoing collaboration focused on developing integrated hardware and software performance systems to extend the timbral and expressive capabilities of traditional musical instruments and to generate musical structure in response to information retrieved from human performers in real-time. Artistic motivations and prior related work are presented here, along with a summary of the programmatic narrative behind Cross-talk and an accompanying qualitative description of the piece. Technical details are provided for important components of the work, including the Gesturally Extended Piano and the “factorOracle” software module, which is used to facilitate the system\u27s machine improvisation capability

The Variable Markov Oracle: Algorithms for Human Gesture Applications

This article introduces the Variable Markov Oracle (VMO) data structure for multivariate time series indexing. VMO can identify repetitive fragments and find sequential similarities between observations. VMO can also be viewed as a combination of online clustering algorithms with variable-order Markov constraints. The authors use VMO for gesture query-by-content and gesture following. A probabilistic interpretation of the VMO query-matching algorithm is proposed to find an analogy to the inference problem in a hidden Markov model (HMM). This probabilistic interpretation extends VMO to be not only a data structure but also a model for time series. Query-by-content experiments were conducted on a gesture database that was recorded using a Kinect 3D camera, showing state-of-the-art performance. The query-by-content experiments' results are compared to previous works using HMM and dynamic time warping. Gesture following is described in the context of an interactive dance environment that aims to integrate human movements with computer-generated graphics to create an augmented reality performance

Creative Support Musical Composition System: a study on Multiple Viewpoints Representations in Variable Markov Oracle

Em meados do século XX, assistiu-se ao surgimento de uma área de estudo focada na geração au-tomática de conteúdo musical por meios computacionais. Os primeiros exemplos concentram-se no processamento offline de dados musicais mas, recentemente, a comunidade tem vindo a explorar maioritariamente sistemas musicais interativos e em tempo-real. Além disso, uma tendência recente enfatiza a importância da tecnologia assistiva, que promove uma abordagem centrada em escolhas do utilizador, oferecendo várias sugestões para um determinado problema criativo. Nesse contexto, a minha investigação tem como objetivo promover novas ferramentas de software para sistemas de suporte criativo, onde algoritmos podem participar colaborativamente no fluxo de composição. Em maior detalhe, procuro uma ferramenta que aprenda com dados musicais de tamanho variável para fornecer feedback em tempo real durante o processo de composição. À luz das características de multi-dimensionalidade e hierarquia presentes nas estruturas musicais, pretendo estudar as representações que abstraem os seus padrões temporais, para promover a geração de múltiplas soluções ordenadas por grau de optimização para um determinado contexto musical. Por fim, a natureza subjetiva da escolha é dada ao utilizador, ao qual é fornecido um número limitado de soluções 'ideais'. Uma representação simbólica da música manifestada como Modelos sob múltiplos pontos de vista, combinada com o autómato Variable Markov Oracle (VMO), é usada para testar a interação ideal entre a multi-dimensionalidade da representação e a idealidade do modelo VMO, fornecendo soluções coerentes, inovadoras e estilisticamente diversas. Para avaliar o sistema, foram realizados testes para validar a ferramenta num cenário especializado com alunos de composição, usando o modelo de testes do índice de suporte à criatividade.The mid-20th century witnessed the emergence of an area of study that focused on the automatic generation of musical content by computational means. Early examples focus on offline processing of musical data and recently, the community has moved towards interactive online musical systems. Furthermore, a recent trend stresses the importance of assistive technology, which pro-motes a user-in-loop approach by offering multiple suggestions to a given creative problem. In this context, my research aims to foster new software tools for creative support systems, where algorithms can collaboratively participate in the composition flow. In greater detail, I seek a tool that learns from variable-length musical data to provide real-time feedback during the composition process. In light of the multidimensional and hierarchical structure of music, I aim to study the representations which abstract its temporal patterns, to foster the generation of multiple ranked solutions to a given musical context. Ultimately, the subjective nature of the choice is given to the user to which a limited number of 'optimal' solutions are provided. A symbolic music representation manifested as Multiple Viewpoint Models combined with the Variable Markov Oracle (VMO) automaton, are used to test optimal interaction between the multi-dimensionality of the representation with the optimality of the VMO model in providing both style-coherent, novel, and diverse solutions. To evaluate the system, an experiment was conducted to validate the tool in an expert-based scenario with composition students, using the creativity support index test

Computational modeling of improvisation in Turkish folk music using Variable-Length Markov Models

The thesis describes a new database of uzun havas, a non-metered structured improvisation form in Turkish folk music, and a system, which uses Variable-Length Markov Models (VLMMs) to predict the melody in the uzun hava form. The database consists of 77 songs, encompassing 10849 notes, and it is used to train multiple viewpoints, where each event in a musical sequence are represented by parallel descriptors such as Durations and Notes. The thesis also introduces pitch-related viewpoints that are specifically aimed to model the unique melodic properties of makam music. The predictability of the system is quantitatively evaluated by an entropy based scheme. In the experiments, the results from the pitch-related viewpoints mapping 12-tone-scale of Western classical theory and 17 tone-scale of Turkish folk music are compared. It is shown that VLMMs are highly predictive in the note progressions of the transcriptions of uzun havas. This suggests that VLMMs may be applied to makam-based and non-metered musical forms, in addition to Western musical styles. To the best of knowledge, the work presents the first symbolic, machine-readable database and the first application of computational modeling in Turkish folk music.MSCommittee Chair: Parag Chordia; Committee Member: Gil Weinberg; Committee Member: Jason Freema

A Functional Taxonomy of Music Generation Systems

Digital advances have transformed the face of automatic music generation since its beginnings at the dawn of computing. Despite the many breakthroughs, issues such as the musical tasks targeted by different machines and the degree to which they succeed remain open questions. We present a functional taxonomy for music generation systems with reference to existing systems. The taxonomy organizes systems according to the purposes for which they were designed. It also reveals the inter-relatedness amongst the systems. This design-centered approach contrasts with predominant methods-based surveys and facilitates the identification of grand challenges to set the stage for new breakthroughs.Comment: survey, music generation, taxonomy, functional survey, survey, automatic composition, algorithmic compositio

Planning Human-Computer Improvisation

International audienceImprovisation intrinsically carries a dialectic between spontaneity/reactivity and long-term planning/organization. This paper transposes this dialectic to interactive human-computer improvisation where the computer has to interleave various generative processes. They require different levels of prior knowledge, and follow a coarser improvisation plan driven by real-time interactions with the musicians. We propose a time-aware extensible architecture allowing the temporal coordination of different improvisation strategies. It integrates various generative strategies capitalizing on the system Improtek into the high-level structure provided by the language associated to the score follower Antescofo. The resulting framework manages the musical events, the triggering of generative processes at different time scales and the declarative specification of improvisation plans driven by the occurrence of complex events