39 research outputs found
NON-VERBAL COMMUNICATION WITH PHYSIOLOGICAL SENSORS. THE AESTHETIC DOMAIN OF WEARABLES AND NEURAL NETWORKS
Historically, communication implies the transfer of information between bodies, yet this
phenomenon is constantly adapting to new technological and cultural standards. In a
digital context, it’s commonplace to envision systems that revolve around verbal modalities.
However, behavioural analysis grounded in psychology research calls attention to
the emotional information disclosed by non-verbal social cues, in particular, actions that
are involuntary. This notion has circulated heavily into various interdisciplinary computing
research fields, from which multiple studies have arisen, correlating non-verbal
activity to socio-affective inferences. These are often derived from some form of motion
capture and other wearable sensors, measuring the ‘invisible’ bioelectrical changes that
occur from inside the body.
This thesis proposes a motivation and methodology for using physiological sensory
data as an expressive resource for technology-mediated interactions. Initialised from a
thorough discussion on state-of-the-art technologies and established design principles
regarding this topic, then applied to a novel approach alongside a selection of practice
works to compliment this. We advocate for aesthetic experience, experimenting with
abstract representations. Atypically from prevailing Affective Computing systems, the
intention is not to infer or classify emotion but rather to create new opportunities for rich
gestural exchange, unconfined to the verbal domain.
Given the preliminary proposition of non-representation, we justify a correspondence
with modern Machine Learning and multimedia interaction strategies, applying an iterative,
human-centred approach to improve personalisation without the compromising
emotional potential of bodily gesture. Where related studies in the past have successfully
provoked strong design concepts through innovative fabrications, these are typically limited
to simple linear, one-to-one mappings and often neglect multi-user environments;
we foresee a vast potential. In our use cases, we adopt neural network architectures to
generate highly granular biofeedback from low-dimensional input data.
We present the following proof-of-concepts: Breathing Correspondence, a wearable
biofeedback system inspired by Somaesthetic design principles; Latent Steps, a real-time auto-encoder to represent bodily experiences from sensor data, designed for dance performance;
and Anti-Social Distancing Ensemble, an installation for public space interventions,
analysing physical distance to generate a collective soundscape. Key findings are
extracted from the individual reports to formulate an extensive technical and theoretical
framework around this topic. The projects first aim to embrace some alternative perspectives
already established within Affective Computing research. From here, these concepts
evolve deeper, bridging theories from contemporary creative and technical practices with
the advancement of biomedical technologies.Historicamente, os processos de comunicação implicam a transferência de informação
entre organismos, mas este fenómeno está constantemente a adaptar-se a novos padrões
tecnológicos e culturais. Num contexto digital, é comum encontrar sistemas que giram
em torno de modalidades verbais. Contudo, a análise comportamental fundamentada
na investigação psicológica chama a atenção para a informação emocional revelada por
sinais sociais não verbais, em particular, acções que são involuntárias. Esta noção circulou
fortemente em vários campos interdisciplinares de investigação na área das ciências da
computação, dos quais surgiram múltiplos estudos, correlacionando a actividade nãoverbal
com inferências sócio-afectivas. Estes são frequentemente derivados de alguma
forma de captura de movimento e sensores “wearable”, medindo as alterações bioeléctricas
“invisíveis” que ocorrem no interior do corpo.
Nesta tese, propomos uma motivação e metodologia para a utilização de dados sensoriais
fisiológicos como um recurso expressivo para interacções mediadas pela tecnologia.
Iniciada a partir de uma discussão aprofundada sobre tecnologias de ponta e princípios
de concepção estabelecidos relativamente a este tópico, depois aplicada a uma nova abordagem,
juntamente com uma selecção de trabalhos práticos, para complementar esta.
Defendemos a experiência estética, experimentando com representações abstractas. Contrariamente
aos sistemas de Computação Afectiva predominantes, a intenção não é inferir
ou classificar a emoção, mas sim criar novas oportunidades para uma rica troca gestual,
não confinada ao domínio verbal.
Dada a proposta preliminar de não representação, justificamos uma correspondência
com estratégias modernas de Machine Learning e interacção multimédia, aplicando uma
abordagem iterativa e centrada no ser humano para melhorar a personalização sem o
potencial emocional comprometedor do gesto corporal. Nos casos em que estudos anteriores
demonstraram com sucesso conceitos de design fortes através de fabricações
inovadoras, estes limitam-se tipicamente a simples mapeamentos lineares, um-para-um,
e muitas vezes negligenciam ambientes multi-utilizadores; com este trabalho, prevemos
um potencial alargado. Nos nossos casos de utilização, adoptamos arquitecturas de redes
neurais para gerar biofeedback altamente granular a partir de dados de entrada de baixa dimensão.
Apresentamos as seguintes provas de conceitos: Breathing Correspondence, um sistema
de biofeedback wearable inspirado nos princípios de design somaestético; Latent
Steps, um modelo autoencoder em tempo real para representar experiências corporais
a partir de dados de sensores, concebido para desempenho de dança; e Anti-Social Distancing
Ensemble, uma instalação para intervenções no espaço público, analisando a
distância física para gerar uma paisagem sonora colectiva. Os principais resultados são
extraídos dos relatórios individuais, para formular um quadro técnico e teórico alargado
para expandir sobre este tópico. Os projectos têm como primeiro objectivo abraçar algumas
perspectivas alternativas às que já estão estabelecidas no âmbito da investigação
da Computação Afectiva. A partir daqui, estes conceitos evoluem mais profundamente,
fazendo a ponte entre as teorias das práticas criativas e técnicas contemporâneas com o
avanço das tecnologias biomédicas
La tecnología como extensión de las facultades sensoriales en las artes escénicas
Programa de Doctorado en Humanidades por la Universidad Carlos III de MadridPresidente: Ricardo Iglesias García.- Secretario: Eduardo Blázquez Mateos.- Vocal: Victoria Pérez Arroy
Music and Digital Media
Anthropology has neglected the study of music. Music and Digital Media shows how and why this should be redressed. It does so by enabling music to expand the horizons of digital anthropology, demonstrating how the field can build interdisciplinary links to music and sound studies, digital/media studies, and science and technology studies.
Music and Digital Media is the first comparative ethnographic study of the impact of digital media on music worldwide. It offers a radical and lucid new theoretical framework for understanding digital media through music, showing that music is today where the promises and problems of the digital assume clamouring audibility. The book contains ten chapters, eight of which present comprehensive original ethnographies; they are bookended by an authoritative introduction and a comparative postlude. Five chapters address popular, folk, art and crossover musics in the global South and North, including Kenya, Argentina, India, Canada and the UK. Three chapters bring the digital experimentally to the fore, presenting pioneering ethnographies of anextra-legal peer-to-peer site and the streaming platform Spotify, a series of prominent internet-mediated music genres, and the first ethnography of a global software package, the interactive music platform Max.
The book is unique in bringing ethnographic research on popular, folk, art and crossover musics from the global North and South into a comparative framework on a large scale, and creates an innovative new paradigm for comparative anthropology. It shows how music enlarges anthropology while demanding to be understood with reference to classic themes of anthropological theory.
Praise for Music and Digital Media
‘Music and Digital Media is a groundbreaking update to our understandings of sound, media, digitization, and music. Truly transdisciplinary and transnational in scope, it innovates methodologically through new models for collaboration, multi-sited ethnography, and comparative work. It also offers an important defense of—and advancement of—theories of mediation.’ Jonathan Sterne, Communication Studies and Art History, McGill University
'Music and Digital Media is a nuanced exploration of the burgeoning digital music scene across both the global North and the global South. Ethnographically rich and theoretically sophisticated, this collection will become the new standard for this field.' Anna Tsing, Anthropology, University of California at Santa Cruz 'The global drama of music's digitisation elicits extreme responses – from catastrophe to piratical opportunism – but between them lie more nuanced perspectives. This timely, absolutely necessary collection applies anthropological understanding to a deliriously immersive field, bringing welcome clarity to complex processes whose impact is felt far beyond what we call music.' David Toop, London College of Communication, musician and writer
‘Spanning continents and academic disciplines, the rich ethnographies contained in Music and Digital Media makes it obligatory reading for anyone wishing to understand the complex, contradictory, and momentous effects that digitization is having on musical cultures.’ Eric Drott, Music, University of Texas, Austin
‘This superb collection, with an authoritative overview as its introduction, represents the state of the art in studies of the digitalisation of music. It is also a testament to what anthropology at its reflexive best can offer the rest of the social sciences and humanities.’ David Hesmondhalgh, Media and Communication, University of Leeds
‘This exciting volume forges new ground in the study of local conditions, institutions, and sounds of digital music in the Global South and North. The book’s planetary scope and its commitment to the “messiness” of ethnographic sites and concepts amplifies emergent configurations and meanings of music, the digital, and the aesthetic.’ Marina Peterson, Anthropology, University of Texas, Austi
Music and Digital Media: A planetary anthropology
Anthropology has neglected the study of music. Music and Digital Media shows how and why this should be redressed. It does so by enabling music to expand the horizons of digital anthropology, demonstrating how the field can build interdisciplinary links to music and sound studies, digital/media studies, and science and technology studies.
Music and Digital Media is the first comparative ethnographic study of the impact of digital media on music worldwide. It offers a radical and lucid new theoretical framework for understanding digital media through music, showing that music is today where the promises and problems of the digital assume clamouring audibility. The book contains ten chapters, eight of which present comprehensive original ethnographies; they are bookended by an authoritative introduction and a comparative postlude. Five chapters address popular, folk, art and crossover musics in the global South and North, including Kenya, Argentina, India, Canada and the UK. Three chapters bring the digital experimentally to the fore, presenting pioneering ethnographies of an extra-legal peer-to-peer site and the streaming platform Spotify, a series of prominent internet-mediated music genres, and the first ethnography of a global software package, the interactive music platform Max.
The book is unique in bringing ethnographic research on popular, folk, art and crossover musics from the global North and South into a comparative framework on a large scale, and creates an innovative new paradigm for comparative anthropology. It shows how music enlarges anthropology while demanding to be understood with reference to classic themes of anthropological theory
Music in the Body –The Body in Music
The body matters in the humanities and within social and cultural studies. It is variously understood as a knowledge store and transmitter, as a node of perception and cognition, as a site of discipline and power and as a locus of identity and agency. But how is the body integral to our concept of music? With increasing interest, Musicology is discovering the epistemological role of the body and its potential as analytical tool, pursuing avenues such as affect studies, performance studies, gender in music and musical perception and cognition.
This volume of collected works draws on an international conference, held at the Department of Musicology at the University of Göttingen in 2019, that aimed to bring together various theoretical perspectives relating to the body and evaluating its present musicological relevance. It explores pathways into a fundamental debate on the body as a central musicological category and reflects on the relevance of this category in the application of diverse musical objects and practices. Composition and performance, aesthetic discourse and sociological analysis, perception and production are all discussed in relation to bodily knowledge, bodily practice and bodily norms. Historical, contemporary, analytical, ethnographic and artistic-experimental approaches reflect the richness of the musicological discipline and its forays into the musical body.
The publication contains twelve different approaches to the body in music in German and English by Sylvain Brétéché, Max Ischebeck, Werner Jauk, Jasna Jovicevic, Moritz Kelber, Tobias Knickmann, Ina Knoth, Madeleine Le Bouteiller, Alastair White, Martin Winter, Stefanie Schroedter and Martin Zenck.:Abbreviations 7
Christine Hoppe, Sarah Avischag Müller: Musicological Pathways into the Body in Music 9
Moving Sounds, Moving Bodies
Stephanie Schroedter: Körper und Klänge in Bewegung – Körperliche Dimensionen von Musik zwischen Embodiment und Enaction 29
Moritz Kelber: Mehr als nur berührend. Die Hand in der Musikpraxis der Frühen Neuzeit 57
Martin Zenck: Musik – eine taktile Kunst? Hand, Auge und Mund in den Dirigierlehren von Hermann Scherchen und Pierre Boulez 91
Body Discourses and Sociological Perspectives
Martin Winter: Musik als Technologie der Körper. Eine Skizze der Ko-Produktionen von Klang, Körper und Subjekt 115
Max Ischebeck: Der musikalische Körper als Wunschmaschine: Wilhelm Heinrich Wackenroder und die Bedeutung des Leibes für die gesellschafliche Wirksamkeit der Musik 133
Ina Knoth: Körpervorstellung und Musikwahrnehmung englischer Virtuosi um 1700 155
Musical Composition – Body Images – Musical Instruments
Alastair White: Material Music: Reclaiming Freedom in Spatialised Time 175
Tobias Knickmann: As if “moving a mountain” – Te Auditive, Visual and Semantic Potential of Performing Chaya Czernowin’s String Quartet and Te last leaf 195
Performance – Body – Perception
Madeleine Le Bouteiller: The Body as a Musical Instrument: Reconsidering Performances with Biosignals 215
Sylvain Brétéché: ‘Body Ways’: The Extra-ordinary Music of the Deaf 229
Jasna Jovicevic: Mapping the Performative Body in the Practice of Jazz Improvisation 255
Werner Jauk: Sound-gesture und ihre Mediatisierungen – musikalische als symbolische Formen von embodied cognitions aus der Natur sonisch performativen Erlebens 273
Appendix
Conference programme 295
Authors 29
Music and the Brain: Composing with the Electroencephalogram
The field of brain-computer music interfaces have been expanding ever since the first development in the 1960s. Although many brain-computer music interfaces have been developed for the purpose of composing music, these systems have not been accessible to musicians due to the use of complex programming and signal processing mathematics. In addition, these systems have often focused on the process of composing music by allowing for specific note selection, or the formation of structure. While these systems are beneficial for those who cannot compose music the traditional way, such as those with motor disabilities, they do not offer a new compositional tool to those with full muscular control. This thesis aims to develop an interactive compositional tool using a brain-computer music interface, utilising methods and techniques that will make it accessible to musicians. An investigation into harnessing brainwaves for musical control has been explored, determining that the alpha and beta waves will be most appropriate for this system. Furthermore, a survey of current BCMI systems have been provided, concluding that the musification approach is best suited to harness the user’s alpha and beta waves. This approach allows for a set of musical rules to be implemented, as well as allowing for the creation of music that is representative of the user’s biological state. The system developed for this thesis used the alpha and beta waves to control parameters of a pre-composed composition. The composition featured elements which were aimed to elicit a change in the user’s brainwave frequency, such as a rise in pitch, rhythmic changes, and key changes. The frequency bands are mapped to the parameters of note velocity, chord changes, and key changes
Designing and Composing for Interdependent Collaborative Performance with Physics-Based Virtual Instruments
Interdependent collaboration is a system of live musical performance in which performers can directly manipulate each other’s musical outcomes. While most collaborative musical systems implement electronic communication channels between players that allow for parameter mappings, remote transmissions of actions and intentions, or exchanges of musical fragments, they interrupt the energy continuum between gesture and sound, breaking our cognitive representation of gesture to sound dynamics.
Physics-based virtual instruments allow for acoustically and physically plausible behaviors that are related to (and can be extended beyond) our experience of the physical world. They inherently maintain and respect a representation of the gesture to sound energy continuum.
This research explores the design and implementation of custom physics-based virtual instruments for realtime interdependent collaborative performance. It leverages the inherently physically plausible behaviors of physics-based models to create dynamic, nuanced, and expressive interconnections between performers. Design considerations, criteria, and frameworks are distilled from the literature in order to develop three new physics-based virtual instruments and associated compositions intended for dissemination and live performance by the electronic music and instrumental music communities. Conceptual, technical, and artistic details and challenges are described, and reflections and evaluations by the composer-designer and performers are documented
Data Sonification in Creative Practice
Sonification is the process of data transmission with non-speech audio. While finding increasing acceptance as a scientific method, particularly where a visual representation of data is inadequate, it is still often derided as a ‘gimmick’. Composers have also shown growing interest in sonification as a compositional method. Both in science and in music, the criticism towards this method relates to poor aesthetics and gratuitous applications. This thesis aims to address these issues through an accompanying portfolio of pieces which use sonification as a compositional tool. It establishes the principles of ‘musification’, which can be defined as a sonification which uses musical structures; a sonification organised by musical principles. The practice-as-research portfolio explores a number of data sources, musical genres and science-music collaborations.
The main contributions to knowledge derived from the project are a portfolio of compositions, a compositional framework for sonification and an evaluation framework for musification. This thesis demonstrates the validity of practice-as-research as a methodology in sonification research