Specialized CNT-based Sensor Framework for Advanced Motion Tracking

In this work, we discuss the design and development of an advanced framework for high-fidelity finger motion tracking based on Specialized Carbon Nanotube (CNT) stretchable sensors developed at our research facilities. Earlier versions of the CNT sensors have been employed in the high-fidelity finger motion tracking Data Glove commercialized by Yamaha, Japan. The framework presented in this paper encompasses our continuing research and development of more advanced CNT-based sensors and the implementation of novel high-fidelity motion tracking products based on them. The CNT sensor production and communication framework components are considered in detail and wireless motion tracking experiments with the developed hardware and software components integrated with the Yamaha Data Glove are reported

16th Sound and Music Computing Conference SMC 2019 (28–31 May 2019, Malaga, Spain)

The 16th Sound and Music Computing Conference (SMC 2019) took place in Malaga, Spain, 28-31 May 2019 and it was organized by the Application of Information and Communication Technologies Research group (ATIC) of the University of Malaga (UMA). The SMC 2019 associated Summer School took place 25-28 May 2019. The First International Day of Women in Inclusive Engineering, Sound and Music Computing Research (WiSMC 2019) took place on 28 May 2019. The SMC 2019 TOPICS OF INTEREST included a wide selection of topics related to acoustics, psychoacoustics, music, technology for music, audio analysis, musicology, sonification, music games, machine learning, serious games, immersive audio, sound synthesis, etc

Embodied Expressions

Embodied Expressions is an exploration into the possibilities of creating live audiovisual performance systems with the aim of expanding my own artistic practice. This thesis investigates how a performer could express more intuitively with physical and mechanical audiovisual interfaces, and stands in revolt against the total digitalization of live electronic music. I argue that modern electronic music performance has led to a decoupling of sound from its traditional origins in real physical objects and has created a limitation in the live performer’s ability to express themselves through the embodied aspects of their live musical expression. This thesis reflects on the artistic exploration of embodied instruments made up of found objects and their control by a performer’s gestural input. The final artifact of this thesis is a set of heard and seen mechanical instruments placed about the audience’s environment that are controlled by the performer using a custom-designed gesture-based control system

End-user action-sound mapping design for mid-air music performance

How to design the relationship between a performer’s actions and an instrument’s sound response has been a consistent theme in Digital Musical Instrument (DMI) research. Previously, mapping was seen purely as an activity for DMI creators, but more recent work has exposed mapping design to DMI musicians, with many in the field introducing soware to facilitate end-user mapping, democratising this aspect of the DMI design process. This end-user mapping process provides musicians with a novel avenue for creative expression, and offers a unique opportunity to examine how practising musicians approach mapping design.Most DMIs suffer from a lack of practitioners beyond their initial designer, and there are few that are used by professional musicians over extended periods. The Mi.Mu Gloves are one of the few examples of a DMI that is used by a dedicated group of practising musicians, many of whom use the instrument in their professional practice, with a significant aspect of creative practice with the gloves being end-user mapping design. The research presented in this dissertation investigates end-user mapping practice with the Mi.Mu Gloves, and what influences glove musicians’ design decisions based on the context of their music performance practice, examining the question: How do end-users of a glove-based mid-air DMI design action–sound mapping strategies for musical performance?In the first study, the mapping practice of existing members of the Mi.Mu Glove community is examined. Glove musicians performed a mapping design task, which revealed marked differences in the mapping designs of expert and novice glove musicians, with novices designing mappings that evoked conceptual metaphors of spatial relationships between movement and music, while more experienced musicians focused on designing ergonomic mappings that minimised performer error.The second study examined the initial development period of glove mapping practice. A group of novice glove musicians were tracked in a longitudinal study. The findings supported the previous observation that novices designed mappings using established conceptual metaphors, and revealed that transparency and the audience’s ability to perceive their mappings was important to novice glove musicians. However, creative mapping was hindered by system reliability and the novices’ poorly trained posture recognition.The third study examined the mapping practice of expert glove musicians, who took part in a series of interviews. Findings from this study supported earlier observations that expert glove musicians focus on error minimisation and ergonomic, simple controls, but also revealed that the expert musicians embellished these simple controls with performative ancillary gestures to communicate aesthetic meaning. The expert musicians also suffered from system reliability, and had developed a series of gestural techniques to mitigate accidental triggering.The fourth study examined the effects of system-related error in depth. A laboratory study was used to investigate how system-related errors impacted a musician’s ability to acquire skill with the gloves, finding that a 5% rate of system error had a significant effect on skill acquisition.Learning from these findings, a series of design heuristics are presented, applicable for use in the fields of DMI design, mid-air interaction design and end-user mapping design

Ihmiskehonjatkesoittimet. Instruments extending the body in sonic performance: creation and development of three tangible string instruments for performative sound art

Ihmiskehonjatkesoittimet is an artistic research on tangible instruments extending the human body. The starting point is to research if the intertwined relationship between a musician and a musical instrument could be emphasized by creating instruments that physically extend the body. The author’s own experiences as a musician lead to discussion about the ergonomics of instruments, which is introduced in the theoretical framework of the thesis. As a context and inspiration for this thesis, instruments and artworks by other artists are being presented. The research is based on reflection of the author during the process of designing and build-ing the instruments, and practising and performing with them. Extensive documentation material of the process has been used as a tool for recalling the process and to illustrate the process for the reader. Two main goals were set for the project by the author: (1) to build durable and innovative instruments that physically can be seen as body extensions and (2) to develop as a performing musician by viewing the new instruments from the perspective of performance art. The results of the thesis support these set goals. Each of the instruments has a different approach to them both from the viewpoint of building and strategies of playing – the instruments provide a diverse view on the topic. The instruments are durable, but some different choices in the design would have made the instruments more practical and ergonomic. The amplification of the instruments as well as the design of the sound manipulation makes the instruments sonically versatile. The concept of extending and freeing the body while playing is felt by the performer, and the documentation material suggests that performing with the instruments create interesting visual images of the musician-instrument relationship while creating new sonic ideas and playing techniques. Performing with these instruments has broadened the author’s skills on planning a performance and being a performer – how to make the gestures on stage more meaningful and how to seek contact with the audience as a performer. The thesis aims to give inspiration and knowledge to artists and instrument builders with the emphasis on embodiment and criticism on unergonomic music making. It sends a message that instrument building can be regarded as innovative art and meaningful tool for musicians to develop as performers.Ihmiskehonjatkesoittimet on taiteellinen tutkimus ihmiskehoa jatkavista soittimista. Tutkimuksen lähtökohtana on tutkia, että voiko soittajan ja soittimen välistä yhteenpunoutunutta suhdetta korostaa kehoa jatkavien soitinten avulla. Kirjoittajan omat kokemukset muusikkona taustoittavat tutkimuksessa esiteltyjä teoreettisia pohdintoja soittamisen ergonomiasta. Tämän lisäksi teoreettisessa viitekehyksessä kontekstualisoidaan tutkimusta tutustumalla muihin soittimiin ja taideteoksiin, jotka ovat inspiroineet maisterityötä. Tutkimus perustuu kirjoittajan pohdintaan liittyen soittimien suunnitteluun ja rakentamiseen sekä niiden kanssa harjoitteluun ja esiintymiseen. Tutkimuksen aikana kerättyä laajaa dokumentaatioaineistoa on käytetty työkaluna prosessin muistiin palauttamisessa sekä projektin kuvaamisessa lukijalle. Tutkimukselle asetettiin kaksi tavoitetta: (1) fyysisesti kehoa jatkavien soittimien rakentaminen kestäviksi ja kekseliäiksi, ja (2) esiintymistaitojen kehittäminen tarkastelemalla soittimia performanssitaiteen näkökulmasta. Tutkimuksen tulokset tukevat asetettuja tavoitteita. Jokainen soitin on rakennettu eri näkökulmasta ja niitä soitetaan eri tavoin – soittimet tarjoavat laajan näkökannan tutkimusaiheesta. Soittimet ovat kestäviä, mutta muuttamalla muutaman yksityiskohdan soitinten suunnittelussa soittimet voisivat olla käytännöllisempiä ja ergonomisempia. Soittimien äänen vahvistaminen ja äänen muokkaaminen tekevät soittimista soinnillisesti monipuolisia. Ajatus kehon jatkamisesta ja kehon vapautumisesta toteutuu soittaessa esiintyjän näkökulmasta ja dokumentaatiomateriaalien perusteella soittaminen muodostaa mielenkiintoisia visuaalisia kuvia soittajan ja soittimen välisestä suhteesta luoden samalla uusia äänellisiä ideoita ja soittotekniikoita. Soitinten kanssa esiintyminen on laajentanut kirjoittajan taitoja suunnitella performansseja sekä kartuttanut tietotaitoa siitä, että miten lavalla tehtävistä eleistä voi tehdä merkityksellisimpiä, ja miten etsiä kontaktia yleisöön. Tutkimus pyrkii inspiroimaan ja antamaan tietoa taiteilijoille ja soitinrakentajille näkökulmasta, joka korostaa soittamisen kehollisuutta ja on samalla kriittinen epäergonomisuutta kohtaan. Projekti pyrkii lähettämään viestin – soitinten rakentaminen voidaan nähdä innovatiivisena taiteena ja merkittävänä työkaluna muusikoille esiintymistaitojen kehittämiseen

DigiDrum:A Haptic-based Virtual Reality Musical Instrument and a Case Study

This paper presents DigiDrum – a novel virtual reality musical instrument (VRMI) which consists of a physical drum augmented by virtual reality (VR) to produce enhanced auditory and haptic feedback. The physical drum membrane is driven by a simulated membrane of which the parameters can be changed on the fly. The design and implementation of the instrument setup are detailed together with the preliminary results of a user study which investigates users’ haptic perception of the material stiffness of the drum membrane. The study tests whether the tension in the membrane simulation and the sound damping (how fast the sound dies out) changes users’ perception of drum membrane stiffness. Preliminary results show that higher values for both tension and damping give the illusion of higher material stiffness in the drum membrane, where the damping appears to be the more important factor. The goal and contribution of this work is twofold: on the one hand it introduces a musical instrument which allows for enhanced musical expression possibilities through VR. On the other hand, it presents an early investigation on how haptics influence users’ interaction in VRMIs by presenting a preliminary study

A virtual musical instrument exhibit for a science centre.

Virtual reality is a technology rapidly gaining interest from research and commercial groups around the world, but it's introduction into New Zealand has been slow. The majority of the general public have no concept of virtual reality, and only a few research institutes have begun virtual reality programmes of any sort. Partially this is due to the high cost of 'off the shelf' virtual reality systems, which is usually beyond the range of many organisations. Also the complexity of the software and the knowledge required to create and manipulate this software makes it a daunting prospect for many. This work describes the development of an economical system for the demonstration of virtual reality and some of its concepts and applications to the general public, in the form of an educational science centre exhibit. The system creates virtual musical instruments, overlayed onto the real world, and the user experiences these instruments as if they were in physical existence

Controllers as musical instruments, controllerism as musical practice - practices of a new 21st century musical culture

This thesis consists of an ethnomusicological approach to the development of Controllers as musical instruments, and conceptualizes Controllerism as a musical practice. I make a case for a revision in organology that includes Controllers, and other instruments of the computer society, by seeking out commonalities and providing comparative analyses between historical instruments and modern Controllers. I then provide definitions of the term Controllerism; by discussing its origins, history, musical logics, strains of musical practice, and current technological explorations. By situating the Controller and Controllerism in a cultural and historical timeline, I have traced informing logics that have led to the development of this new instrument and musical practice. Ethnography has been undertaken with informants from Europe, America and Japan in order to ascertain generalized understandings of the instrument and musical practice; and participatory action research undergone in three separate artist residencies with the intent of determining common perspectives and concerns of international Controllerists. A Portuguese case-study has provided a unique glimpse, by comparison, of this emerging art-form and growing mind-set in modern music.Esta tese é uma aproximação etnomusicológica ao desenvolvimento dos controladores como instrumentos musicais, e conceitualiza o chamado Controllerism como uma prática musical. Defendo a ideia de uma revisão no campo da organologia que inclua os controladores e outros instrumentos da computer society, procurando pontos em comum e providenciando análises comparativas entre instrumentos históricos e controladores modernos. Apresentarei definições do termo Controllerism; discutindo as suas origens, história, lógicas musicais, vertentes de prática musical, e as atuais explorações tecnológicas. Situando o controlador e o Controllerism numa linhagem cultural e histórica, identifico as lógicas que levaram ao desenvolvimento deste novo instrumento e desta nova prática musical. Para tal, elaborei uma etnografia com informantes da Europa, América e Japão com o objetivo de compreender noções comuns sobre o instrumento e a prática musical; fiz também pesquisa participativa em três residências artísticas, com a intenção de determinar perspetivas e preocupações comuns entre Controllerists internacionais. Finalmente, através de um estudo de caso em Portugal, providencio uma visão única, comparativamente falando, desta forma de arte emergente e estilo de vida na música moderna

XR, music and neurodiversity: design and application of new mixed reality technologies that facilitate musical intervention for children with autism spectrum conditions

This thesis, accompanied by the practice outputs,investigates sensory integration, social interaction and creativity through a newly developed VR-musical interface designed exclusively for children with a high-functioning autism spectrum condition (ASC).The results aim to contribute to the limited expanse of literature and research surrounding Virtual Reality (VR) musical interventions and Immersive Virtual Environments (IVEs) designed to support individuals with neurodevelopmental conditions. The author has developed bespoke hardware, software and a new methodology to conduct field investigations. These outputs include a Virtual Immersive Musical Reality Intervention (ViMRI) protocol, a Supplemental Personalised, immersive Musical Experience(SPiME) programme, the Assisted Real-time Three-dimensional Immersive Musical Intervention System’ (ARTIMIS) and a bespoke (and fully configurable) ‘Creative immersive interactive Musical Software’ application (CiiMS). The outputs are each implemented within a series of institutional investigations of 18 autistic child participants. Four groups are evaluated using newly developed virtual assessment and scoring mechanisms devised exclusively from long-established rating scales. Key quantitative indicators from the datasets demonstrate consistent findings and significant improvements for individual preferences (likes), fear reduction efficacy, and social interaction. Six individual case studies present positive qualitative results demonstrating improved decision-making and sensorimotor processing. The preliminary research trials further indicate that using this virtual-reality music technology system and newly developed protocols produces notable improvements for participants with an ASC. More significantly, there is evidence that the supplemental technology facilitates a reduction in psychological anxiety and improvements in dexterity. The virtual music composition and improvisation system presented here require further extensive testing in different spheres for proof of concept