Caramiaux, Baptiste

Donnarumma, Marco

Tanaka, Atau

Goldsmiths Research Online

Body and Space: CombiningModalities for Musical ExpressionMarco DonnarummaDepartment of ComputingGoldsmiths, University ofLondonmd@goldsmithsdigital.comBaptiste CaramiauxDepartment of ComputingGoldsmiths, University ofLondonbc@goldsmithsdigital.comAtau TanakaDepartment of ComputingGoldsmiths, University ofLondonatau@goldsmithsdigital.comCopyright is held by the author/owner(s).TEI 2013, February 10-13, 2013, Barcelona, SpainACMAbstractThis paper presents work in progress on applying aMultimodal interaction (MMI) approach to studyinginteractive music performance. We report on a studywhere an existing musical work was used to provide agesture vocabulary. The biophysical sensing already usedin the work was used as input modality, and augmentedwith several other input sensing modalities not in theoriginal piece. The bioacoustics-based sensor,accelerometer sensors, and full-body motion capturesystem generated data recorded into a multimodaldatabase. We plotted the data from the differentmodalities and offer observations based on visual analysisof the collected data. Our preliminary results show thatthere is complementarity of different forms in theinformation. We noted three types of complementarity:synchronicity, coupling, and correlation.Author Keywordsmultimodal interaction; NIME; interactive musicperformance; gesture; expressivity; biosignalsACM Classification KeywordsH.5.5 [Sound and Music Computing]: Methodologies andtechniques.IntroductionMultimodal interaction (MMI) is the integration ofmultiple modalities (or input channels) to increaseinformation and bandwidth of interaction. MMI usesdifferent modalities that offer complementary informationabout user input. These modalities might include, forexample, voice input to complement pen-based input [9].The combination of complementary modalities providesinformation to better understand aspects of the user inputthat cannot be deduced from a single input modality. Theanalysis of combined information could be useful in thecontexts involving forms of user expressivity in input.Research in game control, sign language recognition, andprosthesis control looks at complementary use of spatialaspects of hand gesture, and physiological biosignals suchas muscle tension (electromyogram, or EMG). Zhang etal. combined EMG and accelerometer (ACC) data in agesture recognition system for the manipulation of virtualobjects [11]. Li et al. [7] demonstrate an automaticChinese sign language recognition system based on EMGand ACC. Fougner et al. show that the multimodal use ofEMG and ACC was effective in reducing the number ofEMG channels needed compared to a biosignal-onlyprosthetic interface [6].Audio, video, and motion capture modalities were usedalongside EMG, heart rate or EKG,electroencephalography or EEG by [8] in a study focusingon social interaction in traditional musical ensembleperformance.In the field of New Interfaces for Musical Expression(NIME), sensor-based systems capture gesture in livemusical performance. In contrast with studio-based musiccomposition, NIME (which began as a workshop at CHI2001) focuses on real-time performance. Early examplesof interactive musical instrument performance thatpre-date the NIME conference include the work of MichelWaisvisz and his instrument, The Hands, a set ofaugmented gloves which captures data fromaccelerometers, buttons, mercury orientation sensors, andultrasound distance sensors [5]. The use of multiplesensors on one instrument points to complementarymodes of interaction with an instrument [1]. Howeverthese NIME instruments have for the most part not beendeveloped or studied explicitly from an MMI perspective.In this work-in-progress report, we show the relevance ofconsidering biosignal-based multi-modality for the analysisof expressive musical gesture. This follows initial worklooking at the integration of biosignals with relativeposition sensing [10]. Here, we describe a system forcapturing three input modalities from the arm gestures ofa musician. We then present the data from the differentmodalities together to look at the relationships amongstthem. We identify three types of complementarity anddiscuss perspectives for future work.MethodWe conducted a pilot study recording musical gesturesusing 3 input modalities, and performed anobservation-based analysis on mutual relationships in thedata. The different modalities (mechanomyogram orMMG, accelerometer, and motion capture) detectphysiological, movement, and spatial position information,respectively.Sensor apparatusThe MMG is a signal generated by subcutaneousmechanical vibrations resulting from muscle contraction.For this we used the Xth Sense (XS), a biophysical NIMEinstrument1. The XS consists of an arm band containingan electret condenser microphone (KingstateKECG2742PBL-A) where acoustic perturbations frommuscle contraction are digitised as sound at a samplingrate of 48000 Hz. Two channels of MMG were recorded,from the right and left arms over the wrist flexors, amuscle group close to the elbow joint that controls fingermovement. The right sensor was located on the ringfinger flexor, and the left one was slightly offset towardsthe little finger flexor.The accelerometer sensor was a 3-axis DUL Radio(Analog Devices ADXL345) sending wireless data througha Nordic transceiver with a bandwidth of 2.4GHz, andsampling rate of 100Hz. The sensor was located on theback of the forearm, close to the wrist.The motion capture system was an Optitrack Arena. Thisconsisted of a full body suit, with 34 markers, and 11LED/Infrared cameras (V100:R2) with a sampling rate of60 FPS. We recorded 24 rigid bodies, that is, groups ofmarkers representing limb part positions.Data acquisitionThe data collected was comprised of:• 2 MMG audio signals, along with 6 amplitude/timedomain sub-features extracted from analysis of theMMG audio stream;• one 3D vector from the accelerometer;• 24 rigid bodies, consisting of 7D signal for eachrigid body: 3D position and 4D quaternions1Developed by the first author.http://res.marcodonnarumma.com/projects/xth-sense/The data was synchronised and captured by customsoftware developed in the Max/MSP graphicalprogramming environment2. The data was time tagged,and stored in text format for oﬄine analysis.Gesture vocabularyWe used an existing piece of interactive music entitledMusic for Flesh II, a work for XS that has had repeatedperformances by author 1 [4]. In the piece, the XS sendsMMG from arm gesture to a computer programme toarticulate sound and further process sound with the samemuscle data. The composition is based on a vocabulary of12 arm gestures. These gestures comprised the gesturevocabulary for our experiment.The gestures, described in metaphorical, musical termsare:1. Shaking bells2. Scattering sound grains 13. Stretching sound 14. Stretching sound 25. Dropping something small6. Rotating bells7. Grasping the void8. Shaping a wave9. Throwing a sound wave10. Holding a growing force11. Scattering sound grains 212. Rotating wheel2http://www.cycling74.comFigure 1: Example gesture Stretching sound 2Each gesture was repeated 10 times with differentexpressive articulation, that is, changing speed andintensity in each iteration. Fig. reffig:gesture illustrates atrial of gesture 4 (Stretching sound 2). This gestureconsists of fast wrist rotation, and faster flexion of thedistal phalanges.A detailed description of each gesture as well as thecomplete database can be seen on-line3.ResultsIn this section we report results focusing on themechano-myogram and accelerometer data. In thefollowing graphs the accelerometer x, y, and z axes are thegreen, red, and blue traces, respectively. The MMG soundwave is the black trace.3http://marcodonnarumma.com/submissions/TEI2013/6.85 6.9 6.95 7 7.05 7.1 7.15−1.5−1−0.500.5Gesture 04 − Trial 01Time (s)Input Channels (a.u.)Figure 2: Detail of the MMG and ACC attack envelopes.Fig. 2 shows the MMG (black) and ACC (blue, green,red) data recording from Gesture 4 (fig. 1). The red zonehighlights the onset of muscle activity. The grey zonehighlights the onset of accelerometer data. The initialactivity in the muscle anticipates accelerometer data, witha second peak in MMG coinciding with the first peak inACC. This shows the preparatory activity in the muscle atthe beginning of a gesture that is not reported by theaccelerometer sensor.The graphs in Fig. 3 show two iterations of Gesture 1(Shaking bells), a gesture that consists of multiplerepeated contractions of the right hand fingers. Thegesture was executed with two different kinds ofexpression, weak (low muscle force) and strong (highmuscle force). By plotting the MMG and the 3 axes ofthe ACC data, we are able to see if the signals are coupled- whether the different modalities parallel each other, andcoincide, or not. At a weak articulation the oscillatingnature of the finger grasping is slightly visible in the MMGtrace. The accelerometer axes do not seem coupled to theMMG or to each other. At a higher articulation force, thesignal for all the modalities undulates at close frequency ina coupled manner.1 2 3 4 5 6−1.2−1−0.8−0.6−0.4−0.200.20.40.60.8Gesture 01 −0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−2−1.5−1−0.500.51Gesture 01 − Trial 08Input Channels (a.u.)Input Channels (a.u.)Time (s) Time (s)Figure 3: Gesture 1 executed at two different force levels.Fig. 4 shows a more complex gesture, (Scattering soundgrains 2). This consists of repeated subito finger claspingalongside swift wrist rotation. Here we see a change incorrelation across modalities over the course of thegesture. Based on this shifting correlation, we divided thegesture into three segments. The first and third segmentsexhibit high correlation between MMG and ACCcomponents, while in the second segment the subtleties ofthe MMG envelope are not correlated with the peaks inthe ACC data.2 4 6 8 10 12−2.5−2−1.5−1−0.500.51Gesture 11 − Trial 01Time (s)Input Channels (a.u.)Segment #1 Segment #2 Segment #3Figure 4: Correlation and independent variations of MMG andACC channels within the same gesture.DiscussionSynchronicityIn Gesture 4, we noted a time offset between MMG andACC onsets. This may be due to different factors. Thetwo modalities - muscle force and limb acceleration - maybe asynchronous. One modality may have a lower latencythan another. Or, the corporeal sensor may detectpreparatory activity of the body that is not seen in thespatial/physical sensors. This could have usefulapplication whereby the preparatory nature of onemodality may aid in anticipating the arrival of data onanother modality.CouplingWe noted that coupling of modalities can vary withchanging expressivity of the same gesture. In Gesture 1,during weaker articulation of the gesture, the MMG andthe 3 ACC axes were decoupled while at strongerexpression of the same gesture, we observed tightercoupling in these modalities. This may indicate that two”regimes” can be engaged: either a low force with fairlyindependent signals or high force with coupled signals.This has potential to be the subject of further study.CorrelationWe observed the shifting of cross-modal correlation withinthe span of one gesture. We visually segmented zonesthat displayed high and low correlation between MMG andACC. This may point out differing levels of dependencybetween muscle control and the resulting movement.Interestingly, the ACC data in Gesture 11 have more orless the same amplitude throughout the gesture, but theMMG amplitude varies greatly, pointing out a differencein dynamic range in the different input modalities. Thismay be studied in more detail using automatic featureextraction methods [2], to examine quantitatively whichfeatures of the MMG and ACC signals are correlated.Observations on other modalitiesIn gestures characterised by finger contractions (1, 2, 4,and 11), unless a marker is positioned on each finger, thetracking of phalange movement is not detected. Motioncapture is, however, useful in conveying information aboutthe movement of other parts of the body such as headand torso. Studies of musical instrument performance [3]point out the importance of such corporeal movement tomusical expressivity. This auxiliary information, beyondthe constraint space of multiple input modes sensing limbgesture, could be useful in multimodal musical interaction.ConclusionThis paper reported on preliminary work applying aMultimodal Interaction approach to analyse expressivemusical gesture. By using three distinct modalities totrack gesture in the performance of an existing NIME-typework, we were able to make several early observations onthe relationships between the kinds of informationreported by the different input channels. We found that:the complementarity across different input modalities maybe due to differing sensitivity of sensors to the preparationof a gesture; what seems like a single musical gesture maybe comprised of different sections where the relationshipamongst modalities may change over time; a range ofmodalities might detect the independent control differentaspects of a single gesture.The results presented are initial observations on a subsetof the data we collected. One challenge in working acrossa diverse range of information-rich modalities is in thedata management and numerical processing. Signalacquisition and synchronisation across free-standinghardware systems, different sampling rates, and networkcommunication latency pose additional technicalchallenges that would need to be rigorously addressed inan in-depth study.One element that made up part of our capture sessionwhich we did not report on in this paper is the capture ofaudio (of the musical output) and video (of the performerconducting the gesture). These media channels couldprovide useful points of reference, and might, given theappropriate configuration, even be exploited as auxiliaryinput modalities.References[1] A. Camurri, P. Coletta, G. Varni, and S. Ghisio. Developingmultimodal interactive systems with EyesWeb XMI.Proceedings of the 7th international conference on Newinterfaces for musical expression - NIME ’07, page 305, 2007.[2] B. Caramiaux, F. Bevilacqua, and N. Schnell. Towards agesture-sound cross-modal analysis. In In EmbodiedCommunication and Human-Computer Interaction, volume5934 of Lecture Notes in Computer Science, pages 158—-170.Springer Verlag, 2010.[3] J. W. Davidson. Qualitative insights into the use of expressivebody movement in solo piano performance: a case studyapproach. Psychology of Music, 35(3):381–401, July 2007.[4] M. Donnarumma. Incarnated sound in Music for Flesh II.Defining gesture in biologically informed musical performance.Leonardo Electronic Almanac (Touch and Go), 18(3):164–175,2012.[5] E. Dykstra-Erickson and J. Arnowitz. Michel Waisvisz: theman and the hands. Interactions, 12(5):63–67, Sept. 2005.[6] A. Fougner, E. Scheme, A. D. C. Chan, K. Englehart, andO. Stavdahl. A multi-modal approach for hand motionclassification using surface EMG and accelerometers.Conference proceedings : ... Annual International Conference ofthe IEEE Engineering in Medicine and Biology Society. IEEEEngineering in Medicine and Biology Society. Conference,2011(Grant 192546):4247–50, Jan. 2011.[7] Y. Li, X. Chen, J. Tian, X. Zhang, K. Wang, and J. Yang.Automatic recognition of sign language subwords based onportable accelerometer and EMG sensors. InternationalConference on Multimodal Interfaces and the Workshop onMachine Learning for Multimodal Interaction on - ICMI-MLMI’10, page 1, 2010.[8] O. Mayor, J. Llop, and E. Maestre. RepoVizz: A multimodalon-line database and browsing tool for music performanceresearch. In 12th International Society for Music InformationRetrieval Conference (ISMIR 2011), Miami, USA, 2011.[9] S. Oviatt, R. Coulston, S. Tomko, B. Xiao, R. Lunsford,M. Wesson, and L. Carmichael. Toward a theory of organizedmultimodal integration patterns during human-computerinteraction. Proceedings of the 5th international conference onMultimodal interfaces, pages 44–51, 2003.[10] A. Tanaka and R. B. Knapp. Multimodal Interaction in MusicUsing the Electromyogram and Relative Position Sensing.Proceedings of the 2002 conference on New interfaces formusical expression, pages 1–6, 2002.[11] X. Zhang, X. Chen, W.-h. Wang, J.-h. Yang, V. Lantz, andK.-q. Wang. Hand gesture recognition and virtual game controlbased on 3D accelerometer and EMG sensors. In Proceedingscof the 13th international conference on Intelligent userinterfaces - IUI ’09, page 401, New York, New York, USA,2008. ACM Press.

Body and Space : Combining Modalities for Musical Expression

https://research.gold.ac.uk/10632/1/Donnarumma-TEI13wip.pdf

Body and Space : Combining Modalities for Musical Expression

Abstract

Similar works

Full text

Available Versions

Goldsmiths Research Online