Reality-based interfaces have the property of linking the user's physical space with the computer digital content, bringing in intuition, plasticity and expressiveness.
Moreover, applications designed upon motion and gesture tracking technologies involve a lot of psychological features, like space cognition and implicit knowledge.
All these elements are the background of three presented music applications, employing the characteristics of three different interactive spaces: a user centered three dimensional space, a floor bi-dimensional camera space, and a small sensor centered three dimensional space.
The basic idea is to deploy the application's spatial properties in order to convey some musical knowledge, allowing the users to act inside the designed space and to learn through it in an enactive way

Mandanici, Marcellina

English

Padua Research Archive

Interactive Spaces. Models and Algorithms for Reality-based Music Applications

Reality-based interfaces have the property of linking the user's physical space with the computer digital content, bringing in intuition, plasticity and expressiveness.
Moreover, applications designed upon motion and gesture tracking technologies involve a lot of psychological features, like space cognition and implicit knowledge.
All these elements are the background of three presented music applications, employing the characteristics of three different interactive spaces: a user centered three dimensional space, a floor bi-dimensional camera space, and a small sensor centered three dimensional space.
The basic idea is to deploy the application's spatial properties in order to convey some musical knowledge, allowing the users to act inside the designed space and to learn through it in an enactive way

MANDANICI, MARCELLINA

Archivio istituzionale della ricerca - Università di Padova

Original Citation:Interactive Spaces. Models and Algorithms for Reality-based Music ApplicationsAssociation for Computing Machinery (ACM)Publisher:Published version:DOI:Terms of use:Open Access(Article begins on next page)This article is made available under terms and conditions applicable to Open Access Guidelines, as described athttp://www.unipd.it/download/file/fid/55401 (Italian only)Availability:This version is available at: 11577/3167548 since: 2015-11-23T16:47:52Zdx.doi.org/10.1145/2817721.2820986Università degli Studi di PadovaPadua Research Archive - Institutional RepositoryInteractive Spaces: Models andAlgorithms for Reality-Based MusicApplicationsMarcella MandaniciCSC - Sound and MusicComputing GroupDepartment of InformationEngeneeringUniversity of Padova, ITALYmandanici@dei.unipd.itPermission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).ITS ’15, November 15-18, 2015, Funchal, PortugalACM 978-1-4503-3899-8/15/11. http://dx.doi.org/10.1145/2817721.2820986AbstractReality-based interfaces have the property of linking theuser’s physical space with the computer digital content,bringing in intuition, plasticity and expressiveness.Moreover, applications designed upon motion and gesturetracking technologies involve a lot of psychologicalfeatures, like space cognition and implicit knowledge. Allthese elements are the background of three presentedmusic applications, employing the characteristics of threedi↵erent interactive spaces: a user centered threedimensional space, a floor bi-dimensional camera space,and a small sensor centered three dimensional space. Thebasic idea is to deploy the application’s spatial propertiesin order to convey some musical knowledge, allowing theusers to act inside the designed space and to learnthrough it in an enactive way.Author KeywordsBlended interaction; Sound augmented reality; Learningenvironments.ACM Classification KeywordsH.5.1 [Artificial, augmented, and virtual realities]; H.5.2[User Interfaces]; H.5.5 [Sound and Music Computing]Motivation and Research ApproachThe widespread utilization of cameras, camera softwaresystems and motion tracking devices like Kinect(https://www.microsoft.com/en-us/kinectforwindows/)and the Leap Motion Sensor(https://www.leapmotion.com/), has brought to theattention of a continously growing audience of researchers,designers and practitioners the great potentialities ofnewborn interaction styles. Physical environments lying inthe range of these sensors, allow the users to move freelyin everyday spaces, without anything to wear or to hold intheir hands, and to produce some audio or/and visualfeedback from a computer system, connecting in this waythe digital contents to their movements or actions. MyFigure 1: The floor interface ofthe “Harmonic Walk” applicationwhile being tested by a highschool studentresearch focuses on these kinds of spatial interaction, andon how interfaces belonging to the real world can bearranged to deploy the uniqueness of these interactionstyles. In particular, my design experiences concern musicexpressive interaction, music learning environments (seeFig.1) and music production (see Fig.2). Many musicalfeatures, like harmony or melodic movements, have beenhistorically depicted by meaningful spatial representations,e.g. the Euler’s Tonnetz (see Fig.3, literally “web oftones”, a spatial schema showing the triadic relationshipsupon which tonal harmony is based), or the GregorianChant Neumatic Notation and Chironomy, a gesturalsystem expressing melodic contour variations. The verynotion of musical instrument involves precise spatial rulesand element displacement, whose study has produced alarge body of research about new music productioninterfaces in the field of the Sound and Music Computing.Another interesting example of music spatial interaction isthe conductor model, where expressive performanceinformations are transmitted to performers throughfree-hands open-space gestures. This led me to considerthe possibility of employing physical two dimensionalsurfaces or 3D spaces to project geometricalrepresentations of musical concepts, thus enabling usersto enter and to navigate conceptual maps representingsome musical knowledge. The experimental hypothesisupon which my work is based, is that in this situationimplicit and tacit knowledge may emerge and drive theusers to also accomplish very complex tasks, like melodyharmonization, without delivering them any previousinformation. Implicit knowledge refers to our brain’s abilityto acquire an abstract representation of any environmentalstimulus (like for instance an unknown language), learningits rules in an unconscious way. Implicit learning is linkedto the idea of enactivism, that is the ability of learning bydoing. These important cognitive and psychologicalconcepts are both involved in reality-based learningenvironments, which can deploy such powerful tools toconvey information in a faster and more direct way. Theaim of my work is also to study the cognitive content ofmusical knowledge and to try to expand the same learningmethodology towards other domains, like the STEMdisciplines (Science, Technology, Engineering andMathematics), which very often benefit from beingintroduced through creativity and arts-based learning.Research OutlookIn 2010 while attending the 7th Sound and MusicComputing Conference and Summer School in Barcelona Iwas introduced to the Stanza Logomotoria [18], a motiontracking, camera-based application where children areasked to match a story tale with sounds positioned on themapped surface (see Fig.5). Further, the application waspresented in some experimental sessions with musicdidactics students in Conservatories and elementaryschools. Nevertheless, my actual research begun in 2012,when I graduated in Electronic Music with a master thesison Disembodied voices, the first of the three case studiesdescribed below. This experience disclosed to me thegreat potentialities of reality based, full body and freehanded interaction, especially when coupled with a wellestablished human-to-human communication model, likethe one of the music conductor. During my first year ofPhD at the Department of Information Engineering of theUniversity of Padova with the Sound and MusicComputing Group (2013), I examined in depth varioustheories about the representation of music harmony spacesand music image schemas. In 2014 I realized my firstcamera space music application, the Harmonic Walk, thesecond of the three case studies reported below. By theend of 2015 I will conclude my PhD studies, after havingdeveloped several applications for entertainment, learningand creative and expressive music interaction. Presently,together with other members of the CSC research group, Iam working to a spatial collaborative game to enhancemusic listening (Good or Bad) and to a spatial display foran interactive image sonification installation.Figure 2: A child playing aDisklavier through gestural inputdetected by a Leap MotionSensorFigure 3: Euler’s Tonnetz, 1739.Related WorkMy work is based on two di↵erent kinds of background.Firstly the analysis of the origins ([4] and [3]) and of thestate of art of interactive spaces like• Google’s Interactive Spaces and Liquid Galaxy(http://www.interactive-spaces.org/),• Aarhus University Research Centre(http://www.interactivespaces.net/,• Blended Interactions Design Studio, Rochester (NY)(http://blendedinteractions.com/)• UCLA REMAP University of California(http://openptrack.org/)and blended interaction theory [5], providing also a shortsurvey of interactive spaces definitions, typologies,available platforms and themes. I pay also particularattention to existing learning environments like• WizeFloor (https://www.wizefloor.com/)• Smallab (http://smallablearning.com/)• STEP (http://remap.ucla.edu/research/cultural-civic-computing/791-science-through-technology-enhanced-play-step)research centres, arts and culture installations and musicproduction environments. Secondly, all the theoreticalstudies concerning space cognition ([11] and [17]),implicit knowledge ([14] and [16]), enactivism [2] andimage schemas ([6] and [1]), which form aninterconnected interdisciplinary field, very useful tounderstand how reality-based applications work. In Fig.4 atentative conceptual map of the relationships existingamong reality-based applications, implicit knowledge andspace cognition is proposed, outlining reciprocal interplayand dependencies.Three Case StudiesIn this Section three case studies of music applications arepresented. The three cases employ di↵erent motiontracking devices and convey musical concepts throughgeometrical interpretation and spatial representation. Alsothe projections refer to di↵erent spatial models, as thefirst employs the 3D spherical-polar coordinate system,the second 2D Cartesian coordinates on a flat floorsurface and the third 3D space with x, y Cartesiancoordinates plus z plane for depth data.Disembodied Voices[9] is an interactive environment designed for anexpressive, gesture-based musical performance. Themotion sensor Kinect, placed in front of the user, providesthe computer with the 3D polar coordinates of the twoFigure 4: A conceptual map of the main relationships existing among reality-based environments (blue connections), space cognition(green connections) and implicit knowledge (brown connections).hands. The application is designed according to theinteraction model of the music conductor: the user,through gestures, is able to run a score and to produce areal-time expressive interpretation. The conductor movesher/his arms and hands in the space around her/his torsoand in the direction of the performers. Movement analysis([10] and [12]) as well as the teaching practice[15]subdivide the role of the two hands; in general, the rightexecutes musical cues while the left is devoted to iconics,metaphorics and dynamics. As can be seen from Fig.6,the geometrical interpretation of the conductor’sinteraction space is a hemisphere with the center at thelevel of the breastbone of the conductor and the diametercorresponding approximately to the two stretched armslength. Following the conductor’s interaction model, theFigure 5: The “StanzaLogomotoria” basicconfiguration, with computer,audio monitors and ceilingmounted camera.Figure 6: The hemisphericalregions for user hands interactionin “Disembodied Voices”. Forthe left hand there is an innerregion and two outer regions, onein front and the other at the sideof the user.hemisphere is subdivided in two parts, one for the rightand the other one for the left hand. For the left handthree di↵erent regions are shaped, triggering variousdigital sound processing e↵ects. (video available athttps://www.youtube.com/watch?v=oyf7GrMMrL8)The Harmonic Walk[8] is an interactive physical environment designed forexperiencing a novel spatial approach to musical creation.In particular, the system allows the user to get in touchwith some fundamental tonal music features in a verysimple and readily available way. The application’sinterface consists of a camera placed on the ceiling whichcan trace the presence of a user who walks on a flatsurface within the camera’s view. The Harmonic Walk,through the body movement in space, can provide a liveexperience of tonal melody structure, chord progressions,melody accompaniment and improvisation. Enactiveknowledge and embodied cognition allow the user to buildan inner map of these musical features, which can beacted by moving on the mapped surface with a simplestep. Listeners interpret a tonal melody grouping theperceived sequence of events after a metrical andharmonic frame [13]. This produces a segmentation of thecomposition into di↵erent harmonic regions which, withthe underlying harmonic structure, are the leading featuresof a tonal composition. The time proceeding of thevarious musical units is led by the melody, whosemetaphoric scheme is expressed by the so-called“source-path-goal” schema [6]. Following this metaphorFigure 7: Visual tags of thestraight and circular path of the“Harmonic Walk’s” interface.and employing the simplest motion in space a human cando - the walk - it is possible to represent a tonalcomposition as a sequence of spatial blocks, where eachstep corresponds to the next musical unit (white crosses inFig.7), while the harmonic space is represented by a sixparts sliced circular ring containing the six roots of thetonality harmonic space (black crosses). (video availableat https://www.youtube.com/watch?v=OjwXfzq_CkU&index=1&list=UU1E9xCq8TWqlzessRIzUGxw)Hand Composer[7] is a gesture-driven composition system, based on theanalysis of the existing relationships between musicgenerative models and musical composition in the contextof the XX century music history background. The systemframework is based on a number of interactive machinesperforming various patterns of music composition andproducing a stream of MIDI data to be compatible with aDisklavier performance. Hand gestural input, captured bythe Leap Motion sensor (see Fig.8), can control someparameters of the music composing machines, changinginteractively their musical output. (video available athttps://www.youtube.com/watch?v=mdsn9_5Ig_A&list=UU1E9xCq8TWqlzessRIzUGxw)Figure 8: The 3D interaction space of the Hand Composerapplication. x, y Cartesian coordinates are employed to mapthe two-dimensional vertical plane, while z coordinates mapthe depth data.Thesis Scope and Expected ContributionsThe experimental hypothesis upon which all my researchis based, is that abstract knowledge like musical harmonyor tonal music composition structure can be conveyed tothe users through the enactive learning induced byinteractive space environments. Thus, my thesis will focuson the relationship between space and cognition, and onthe ways knowledge can be spatially represented. As Ihave worked with various kinds of interactive spaces(bi-dimensional floors as well as three-dimensionalvolumes), I will try to work out a comprehensive, unifyingframework for all of them and to highlight their respectiveproperties and di↵erences. If my experimental hypothesisis verified, reality-based learning environments designcould benefit of very powerful cognitive tools, which couldbe extended beyond music also to other knowledge fields,like science, mathematics and technology.AcknowledgementsI am grateful to my supervisor Sergio Canazza for hisprecious support and help. I also want to express mygratitude to Antonio Roda` who followed step by step thedevelopment of my work and to Davide Rocchesso for hisadvices and presence.References[1] Brower, C. A cognitive theory of musical meaning.Journal of Music Theory (2000), 323–379.[2] Bruner, J. S. The act of discovery. Harvardeducational review (1961).[3] Engelbart, D. C. Augmenting human intellect: aconceptual framework (1962). PACKER, Randall andJORDAN, Ken. Multimedia. From Wagner to VirtualReality. New York: WW Norton & Company (2001),64–90.[4] Harrison, S., and Dourish, P. Re-place-ing space: theroles of place and space in collaborative systems. InProc. ACM conference on Computer supportedcooperative work, ACM (1996), 67–76.[5] Jetter, H.-C., Reiterer, H., and Geyer, F. Blendedinteraction: understanding natural human–computerinteraction in post-wimp interactive spaces. Personaland Ubiquitous Computing 18, 5 (2014), 1139–1158.[6] Lako↵, G., and Johnson, M. Metaphors we live by.University of Chicago press, 2008.[7] Mandanici, M., and Canazza, S. The “handcomposer”: gesture-driven music compositionmachines. In Proc. 13th Conf. on IntelligentAutonomous Systems (2014), 553–560.[8] Mandanici, M., Roda`, A., and Canazza, S. Theharmonic walk: an interactive educationalenvironment to discover musical chords. In Proc.ICMC-SMC Conference (2014).[9] Mandanici, M., and Sapir, S. Disembodied voices: akinect virtual choir conductor. In Proc. 9th Soundand Music Computing Conference (2012), 271–276.[10] Marrin, T., and Picard, R. The ‘conductor’s jacket’:A device for recording expressive musical gestures. InProc. International Computer Music Conference(1998), 215–219.[11] Montello, D. R. International Encyclopedia of theSocial and Behavioral Sciences. Oxford, PergamonPress, 2001, ch. Spatial Cognition, 14771 – 14775.[12] Murphy, D., Andersen, T. H., and Jensen, K.Conducting audio files via computer vision. InGesture-based communication in human-computerinteraction. Springer, 2004, 529–540.[13] Povel, D.-J., and Jansen, E. Harmonic factors in theperception of tonal melodies. Music Perception 20, 1(2002), 51–85.[14] Reber, A. S. Implicit learning and tacit knowledge.Journal of experimental psychology: General 118, 3(1989), 219.[15] Rudolf, M., and Stern, M. The grammar ofconducting: A comprehensive guide to batontechnique and interpretation. Schirmer Books, NewYork, 1994.[16] Tillmann, B., Bharucha, J. J., and Bigand, E.Implicit learning of tonality: a self-organizingapproach. Psychological review 107, 4 (2000), 885.[17] Tversky, B. Functional Significance of VisuospatialRepresentations. Handbook of higher-levelvisuospatial thinking Handbook of (2005), 1–34.[18] Zanolla, S., Canazza, S., Roda`, A., Camurri, A., andVolpe, G. Entertaining listening by means of theStanza Logo-Motoria: an Interactive MultimodalEnvironment. Entertainment Computing 2013, 4(2013), 213–220.

https://www.research.unipd.it/bitstream/11577/3167548/1/ITS15_DS_Mandanici_def.pdf

Interactive Spaces. Models and Algorithms for Reality-based Music Applications

Abstract

Similar works

Full text

Available Versions

Padua Research Archive

Archivio istituzionale della ricerca - Università di Padova