Interactive Spaces. Models and Algorithms for Reality-based Music Applications

Reality-based interfaces have the property of linking the user's physical space with the computer digital content, bringing in intuition, plasticity and expressiveness. Moreover, applications designed upon motion and gesture tracking technologies involve a lot of psychological features, like space cognition and implicit knowledge. All these elements are the background of three presented music applications, employing the characteristics of three different interactive spaces: a user centered three dimensional space, a floor bi-dimensional camera space, and a small sensor centered three dimensional space. The basic idea is to deploy the application's spatial properties in order to convey some musical knowledge, allowing the users to act inside the designed space and to learn through it in an enactive way

Interactive Spaces: Model for Motion-based Music Applications

With the extensive utilization of touch screens, smartphones and various reactive surfaces, reality- based and intuitive interaction styles have now become customary. The employment of larger interactive areas, like floors or peripersonal three-dimensional spaces, further increase the reality- based interaction affordances, allowing full-body involvement and the development of a co- located, shared user experience. Embodied and spatial cognition play a fundamental role for the interaction in this kind of spaces, where users act in the reality with no device in the hands and obtain an audio and graphical output depending on their movements. Starting from the early experiments of Myron Krueger in 1971, responsive floors have been developed through various technologies including sensorized tiles and computer vision systems, to be employed in learn- ing environments, entertainment, games and rehabilitation. Responsive floors allow the spatial representation of concepts and for this reason are suitable for immediate communication and engagement. As many musical features have meaningful spatial representations, they can easily be reproduced in the physical space through a conceptual blending approach and be made available to a great number of users. This is the key idea for the design of the original music applications presented in this thesis. The applications, devoted to music learning, production and active listening, introduce a novel creative approach to music, which can be further assumed as a general paradigm for the design of motion-based learning environments. Application assessment with upper elementary and high school students has proved that users engagement and bodily inter- action have a high learning power, which can be a valid resource for deeper music knowledge and more creative learning processes. Although further interface tests showed that touch screen interaction performs better than full-body interaction, some important guidelines for the design of reactive floors applications have been obtained on the basis of these test results. Moreover, the conceptual framework developed for the design of music applications can represent a valid paradigm also in the general field of human-computer interaction

Interactive Spaces: Model for Motion-based Music Applications

With the extensive utilization of touch screens, smartphones and various reactive surfaces, reality- based and intuitive interaction styles have now become customary. The employment of larger interactive areas, like floors or peripersonal three-dimensional spaces, further increase the reality- based interaction affordances, allowing full-body involvement and the development of a co- located, shared user experience. Embodied and spatial cognition play a fundamental role for the interaction in this kind of spaces, where users act in the reality with no device in the hands and obtain an audio and graphical output depending on their movements. Starting from the early experiments of Myron Krueger in 1971, responsive floors have been developed through various technologies including sensorized tiles and computer vision systems, to be employed in learn- ing environments, entertainment, games and rehabilitation. Responsive floors allow the spatial representation of concepts and for this reason are suitable for immediate communication and engagement. As many musical features have meaningful spatial representations, they can easily be reproduced in the physical space through a conceptual blending approach and be made available to a great number of users. This is the key idea for the design of the original music applications presented in this thesis. The applications, devoted to music learning, production and active listening, introduce a novel creative approach to music, which can be further assumed as a general paradigm for the design of motion-based learning environments. Application assessment with upper elementary and high school students has proved that users engagement and bodily inter- action have a high learning power, which can be a valid resource for deeper music knowledge and more creative learning processes. Although further interface tests showed that touch screen interaction performs better than full-body interaction, some important guidelines for the design of reactive floors applications have been obtained on the basis of these test results. Moreover, the conceptual framework developed for the design of music applications can represent a valid paradigm also in the general field of human-computer interaction.Con l’utilizzo intensivo di touch screen, smartphones e varie superfici sensibili al tocco, stili di interazione intuitivi e ispirati alla realtà sono ormai diventati di uso comune. L’utilizzo di più estese superfici interattive quali pavimenti o spazi peripersonali tridimensionali, aumenta ulteriormente le possibilità offerte dall’interazione basata sulla realtà, coinvolgendo l’intero corpo e consentendo la condivisione dell’esperienza da parte di più utenti. La cognizione legata al corpo e allo spazio gioca un ruolo fondamentale per l’interazione in questi tipi di ambienti, dove l’utente agisce nella realtà senza dover tenere in mano alcun sensore, producendo un risultato sonoro e visivo in dipendenza dei suoi movimenti. Partendo dai primi esperimenti di Myron Krueger nel 1971, i pavimenti interattivi si sono sviluppati utilizzando varie tecnologie, dalle mattonelle sensorizzate a vari sistemi di computer vision, al fine di essere impiegati come ambienti per l’apprendimento e l’intrattenimento, per i giochi e per la riabilitazione. I pavimenti interattivi consentono la rappresentazione spaziale dei concetti e per questo motivo si prestano ad una comunicazione immediata e coinvolgente. Poiché molti elementi del linguaggio musicale sono espressi attraverso significative rappresentazioni spaziali, essi possono essere facilmente riprodotti nello spazio fisico utilizzando la teoria del “conceptual blending”, ed essere cos`ı resi disponibili ad un numero assai esteso di utenti. Questo concetto e` alla base della progettazione delle applicazioni musicali originali presentate in questa tesi. Le applicazioni, finalizzate all’apprendimento musicale, alla produzione e all’ascolto attivo, introducono un nuovo approccio creativo alla musica, che può essere considerato paradigmatico per la progettazione di ambienti educativi basati sul movimento. I test effettuati con alunni del secondo ciclo della scuola elementare e con studenti della scuola superiore hanno dimostrato che il coinvolgimento degli utenti e l’interazione “full-body” hanno una grande capacita` di influenzare l’apprendimento, e che possono essere efficacemente utilizzati per un approfondimento della cultura musicale e per un approccio creativo nei processi formativi. Sebbene ulteriori test sull’interfaccia abbiano di- mostrato che gli utenti preferiscono il touch screen all’interazione “full-body”, in base a questi risultati sono state delineate alcune importanti linee guida per la futura progettazione di applicazioni su pavimenti interattivi. Inoltre, la struttura concettuale sviluppata per la progettazione di applicazioni musicali può rappresentare anche un valido paradigma nel campo dell’interazione uomo-macchina

The "Hand Composer": gesture-driven Music Composition Machines

A gesture-driven, composition system framework called "The Hand Composer" is presented. The framework is based on the analysis of the existing relationships between music generative models and musical composition in the context of the 20th century music historical background. A short definition of three music generative models with their control parameters is provided. The system framework is based on a number of interactive machines performing various patterns of music composition and producing a stream of MIDI data to be compatible with a Disklavier performance. Hand gestural input, captured by the Leap Motion Sensor, can control some parameters of the music composing machines, changing interactively and in real time their musical output