Synesthetic music experience communicator

The Synesthetic Music Experience Communicator combines research in the areas of human computer interaction, music technology, and human perception to illustrate the experience of synesthesian mental imagery in response to musical sounds. Synesthesian musicians have reported positive benefits from their augmented awareness of sound in the areas of pitch identification, memorization, composition, and improvisation;This dissertation attempts to communicate both the experience and performance benefits associated with this cognitive phenomenon. Several virtual worlds have been developed to explore group education, rehearsal, and the artistic transformation of live performances into informative and entertaining visual presentations. Initial inspirations, background research, development stages, iterations, user evaluation, and future directions are discussed;Two virtual reality prototype systems are overviewed. The first demonstrates chromesthetic translations of real-time Musical Instrument Digital Interface (MIDI) events. The second demonstrates real-time transformations of multi-band Fast Fourier Transform processed audio into visual scenery. Observations and feedback about the initial prototype systems are summarized;Three systems are proposed which expand the initial prototypes and demonstrate concepts for computer augmented ensemble rehearsals. The Synesthetic Visualizer modules combine real-time audio and MIDI data to demonstrate synesthesia and photism-like imagery. The Computer Augmented Percussion Trainer and Small Ensemble Trainer are proposed to augment musical practices and rehearsals by providing real-time displays of ensemble members\u27 performances and displaying transformations between instrument fingerings;An introductory user study is conducted to determine which aspects of music are best communicated by the visual displays and to evaluate the potential benefits of this synesthetic approach. The user study asks participants to consider this research in relationship to existing music visualization and education methods. Exhibitions and publication efforts are reviewed. The user study, observations, and exhibitions serve to validate the core hypothesis of this research. The dissertation concludes by proposing an intelligent interactive synesthetic software agent to facilitate profile driven multimedia content creation

The Sonic Scanner and the Graphonic Interface

This paper describes the concepts, design, implementation, and evaluation of two new interfaces for music performance and composition. Both of the interfaces were motivated by the idea of creating music through drawing, but they approach the activity in very different ways. While the Graphonic Interface allows you to make music as you are drawing, the Sonic Scanner requires pre-composed graphic material in order to make music. However, both of the devices are real-time controllers that produce sound in an interactive manner, thereby allowing them to be used as performance instruments.