Immersive Composition for Sensory Rehabilitation: 3D Visualisation, Surround Sound, and Synthesised Music to Provoke Catharsis and Healing

There is a wide range of sensory therapies using sound, music and visual stimuli. Some focus on soothing or distracting stimuli such as natural sounds or classical music as analgesic, while other approaches emphasize the active performance of producing music as therapy. This paper proposes an immersive multi-sensory Exposure Therapy for people suffering from anxiety disorders, based on a rich, detailed surround-soundscape. This soundscape is composed to include the users’ own idiosyncratic anxiety triggers as a form of habituation, and to provoke psychological catharsis, as a non-verbal, visceral and enveloping exposure. To accurately pinpoint the most effective sounds and to optimally compose the soundscape we will monitor the participants’ physiological responses such as electroencephalography, respiration, electromyography, and heart rate during exposure. We hypothesize that such physiologically optimized sensory landscapes will aid the development of future immersive therapies for various psychological conditions, Sound is a major trigger of anxiety, and auditory hypersensitivity is an extremely problematic symptom. Exposure to stress-inducing sounds can free anxiety sufferers from entrenched avoidance behaviors, teaching physiological coping strategies and encouraging resolution of the psychological issues agitated by the sound

Development of an Augmented Reality musical instrument

Nowadays, Augmented Reality and Virtual Reality are concepts of which people are becoming more and more aware of due to their application to the video-game industry (speceially in the case of VR). Such raise is partly due to a decrease in costs of Head Mounted Displays, which are consequently becoming more and more accessible to the public and developers worldwide. All of these novelties, along with the frenetic development of Information Technologies applied to essentially, all markets; have also made digital artists and manufacturers aware of the never-ending interaction possibilities these paradigms provide and a variety of systems have appeared, which offer innovative creative capabilities. Due to the personal interest of the author in music and the technologies surrounding its creation by digital means, this document covers the application of the Virtuality- Reality-Continuum (VR and AR) paradigms to the field of interfaces for the musical expression. More precisely, it covers the development of an electronic drumset which integrates Arduino-compatible hardware with a 3D visualisation application (developed based on Unity) to create a complete functioning instrument musical instrument, The system presented along the document attempts to leverage three-dimensional visual feedback with tangible interaction based on hitting, which is directly translated to sound and visuals in the sound generation application. Furthermore, the present paper provides a notably deep study of multiple technologies and areas that are ultimately applied to the target system itself. Hardware concerns, time requirements, approaches to the creation of NIMEs (New Interfaces for Musical Expression), Virtual Musical Instrument (VMI) design, musical-data transmission protocols (MIDI and OSC) and 3D modelling constitute the fundamental topics discussed along the document. At the end of this paper, conclusions reflect on the difficulties found along the project, the unfulfilled objectives and all deviations from the initial concept that the project suffered during the development process. Besides, future work paths will be listed and depicted briefly and personal comments will be included as well as humble pieces of advice targeted at readers interested in facing an ambitious project on their own.En la actualidad, los conceptos de Realidad Aumentada (AR) y Realidad Virtual (VR) son cada vez más conocidos por la gente de a pie, debido en gran parte a su aplicación al ámbito de los videojuegos, donde el desarollo para dispositivos HMDs está en auge. Esta popularidad se debe en gran parte al abaratamiento de este tipo de dispositivos, los cuales son cada vez más accesibles al público y a los desarrolladores de todo el mundo. Todas estas novedades sumadas al frenético desarrollo de la industria de IT han llamado la atención de artistas y empresas que han visto en estos paradigmas (VR and AR) una oportunidad para proporcionar nuevas e ilimitadas formas de interacción y creación de arte en alguna de sus formas. Debido al interés personal del autor de este TFG en la música y las tecnologías que posiblitan la creación musical por medios digitales, este documento explora la aplicación de los paradigmas del Virtuality-Reality Continuum de Milgram (AR y VR) al ámbito de las interfaces para la creación musical. Concretamente, este TFG detalla el desarrollo de una batería electrónica, la cual combina una interfaz tangible creada con hardware compatible con Arduino con una aplicación de generación de sonidos y visualización, desarrollada utilizando Unity como base. Este sistema persigue lograr una interacción natural por parte del usuario por medio de integrar el hardware en unas baquetas, las cuales permiten detectar golpes a cualquier tipo de superficie y convierten estos en mensajes MIDI que son utilizados por el sistema generador de sonido para proporcionar feedback al usuario (tanto visual como auditivo); por tanto, este sistema se distingue por abogar por una interacción que permita golpear físicamente objetos (e.g. una cama), mientras que otros sistemas similates basan su modo de interacción en “air-drumming”. Además, este sistema busca solventar algunos de los inconvenientes principales asociados a los baterías y su normalmente conflictivo instrumento, como es el caso de las limitaciones de espacio, la falta de flexibilidad en cuanto a los sonidos que pueden ser generados y el elevado coste del equipo. Por otro lado, este documento pormenoriza diversos aspectos relacionados con el sistema descrito en cuestión, proporcionando al lector una completa panorámica de sistemas similares al propuesto. Asimismo, se describen los aspectos más importantes en relación al desarrollo del TFG, como es el caso de protocolos de transmisión de información musical (MIDI y OSC), algoritmos de control, guías de diseño para interfaces de creación musical (NIMEs) y modelado 3D. Se incluye un íntegro proceso de Ingeniería de Software para mantener la formalidad y tratar de garantizar un desarrollo más organizado y se discute la metodología utilizada para este proceso. Por último, este documento reflexiona sobre las dificultades encontradas, se enumeran posibilidades de Trabajo Futuro y se finaliza con algunas conclusiones personales derivadas de este trabajo de investigación.Ingeniería Informátic

Micro-motion controller II

The purpose of this project was to improve upon the functionality of a micro-motion controller designed by another senior design group at this university. The original controller design facilitated motion in only two dimensions, and by modifying the platform of the design to accommodate another axis of rotation, we were able to achieve a full range of 3-D motion in our own product. Additionally, we designed a new system in which a motor could be mounted on the base plate of the device which would rotate the upper platform on its own through a simple belt-and-pulley system. And lastly, we designed and added a gripper to the end of the rotating arm that can effectively grab and move objects. Ideally, this project exists as a rudimentary display of the concepts used in various other micro-motion output devices in the biotech industry, such as the DaVinci robotic surgery machine. Practically, we were able to produce a product that was an effective redesign of a system that lacked any sort of 3-D motion

Designing and Composing for Interdependent Collaborative Performance with Physics-Based Virtual Instruments

Interdependent collaboration is a system of live musical performance in which performers can directly manipulate each other’s musical outcomes. While most collaborative musical systems implement electronic communication channels between players that allow for parameter mappings, remote transmissions of actions and intentions, or exchanges of musical fragments, they interrupt the energy continuum between gesture and sound, breaking our cognitive representation of gesture to sound dynamics. Physics-based virtual instruments allow for acoustically and physically plausible behaviors that are related to (and can be extended beyond) our experience of the physical world. They inherently maintain and respect a representation of the gesture to sound energy continuum. This research explores the design and implementation of custom physics-based virtual instruments for realtime interdependent collaborative performance. It leverages the inherently physically plausible behaviors of physics-based models to create dynamic, nuanced, and expressive interconnections between performers. Design considerations, criteria, and frameworks are distilled from the literature in order to develop three new physics-based virtual instruments and associated compositions intended for dissemination and live performance by the electronic music and instrumental music communities. Conceptual, technical, and artistic details and challenges are described, and reflections and evaluations by the composer-designer and performers are documented

The Haptic Bracelets: learning multi-limb rhythm skills from haptic stimuli while reading

The Haptic Bracelets are a system designed to help people learn multi-limbed rhythms (which involve multiple simultaneous rhythmic patterns) while they carry out other tasks. The Haptic Bracelets consist of vibrotactiles attached to each wrist and ankle, together with a computer system to control them. In this chapter, we report on an early empirical test of the capabilities of this system, and consider de-sign implications. In the pre-test phase, participants were asked to play a series of multi-limb rhythms on a drum kit, guided by audio recordings. Participants’ per-formances in this phase provided a base reference for later comparisons. During the following passive learning phase, away from the drum kit, just two rhythms from the set were silently 'played' to each subject via vibrotactiles attached to wrists and ankles, while participants carried out a 30-minute reading comprehen-sion test. Different pairs of rhythms were chosen for different subjects to control for effects of rhythm complexity. In each case, the two rhythms were looped and alternated every few minutes. In the final phase, subjects were asked to play again at the drum kit the complete set of rhythms from the pre-test, including, of course, the two rhythms to which they had been passively exposed. Pending analysis of quantitative data focusing on accuracy, timing, number of attempts and number of errors, in this chapter we present preliminary findings based on participants’ sub-jective evaluations. Most participants thought that the technology helped them to understand rhythms and to play rhythms better, and preferred haptic to audio to find out which limb to play when. Most participants indicated that they would pre-fer using a combination of haptics and audio for learning rhythms to either mo-dality on its own. Replies to open questions were analysed to identify design is-sues, and implications for design improvements were considered

Tabletop Tangible Interfaces for Music Performance: Design and Evaluation

This thesis investigates a new generation of collaborative systems: tabletop tangible interfaces (TTIs) for music performance or musical tabletops. Musical tabletops are designed for professional musical performance, as well as for casual interaction in public settings. These systems support co-located collaboration, offered by a shared interface. However, we still know little about their challenges and opportunities for collaborative musical practice: in particular, how to best support beginners or experts or both. This thesis explores the nature of collaboration on TTIs for music performance between beginners, experts, or both. Empirical work was done in two stages: 1) an exploratory stage; and 2) an experimental stage. In the exploratory stage we studied the Reactable, a commercial musical tabletop designed for beginners and experts. In particular, we explored its use in two environments: a multi-session study with expert musicians in a casual lab setting; and a field study with casual visitors in a science centre. In the experimental stage we conducted a controlled experiment for mixed groups using a bespoke musical tabletop interface, SoundXY4. The design of this study was informed by the previous stage about a need to support better real-time awareness of the group activity (workspace awareness) in early interactions. For the three studies, groups musical improvisation was video-captured unobtrusively with the aim of understanding natural uses during group musical practice. Rich video data was carefully analysed focusing on the nature of social interaction and how workspace awareness was manifested. The findings suggest that musical tabletops can support peer learning during multiple sessions; fluid between-group social interaction in public settings; and a democratic and ecological approach to music performance. The findings also point to how workspace awareness can be enhanced in early interactions with TTIs using auditory feedback with ambisonics spatialisation. The thesis concludes with theoretical, methodological, and practical implications for future research in New Interfaces for Musical Expression (NIME), tabletop studies, and Human-Computer Interaction (HCI)

Musical Instruments for Novices: Comparing NIME, HCI and Crowdfunding Approaches

Designing musical instruments to make performance accessible to novice musicians is a goal which long predates digital technology. However, just in the space of the past 6 years, dozens of instrument designs have been introduced in various academic venues and in commercial crowdfunding campaigns. In this paper, we draw comparisons in design, evaluation and marketing across four domains: crowdfunding campaigns on Kickstarter and Indiegogo; the New Interfaces for Musical Expression (NIME) conference; conferences in human-computer interaction (HCI); and researchers creating accessible instruments for children and adults with disabilities. We observe striking differences in approach between commercial and academic projects, with less pronounced differences between each of the academic communities. The paper concludes with general reflections on the identity and purpose of instruments for novice musicians, with suggestions for future exploration

Multiparametric interfaces for fine-grained control of digital music

Digital technology provides a very powerful medium for musical creativity, and the way in which we interface and interact with computers has a huge bearing on our ability to realise our artistic aims. The standard input devices available for the control of digital music tools tend to afford a low quality of embodied control; they fail to realise our innate expressiveness and dexterity of motion. This thesis looks at ways of capturing more detailed and subtle motion for the control of computer music tools; it examines how this motion can be used to control music software, and evaluates musicians’ experience of using these systems. Two new musical controllers were created, based on a multiparametric paradigm where multiple, continuous, concurrent motion data streams are mapped to the control of musical parameters. The first controller, Phalanger, is a markerless video tracking system that enables the use of hand and finger motion for musical control. EchoFoam, the second system, is a malleable controller, operated through the manipulation of conductive foam. Both systems use machine learning techniques at the core of their functionality. These controllers are front ends to RECZ, a high-level mapping tool for multiparametric data streams. The development of these systems and the evaluation of musicians’ experience of their use constructs a detailed picture of multiparametric musical control. This work contributes to the developing intersection between the fields of computer music and human-computer interaction. The principal contributions are the two new musical controllers, and a set of guidelines for the design and use of multiparametric interfaces for the control of digital music. This work also acts as a case study of the application of HCI user experience evaluation methodology to musical interfaces. The results highlight important themes concerning multiparametric musical control. These include the use of metaphor and imagery, choreography and language creation, individual differences and uncontrol. They highlight how this style of interface can fit into the creative process, and advocate a pluralistic approach to the control of digital music tools where different input devices fit different creative scenarios