Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

An Embodied Perspective on Piano Timbre: Conceptualisation and Communication in Performance and Educational Context.

This thesis presents three empirical studies which explore the conceptualisation and communication of piano timbre from the perspective of the performer, the listener, and the pairing of teacher and student. The research started from the perspective of the debate between acousticians and musicians on touch-tone relationships, with the conviction that piano timbre should not be only concerned with the examination of sonic outcomes, but should aim to understand the phenomenology of experiences and conceptions related to piano timbre. Less is known about what contributes to the conceptualisation of piano timbre in terms of metaphor, cross-modal experience, mental images etc.; whether, and how, a timbral intention in piano performance can be communicated to others via sound and/or the body. By adopting an embodied perspective, this research has focused on an exploration of sound-gesture relationships in the perception and production of piano timbre, using a mix of qualitative and quantitative approaches. The first interview study indicated that expressive gestures and the introspective experience of performers affect the way they perceive and describe piano timbre. The second study examined the embodiment of piano timbre and demonstrated through a perceptual experiment that the visual component of piano performance influences the perceived timbral experience of listeners. The communication of piano timbre is multimodal and integrates aspects from visual, tactile, kinaesthetic and sonic dimensions. The third teaching observation study implied that shared understanding of piano timbre is an emergent and enactive product in a piano lesson through the real-time collaboration and participation of both the teacher and the student. The whole thesis contributes to the understanding of embodied music cognition and has implications for expressive piano performance and teaching practice

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

How touch and hearing influence visual processing in sensory substitution, synaesthesia and cross-modal correspondences

Sensory substitution devices (SSDs) systematically turn visual dimensions into patterns of tactile or auditory stimulation. After training, a user of these devices learns to translate these audio or tactile sensations back into a mental visual picture. Most previous SSDs translate greyscale images using intuitive cross-sensory mappings to help users learn the devices. However more recent SSDs have started to incorporate additional colour dimensions such as saturation and hue. Chapter two examines how previous SSDs have translated the complexities of colour into hearing or touch. The chapter explores if colour is useful for SSD users, how SSD and veridical colour perception differ and how optimal cross-sensory mappings might be considered. After long-term training, some blind users of SSDs report visual sensations from tactile or auditory stimulation. A related phenomena is that of synaesthesia, a condition where stimulation of one modality (i.e. touch) produces an automatic, consistent and vivid sensation in another modality (i.e. vision). Tactile-visual synaesthesia is an extremely rare variant that can shed light on how the tactile-visual system is altered when touch can elicit visual sensations. Chapter three reports a series of investigations on the tactile discrimination abilities and phenomenology of tactile-vision synaesthetes, alongside questionnaire data from synaesthetes unavailable for testing. Chapter four introduces a new SSD to test if the presentation of colour information in sensory substitution affects object and colour discrimination. Chapter five presents experiments on intuitive auditory-colour mappings across a wide variety of sounds. These findings are used to predict the reported colour hallucinations resulting from LSD use while listening to these sounds. Chapter six uses a new sensory substitution device designed to test the utility of these intuitive sound-colour links for visual processing. These findings are discussed with reference to how cross-sensory links, LSD and synaesthesia can inform optimal SSD design for visual processing

Musical Haptics

This Open Access book offers an original interdisciplinary overview of the role of haptic feedback in musical interaction. Divided into two parts, part I examines the tactile aspects of music performance and perception, discussing how they affect user experience and performance in terms of usability, functionality and perceived quality of musical instruments. Part II presents engineering, computational, and design approaches and guidelines that have been applied to render and exploit haptic feedback in digital musical interfaces. Musical Haptics introduces an emerging field that brings together engineering, human-computer interaction, applied psychology, musical aesthetics, and music performance. The latter, defined as the complex system of sensory-motor interactions between musicians and their instruments, presents a well-defined framework in which to study basic psychophysical, perceptual, and biomechanical aspects of touch, all of which will inform the design of haptic musical interfaces. Tactile and proprioceptive cues enable embodied interaction and inform sophisticated control strategies that allow skilled musicians to achieve high performance and expressivity. The use of haptic feedback in digital musical interfaces is expected to enhance user experience and performance, improve accessibility for disabled persons, and provide an effective means for musical tuition and guidance

Timbral Analysis and Recording Parameter Transformations of Snare Drums

A snare drum is capable of producing a wide range of timbres influenced by playing technique, its physical construction, and the recording methods used. When a recording engineer configures drums and studio equipment, they adjust a plethora of real-world recording parameters to achieve the desired timbre. These recording parameters impart their own timbral properties by varying amounts, and in most cases the only way to modify these properties is to re-record the audio with changes applied to the real-world variables. This thesis examines methods for computational transformations of snare drum recordings to elicit perceptual changes that mimic modification of real-world recording variables. This is achieved through four main investigations, presented throughout this thesis, two which cover timbral analysis of snare drum recordings, and two which explore post-hoc recording parameter transformations. Strike velocity and microphone selection are factors known to affect snare drum timbre, the first study analyses timbral differences associated with snare drum strike velocity. Results show that listeners are able to distinguish between high and low velocity strikes using timbral cues alone, with microphone selection having no influence on this perceptual identification. Audio analysis reveals distinct temporal and spectral features, with higher velocity strikes producing greater energy in the lower mid-range and significantly longer decay times. The second study aims to demystify the subjective preference of different microphones for snare drum recording. For the majority of microphones, preference does not change between isolated strikes and those with the presence of bleed from the hi-hat and kick drum. On average, preference is higher for condenser microphones compared to dynamic. Additionally, spectral centroid and an objective measure of brightness positively correlate with subjective scores. The ability to perceptually modify drum recording parameters in a post-recording process would be of great benefit to engineers limited by time or equipment. The first post-hoc recording parameter transformation study focuses on microphone selection, mapping the spectral features from highly-preferred microphones onto a microphone with less favourable timbral characteristics. This investigation also details the development and evaluation of a robotic drum arm for consistent strike velocity. Subjective assessment reveals that participants show no preferences between recordings from highly-preferred microphones and those from a transformed least-preferred microphone. The last study employs a data-driven approach for post-recording modification of dampening and microphone position. The system consists of a autoencoder that analyses an audio input and predicts optimal parameters of one or more third-party audio effects, which process the audio to produce the desired transformations. Two novel audio effects are proposed and compared against existing audio plugins. Perceptual quality of transformations is assessed through a subjective listening test and an object evaluation is used to measure system performance, positive results demonstrate a capacity to emulate snare dampening