To “Sketch-a-Scratch”

A surface can be harsh and raspy, or smooth and silky, and everything in between. We are used to sense these features with our fingertips as well as with our eyes and ears: the exploration of a surface is a multisensory experience. Tools, too, are often employed in the interaction with surfaces, since they augment our manipulation capabilities. “Sketch-a-Scratch” is a tool for the multisensory exploration and sketching of surface textures. The user’s actions drive a physical sound model of real materials’ response to interactions such as scraping, rubbing or rolling. Moreover, different input signals can be converted into 2D visual surface profiles, thus enabling to experience them visually, aurally and haptically

Enhancing the Multisensory Environment with Adaptive Game Audio Techniques

Two workshop case studies are described that document the potential for applying novel approaches to the use of technology in multisensory environments. In contrast to current trends, the environments were regarded as a space within which to work rather than as a given set of technologies. Stimulating interactive story-worlds were enabled for groups of children with mixed Special Educational Needs where technology could empower the different groups to affect the environment as a whole. Arts-based leadership experience was regarded as key within the studies as were methods for moving beyond passive modes of interaction with sound and music. A novel approach to creating interactive ‘soundtracks’ is described that takes adaptive techniques from computer game audio and applies these within a physical space

Keytar: Melodic control of multisensory feedback from virtual strings

A multisensory virtual environment has been designed, aiming at recreating a realistic interaction with a set of vibrating strings. Haptic, auditory and visual cues progressively istantiate the environment: force and tactile feedback are provided by a robotic arm reporting for string reaction, string surface properties, and furthermore defining the physical touchpoint in form of a virtual plectrum embodied by the arm stylus. Auditory feedback is instantaneously synthesized as a result of the contacts of this plectrum against the strings, reproducing guitar sounds. A simple visual scenario contextualizes the plectrum in action along with the vibrating strings. Notes and chords are selected using a keyboard controller, in ways that one hand is engaged in the creation of a melody while the other hand plucks virtual strings. Such components have been integrated within the Unity3D simulation environment for game development, and run altogether on a PC. As also declared by a group of users testing a monophonic Keytar prototype with no keyboard control, the most significant contribution to the realism of the strings is given by the haptic feedback, in particular by the textural nuances that the robotic arm synthesizes while reproducing physical attributes of a metal surface. Their opinion, hence, argues in favor of the importance of factors others than auditory feedback for the design of new musical interfaces

To Sketch-a-Scratch

A surface can be harsh and raspy, or smooth and silky, and everything in between. We are used to sense these features with our fingertips as well as with our eyes and ears: the exploration of a surface is a multisensory experience. Tools, too, are often employed in the interaction with surfaces, since they augment our manipulation capabilities. “Sketch-a-Scratch” is a tool for the multisensory exploration and sketching of surface textures. The user’s actions drive a physical sound model of real materials’ response to interactions such as scraping, rubbing or rolling. Moreover, different input signals can be converted into 2D visual surface profiles, thus enabling to experience them visually, aurally and haptically

Designing for Musical Play

Though sensory spaces are a common feature within many special needs schools, the way in which they are designed and resourced varies greatly between provisions as do the types of activities that take place within them. A short series of case studies has been carried out across a cross section of UK special needs schools to demonstrate this contrast whilst also attempting to better understand the reasoning and motivation behind their design and usage. In 2012, eight schools were visited in England and Wales with the aim of documenting the types of sensory space that were available, the resources that were featured within each space and the types of sensory activities that were being used. The key themes that emerged during the case studies are discussed alongside an historical overview of the conception and evolution of the multisensory environment

Evaluating interactional synchrony in full-body interaction with autistic children

Interactional synchrony, the spontaneous coordination of movements during interaction, is increasingly considered important in research on the development of non-verbal communication by autistic children. There is evidence that interventions using embodied interaction technologies to support interactional synchrony are possible, but we do not have a shared framework in Human-Computer Interaction (HCI) for designing and evaluating such systems. We discuss existing measurement and evaluation tools used in experimental psychology and consider how the prevalent approach could be adapted to naturalistic HCI study contexts, with input from domain experts. We report on an exploratory case study evaluating a full-body interactive musical system with a group of ten autistic children. We provide methodological recommendations for the evaluation of future systems focusing on interactional synchrony, highlight limitations of current measurement tools and suggest mitigations

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments