Sonification of surface tapping changes behavior, surface perception, and emotion

The audio feedback resulting from object interaction provides information about the material of the surface and about one's own motor behavior. With the current developments in interactive sonification, it's now possible to digitally change this audio feedback, making the use of interactive sonification a compelling approach to shape tactile surface interactions. Here, the authors present a prototype for a sonic interactive surface, capable of delivering surface tapping sounds in real time when triggered by users' taps on a real surface or on an imagined 'virtual' surface. In this system, the delivered audio feedback can be varied so that the tapping sounds correspond to different applied strengths during tapping. The authors also propose a multidimensional measurement approach to evaluate user experiences of multimodal interactive systems. They evaluated their system by looking at the effect of the altered tapping sounds on emotional action-related responses, the users' interactions with the surface, and perceived surface hardness. Results show the influence of the sonification of tapping at all levels: emotional, behavioral, and perceptual. These results have implications on the design of interactive sonification displays and tangible auditory interfaces aiming to change perceived and subsequent motor behavior as well as perceived material properties

Sonification of probabilistic feedback through granular synthesis

We describe a method to improve user feedback, specifically the display of time-varying probabilistic information, through asynchronous granular synthesis. We have applied these techniques to challenging control problems as well as to the sonification of online probabilistic gesture recognition. We're using these displays in mobile, gestural interfaces where visual display is often impractical

Multi-touch interactions for model-based sonification

Presented at the 15th International Conference on Auditory Display (ICAD2009), Copenhagen, Denmark, May 18-22, 2009This paper presents novel interaction modes for Model-Based Soni- fication (MBS) via a multi-touch interface. We first lay out details about the constructed multi-touch surface. This is followed by a description of the Data Sonogram Sonification Model and how it is implemented using the system. Modifications from the original sonification model such as the limited space scans are described and discussed with sonification examples. Videos showing exam- ples of interaction are provided for various data sets. Beyond Data Sonograms, the presented system provides a basis for the imple- mentation of known and novel sonification models. We discuss the available interaction modes with multi-touch surfaces and how these interactions can be profitably used to control spatial and non- spatial sonification models

Tac-tiles: multimodal pie charts for visually impaired users

Tac-tiles is an accessible interface that allows visually impaired users to browse graphical information using tactile and audio feedback. The system uses a graphics tablet which is augmented with a tangible overlay tile to guide user exploration. Dynamic feedback is provided by a tactile pin-array at the fingertips, and through speech/non-speech audio cues. In designing the system, we seek to preserve the affordances and metaphors of traditional, low-tech teaching media for the blind, and combine this with the benefits of a digital representation. Traditional tangible media allow rapid, non-sequential access to data, promote easy and unambiguous access to resources such as axes and gridlines, allow the use of external memory, and preserve visual conventions, thus promoting collaboration with sighted colleagues. A prototype system was evaluated with visually impaired users, and recommendations for multimodal design were derived

Advances in Human-Protein Interaction - Interactive and Immersive Molecular Simulations

International audienc

A Study of Material Sonification in Touchscreen Devices

Even in the digital age, designers largely rely on physical material samples to illustrate their products, as existing visual representations fail to sufficiently reproduce the look and feel of real world materials. Here, we investigate the use of interactive material sonification as an additional sensory modality for communicating well-established material qualities like softness, pleasantness or value. We developed a custom application for touchscreen devices that receives tactile input and translate it into material rubbing sound using granular synthesis. We used this system to perform a psychophysical study, in which the ability of the user to rate subjective material qualities is evaluated, with the actual material samples serving as reference stimulus. Our experimental results indicate that the considered audio cues do not significantly contribute to the perception of material qualities but are able to increase the level of immersion when interacting with digital samples.Comment: 9 page

Ritualistic approach to sonic interaction design: A poetic framework for participatory sonification

Presented at the 27th International Conference on Auditory Display (ICAD 2022) 24-27 June 2022, Virtual conference.While sonification is often adopted as an analytical tool to understand data, it can also be an efficient basis for the construction of an interaction model for an aesthetic sound piece. Mirroring the performative arts, a ritualistic approach in participatory sonification can take place whenever a work relies more on the outer form of the piece, than on the meaning attributed to the information communicated, to connect with an audience, losing some degrees of readability and intelligibility in the process while maintaining a reliable data-to-sound relationship. We will present a few examples that anticipate or expand the use of sonification as analytical tool to propose aesthetic approaches, accessing more complex layers of meaning in interactive design. By proposing topological, semantic, and technical perspectives, we demonstrate the functional aspects of the multimedia artwork “The Only Object They Could Retrieve From Earth’s Lost Civilisation” (“The Only Object” from now on). Outcomes will be considered under the multidisciplinary framework here proposed, to conclude with possibilities and implications of a ritualistic approach to interaction design

Taxonomy and Definitions for Sonification and Auditory Display

Hermann T. Taxonomy and Definitions for Sonification and Auditory Display. In: Susini P, Warusfel O, eds. Proceedings of the 14th International Conference on Auditory Display (ICAD 2008). Paris, France: IRCAM; 2008. Sonification is still a young research field and many terms such as sonification, auditory display, auralization, audification have been used without a precise definition. Recent developments such as the introduction of Model-based Sonification, the establishing of interactive sonification and the increased interest in sonification from arts have raised the issue of revisiting the definitions towards a clearer terminology. This paper introduces a new definition for sonification and auditory display that emphasize necessary and sufficient conditions for organized sound to be called sonification. It furthermore suggests a taxonomy, and discusses the relation between visualization and sonification. A hierarchy of closed-loop interactions is furthermore introduced. This paper aims at initiating vivid discussions towards the establishing of a deeper theory of sonification and auditory display

Complexity as Process: Complexity Inspired Approaches to Composition

This article examines the use of Complexity Theory as an inspiration for the creation of new musical works, and highlights problems and possible solutions associated with its application as a compositional tool. In particular it explores how the philosophy behind Complexity Theory affects notions of process-based composition, indeterminacy in music and the performer/listener/environment relationship, culminating in providing a basis for the understanding of music creation as an active process within a context. The author presents one of his own sound installations, Cross-Pollination, as an example of a composition inspired and best understood from the philosophical position as described in Complexity Theory