Listening-Mode-Centered Sonification Design for Data Exploration

Grond F. Listening-Mode-Centered Sonification Design for Data Exploration. Bielefeld: Bielefeld University; 2013.From the Introduction to this thesis: Through the ever growing amount of data and the desire to make them accessible to the user through the sense of listening, sonification, the representation of data by using sound has been subject of active research in the computer sciences and the field of HCI for the last 20 years. During this time, the field of sonification has diversified into different application areas: today, sound in auditory display informs the user about states and actions on the desktop and in mobile devices; sonification has been applied in monitoring applications, where sound can range from being informative to alarming; sonification has been used to give sensory feedback in order to close the action and perception loop; last but not least, sonifications have also been developed for exploratory data analysis, where sound is used to represent data with unknown structures for hypothesis building. Coming from the computer sciences and HCI, the conceptualization of sonification has been mostly driven by application areas. On the other hand, the sonic arts who have always contributed to the community of auditory display have a genuine focus on sound. Despite this close interdisciplinary relation of communities of sound practitioners, a rich and sound- (or listening)-centered concept about sonification is still missing as a point of departure for a more application and task overarching approach towards design guidelines. Complementary to the useful organization along fields of applications, a conceptual framework that is proper to sound needs to abstract from applications and also to some degree from tasks, as both are not directly related to sound. I hence propose in this thesis to conceptualize sonifications along two poles where sound serves either a normative or a descriptive purpose. In the beginning of auditory display research, a continuum between a symbolic and an analogic pole has been proposed by Kramer (1994a, page 21). In this continuum, symbolic stands for sounds that coincide with existing schemas and are more denotative, analogic stands for sounds that are informative through their connotative aspects. (compare Worrall (2009, page 315)). The notions of symbolic and analogic illustrate the struggle to find apt descriptions of how the intention of the listener subjects audible phenomena to a process of meaning making and interpretation. Complementing the analogic-symbolic continuum with descriptive and normative purposes of displays is proposed in the light of the recently increased research interest in listening modes and intentions. Similar to the terms symbolic and analogic, listening modes have been discussed in auditory display since the beginning usually in dichotomic terms which were either identified with the words listening and hearing or understood as musical listening and everyday listening as proposed by Gaver (1993a). More than 25 years earlier, four direct listening modes have been introduced by Schaeffer (1966) together with a 5th synthetic mode of reduced listening which leads to the well-known sound object. Interestingly, Schaeffer’s listening modes remained largely unnoticed by the auditory display community. Particularly the notion of reduced listening goes beyond the connotative and denotative poles of the continuum proposed by Kramer and justifies the new terms descriptive and normative. Recently, a new taxonomy of listening modes has been proposed by Tuuri and Eerola (2012) that is motivated through an embodied cognition approach. The main contribution of their taxonomy is that it convincingly diversifies the connotative and denotative aspects of listening modes. In the recently published sonification handbook, multimodal and interactive aspects in combination with sonification have been discussed as promising options to expand and advance the field by Hunt and Hermann (2011), who point out that there is a big need for a better theoretical foundation in order to systematically integrate these aspects. The main contribution of this thesis is to address this need by providing alternative and complementary design guidelines with respect to existing approaches, all of which have been conceived before the recently increased research interest in listening modes. None of the existing contributions to design frameworks integrates multimodality, and listening modes with a focus on exploratory data analysis, where sonification is conceived to support the understanding of complex data potentially helping to identify new structures therein. In order to structure this field the following questions are addressed in this thesis: • How do natural listening modes and reduced listening relate to the proposed normative and descriptive display purposes? • What is the relationship of multimodality and interaction with listening modes and display purposes? • How can the potential of embodied cognition based listening modes be put to use for exploratory data sonification? • How can listening modes and display purposes be connected to questions of aesthetics in the display? • How do data complexity and Parameter-mapping sonification relate to exploratory data analysis and listening modes

Safe and Sound: Proceedings of the 27th Annual International Conference on Auditory Display

Complete proceedings of the 27th International Conference on Auditory Display (ICAD2022), June 24-27. Online virtual conference

Data Sonification in Creative Practice

Sonification is the process of data transmission with non-speech audio. While finding increasing acceptance as a scientific method, particularly where a visual representation of data is inadequate, it is still often derided as a ‘gimmick’. Composers have also shown growing interest in sonification as a compositional method. Both in science and in music, the criticism towards this method relates to poor aesthetics and gratuitous applications. This thesis aims to address these issues through an accompanying portfolio of pieces which use sonification as a compositional tool. It establishes the principles of ‘musification’, which can be defined as a sonification which uses musical structures; a sonification organised by musical principles. The practice-as-research portfolio explores a number of data sources, musical genres and science-music collaborations. The main contributions to knowledge derived from the project are a portfolio of compositions, a compositional framework for sonification and an evaluation framework for musification. This thesis demonstrates the validity of practice-as-research as a methodology in sonification research

Auditory Display Design : An Investigation of a Design Pattern Approach

PhDThis thesis investigates the design of audio for feedback in human-technology interaction— auditory displays. Despite promising progress in research and the potential benefits, we currently see little impact of audio in everyday interfaces. Changing interaction paradigms, new contexts of use and inclusive design principles, however, increase the need for an efficient, non-visual means of conveying information. Motivated by these needs, this work describes the development and evaluation of a methodological design framework, aiming to enhance knowledge and skill transfer in auditory display design and to enable designers to build more efficient and compelling auditory solutions. The work starts by investigating the current practice in designing audio in the user interface. A survey amongst practitioners and researchers in the field and a literature study of research papers highlighted the need for a structured design approach. Building on these results, paco – pattern design in the context space has been developed, a framework providing methods to capture, apply and refine design knowledge through design patterns. A key element of paco, the context space, serves as the organising principle for patterns, artefacts and design problems and supports designers in conceptualising the design space. The evaluation of paco is the first comparative study of a design methodology in this area. Experts in auditory display design and novice designers participated in a series of experiments to determine the usefulness of the framework. The evaluation demonstrated that paco facilitates the transfer of design knowledge and skill between experts and novices as well as promoting reflection and recording of design rationale. Alongside these principle achievements, important insights have been gained about the design process which lay the foundations for future research into this subject area. This work contributes to the field of auditory display as it reflects on the current practice and proposes a means of supporting designers to communicate, reason about and build on each other’s work more efficiently. The broader field of human-computer interaction may also benefit from the availability of design guidance for exploiting the auditory modality to answer the challenges of future interaction design. Finally, with paco a generic methodology in the field of design patterns was proposed, potentially similarly beneficial to other designing disciplines

Model-based sonification revisited–-authors' comments on Hermann and Ritter, ICAD 2002

Hermann T, Ritter H. Model-based sonification revisited–-authors' comments on Hermann and Ritter, ICAD 2002. ACM Trans. Applied Perception. 2005;2(4):559-563.We discuss the framework of Model-Based Sonification (MBS) and its contribution to a principled design of mediators between high-dimensional data spaces and perceptual spaces, particularly sound spaces. Data Crystallization Sonification, discussed in the reprinted paper, exemplifies the design of sonification models according to this framework. Finally, promising lines of development in this area are pointed out, concerning generalizations, applications, and open research directions

Overviews and their effect on interaction in the auditory interface.

PhDAuditory overviews have the potential to improve the quality of auditory interfaces. However, in order to apply overviews well, we must understand them. Specifically, what are they and what is their impact? This thesis presents six characteristics that overviews should have. They should be a structured representation of the detailed information, define the scope of the material, guide the user, show context and patterns in the data, encourage exploration of the detail and represent the current state of the data. These characteristics are guided by a systematic review of visual overview research, analysis of established visual overviews and evaluation of how these characteristics fit current auditory overviews. The second half of the thesis evaluates how the addition of an overview impacts user interaction. While the overviews do not improve performance, they do change the navigation patterns from one of data exploration and discovery to guided and directed information seeking. With these two contributions, we gain a better understanding of how overviews work in an auditory interface and how they might be exploited more effectively

Developing an interactive overview for non-visual exploration of tabular numerical information

This thesis investigates the problem of obtaining overview information from complex tabular numerical data sets non-visually. Blind and visually impaired people need to access and analyse numerical data, both in education and in professional occupations. Obtaining an overview is a necessary first step in data analysis, for which current non-visual data accessibility methods offer little support. This thesis describes a new interactive parametric sonification technique called High-Density Sonification (HDS), which facilitates the process of extracting overview information from the data easily and efficiently by rendering multiple data points as single auditory events. Beyond obtaining an overview of the data, experimental studies showed that the capabilities of human auditory perception and cognition to extract meaning from HDS representations could be used to reliably estimate relative arithmetic mean values within large tabular data sets. Following a user-centred design methodology, HDS was implemented as the primary form of overview information display in a multimodal interface called TableVis. This interface supports the active process of interactive data exploration non-visually, making use of proprioception to maintain contextual information during exploration (non-visual focus+context), vibrotactile data annotations (EMA-Tactons) that can be used as external memory aids to prevent high mental workload levels, and speech synthesis to access detailed information on demand. A series of empirical studies was conducted to quantify the performance attained in the exploration of tabular data sets for overview information using TableVis. This was done by comparing HDS with the main current non-visual accessibility technique (speech synthesis), and by quantifying the effect of different sizes of data sets on user performance, which showed that HDS resulted in better performance than speech, and that this performance was not heavily dependent on the size of the data set. In addition, levels of subjective workload during exploration tasks using TableVis were investigated, resulting in the proposal of EMA-Tactons, vibrotactile annotations that the user can add to the data in order to prevent working memory saturation in the most demanding data exploration scenarios. An experimental evaluation found that EMA-Tactons significantly reduced mental workload in data exploration tasks. Thus, the work described in this thesis provides a basis for the interactive non-visual exploration of a broad range of sizes of numerical data tables by offering techniques to extract overview information quickly, performing perceptual estimations of data descriptors (relative arithmetic mean) and managing demands on mental workload through vibrotactile data annotations, while seamlessly linking with explorations at different levels of detail and preserving spatial data representation metaphors to support collaboration with sighted users

Sonification of exosolar planetary systems

The purpose of this research is to investigate sonification techniques suitable for astronomers to explore exosolar planetary data. Four studies were conducted, one with sonification specialists and three with exosolar planetary astronomers. The first study was to establish existing practices in sonification design and obtain detailed information about design processes not fully communicated in published papers. The other studies were about designing and evaluating sonifications for three different fields of exosolar astronomy. One, to sonify atmospheric data of an exoplanet in a habitable zone. Another, to sonify accretion discs located in newly developing exosolar systems. The third sonification, planet detection in an asteroid belt. User-centred design was used so that mappings of the datasets could be easily comprehensible. Each sonification was designed to sound like the natural elements that were represented in the data. Spatial separation between overlapping datasets can make hidden information more noticeable and provide additional dimensionality for sound objects. It may also give a more realistic interpretation of the data object in a real-world capacity. Multiple psychoacoustic mappings can convey data dimensionality and immediate recognition of subtle changes. Sound design aesthetics that mimic natural sounds were more relatable for the user. Sonification has been effective within the context of these studies offering new insight by unmasking previously unnoticed data particulars. It has also given the astronomers a broader understanding of the dimension of the data objects that they study and their temporal-spatial behaviours. Future work pertains to the further development and creation of a sonification model consisting of different aspects of exosolar astronomy that could be developed for a platform that houses different data related to this field of study

Concurrency in auditory displays for connected television

Many television experiences depend on users being both willing and able to visually attend to screen-based information. Auditory displays offer an alternative method for presenting this information and could benefit all users. This thesis explores how this may be achieved through the design and evaluation of auditory displays involving varying degrees of concurrency for two television use cases: menu navigation and presenting related content alongside a television show. The first study, on the navigation of auditory menus, looked at onset asynchrony and word length in the presentation of spoken menus. The effects of these on task duration, accuracy and workload were considered. Onset asynchrony and word length both caused significant effects on task duration and accuracy, while workload was only affected by onset asynchrony. An optimum asynchrony was identified, which was the same for both long and short words, but better performance was obtained with the shorter words that no longer overlapped. The second experiment investigated how disruption, workload, and preference are affected when presenting additional content accompanying a television programme. The content took the form of sound from different spatial locations or as text on a smartphone and the programme's soundtrack was either modified or left unaltered. Leaving the soundtrack unaltered or muting it negatively impacted user experience. Removing the speech from the television programme and presenting the secondary content as sound from a smartphone was the best auditory approach. This was found to compare well with the textual presentation, resulting in less visual disruption and imposing a similar workload. Additionally, the thesis reviews the state-of-the-art in television experiences and auditory displays. The human auditory system is introduced and important factors in the concurrent presentation of speech are highlighted. Conclusions about the utility of concurrency within auditory displays for television are made and areas for further work are identified