The Sonic Representation of Mathematical Data

Conveying data and information using non-speech audio is an ever growing field of research. Existing work has been performed investigating sonfication and its applications, and this research seeks to build upon these ideas while also suggesting new areas of potential. In this research, initial work focused on the sonification of DNA and RNA nucleotide base sequences for analysis. A case study was undertaken into the potential of rhythmic parsing of such data sequences, with test results indicating that a more effective method of representing data in a sonification was required. Sonification of complex data such as DNA and RNA was found to require more verbose methods than pitch to parameter mappings, and so investigation was made into musical pattern sonifcation. Existing low level pattern design methods were next evaluated in an experiment concerning the use of musical patterns to represent data. This experiment suggested that while a musical pattern may be made distinct, it does not necessarily follow that it is memorable. This experiment also suggested that concurrent pattern representation was difficult to process, and so improved methods were required. Improvements to pattern design were made with the introduction of contour icons, which allow detectable and memorable musical patterns to be designed using simple shapes. Testing showed contour icons to be significantly more effective than low level patterns in a sonfication, and as such form the basis of the novel contributions of this thesis. Improvements in concurrent representation were considered by the use of harmonic combination, a method of defining intersections in a data set as harmonies of a single common musical pattern. Significant improvements were observed over non-harmonic concurrent representation, although limitations were observed due to constraints in the number of combinations available using a specific value. Harmonic combination has potential for further development, and is a novel contribution of this thesis. The organisation and grouping of data in a sonfication using rhythmic parsing was also investigated. Rhythmic parsing uses rest notes within a musical framework to define sub-groupings in a data sonfication. Tests showed rhythmic parsing significantly improved the comparison of values and intersections between groups in a data sonification, and is another novel contribution of this thesis

Information Delivery on Mobile Devices Using Contour Icon Sonification

This paper examines the use of musical patterns to convey information, specifically in the context of mobile devices. Existing mechanisms (such as the popularity of the Morse code SMS alert) suggest that the use of musical patterns on mobile devices can be a very efficient and powerful method of data delivery. Unique musical patterns based on templates known as Contour Icons are used to represent specific data variables, with the output rendering of these patterns being referred to as a Sonification of that data. Contour Icon patterns mimic basic shapes and structures, thus providing listeners with a means of categorising them in a high level manner. Potential Sonification applications involving mobile devices are already in testing, with the aim of delivering data to mobile users in a fast, efficient and hands-free manner. It is the goal of this research to provide greater functionality on mobile devices using Sonification

Human Pattern Recognition in Data Sonification

Computational music analysis investigates the relevant features required for the detection and classification of musical content, features which do not always directly overlap with musical composition concepts. Human perception of music is also an active area of research, with existing work considering the role of perceptual schema in musical pattern recognition. Data sonification investigates the use of non-speech audio to convey information, and it is in this context that some potential guidelines for human pattern recognition are presented for discussion in this paper. Previous research into the role of musical contour (shape) in data sonification shows that it has a significant impact on pattern recognition performance, whilst investigation in the area of rhythmic parsing made a significant difference in performance when used to build structures in data sonifications. The paper presents these previous experimental results as the basis for a discussion around the potential for inclusion of schema- based classifiers in computational music analysis, considering where shape and rhythm classification may be employed at both the segmental and supra-segmental levels to better mimic the human process of perception

Information Delivery on Mobile Devices Using Contour Icon Sonification

This paper examines the use of musical patterns to convey information, specifically in the context of mobile devices. Existing mechanisms (such as the popularity of the Morse code SMS alert) suggest that the use of musical patterns on mobile devices can be a very efficient and powerful method of data delivery. Unique musical patterns based on templates known as Contour Icons are used to represent specific data variables, with the output rendering of these patterns being referred to as a Sonification of that data. Contour Icon patterns mimic basic shapes and structures, thus providing listeners with a means of categorising them in a high level manner. Potential Sonification applications involving mobile devices are already in testing, with the aim of delivering data to mobile users in a fast, efficient and hands-free manner. It is the goal of this research to provide greater functionality on mobile devices using Sonification

The Potential Role of Listening Modes in Auditory Interfaces for Location-based Services

The auditory modality offers several advantages as a means of communication for the purposes of location-based services (LBS), including fast response time [1], low processing and storage overheads [2], and hands/eyes-free mobility. However, with more and more sound-producing technology being used in day-to-day life, the battle for our acoustic attention has led to a steady rise in acoustic noise levels [3]. In an already noisy environment, it is tempting for the sound designer to simply use more volume as a means of gaining the listener’s attention but this creates a vicious circle of noise whereby every sound designer is merely struggling to be heard over the noise of every other sound designer. The field of soundscape theory however, may offer some potential solutions in this regard. Soundscape theory, as described by Schafer [4, 5] and Truax [6], considers sound from a more holistic point of view and the concept of listening modes considers the different levels of attention we pay to auditory stimuli depending on context and location within the soundscape. While several different theoretical listening modes have been proposed across the various acoustic disciplines, there is still a need for empirical data to support the existence of these modes. One area in which there is a certain amount of empirical data is in relation to spectral bandwidth and what Krause has called his ‘niche theory’ [7-9]. Niche theory describes the way in which different species appear to occupy discrete frequency bandwidths within the soundscapes of natural habitats; it is argued that this natural balance keeps redundant noise to a minimum and enables more efficient acoustic communication. If the principles observed in niche theory were to be observed in human listening behavior then a new approach to sound design might be possible whereby auditory stimuli exploit specific frequency bandwidths in order to maximize information exchange without necessarily raising noise levels. In this paper we outline a proposed experiment whereby listeners are asked to engage in a foreground task that encourages competitive conversation while also attending to a background listening task whereby participants have to acknowledge background non-speech sound bursts of varying spectral bandwidth presented at random intervals. Our aim is to compare the spectrograms of foreground conversations and background stimuli to see if relative spectral bandwidth has any discernible effect on stimulus identification success rate and response time

Emotional Speech Corpus Creation, Structure, Distribution and Re-Use

Abstract This paper details the on-going creation of a natural emotional speech corpus, its structure, distribution, and re-use. Using Mood Induction Procedures (MIPs), high quality emotional speech assets are obtained, analysed, tagged (for acoustic features), annotated and uploaded to an online speech corpus. This method structures the corpus in a logical and coherent manner, allowing it to be utilized for more than one purpose, ensuring distribution via a URL and ease of access through a web browser

Musical Pattern Design Using Contour Icons

This paper considers the use of Contour Icons in the design and implementation of musical patterns, for the purposes of detection and recognition. Research work had endeavoured to deliver musical patterns that were both distinct and memorable, and to this end a set of basic melodic shapes were introduced using a Sonification application called TrioSon that had been designed for the purpose. Existing work in the field (such as that concerning Earcon design) has considered the mechanisms by which patterns may be made distinctive, but it is argued that separate consideration must be given to the method of making such patterns memorable. This work suggests that while segregation and detection can best be facilitated by the individuality of a patterns rhythm, the retention (and hence future recognition) of a musical pattern is concerned more with its melodic range and contour. The detection and comprehension of musical patterns based around basic shapes (known as Contour Icons) was tested, within Sonifications of simple data sets generated using TrioSon. A set of test patterns based on such factors as tonality and key was used for control purposes, with Contour Icons being introduced in the second set of tests. Results suggest that significant improvement was made due to the use of Contour Icons, with further work focusing on the many possibilities that such a design framework would suggest

A VOWEL-STRESS EMOTIONAL SPEECH ANALYSIS METHOD

The analysis of speech, particularly for emotional content, is an open area of current research. This paper documents the development of a vowel-stress analysis framework for emotional speech, which is intended to provide suitable assessment of the assets obtained in terms of their prosodic attributes. The consideration of different levels of vowel-stress provides means by which the salient points of a signal may be analysed in terms of their overall priority to the listener. The prosodic attributes of these events can thus be assessed in terms of their overall significance, in an effort to provide a means of categorising the acoustic correlates of emotional speech. The use of vowel-stress is performed in conjunction with the definition of pitch and intensity contours, alongside other micro-prosodic information relating to voice quality

LinguaTag: an Emotional Speech Analysis Application

The analysis of speech, particularly for emotional content, is an open area of current research. Ongoing work has developed an emotional speech corpus for analysis, and defined a vowel stress method by which this analysis may be performed. This paper documents the development of LinguaTag, an open source speech analysis software application which implements this vowel stress emotional speech analysis method developed as part of research into the acoustic and linguistic correlates of emotional speech. The analysis output is contained within a file format combining SMIL and SSML markup tags, to facilitate search and retrieval methods within an emotional speech corpus database. In this manner, analysis performed using LinguaTag aims to combine acoustic, emotional and linguistic descriptors in a single metadata framework