Ars Informatica -- Ars Electronica: Improving Sonification Aesthetics

In this paper we discuss æsthetic issues of sonifications. We posit that many sonifications have suffered from poor acoustic ecology which makes listening more difficult, thereby resulting in poorer data extraction and inference on the part of the listener. Lessons are drawn from the electro acoustic music community as we argue that it is not instructive to distinguish between sonifications and music/sound art. Edgar Var`ese defined music as organised sound and sonifications organise sound to reflect some aspect of the thing being sonified. Therefore, we propose that sonification designers can improve the communicative ability of their auditory displays by paying attention to the æsthetic issues that are well known to composers, orchestrators, sound designers & artists, and recording engineers

Real-Time Electroencephalogram Sonification for Neurofeedback

Electroencephalography (EEG) is the measurement via the scalp of the electrical activity of the brain. The established therapeutic intervention of neurofeedback involves presenting people with their own EEG in real-time to enable them to modify their EEG for purposes of improving performance or health. The aim of this research is to develop and validate real-time sonifications of EEG for use in neurofeedback and methods for assessing such sonifications. Neurofeedback generally uses a visual display. Where auditory feedback is used, it is mostly limited to pre-recorded sounds triggered by the EEG activity crossing a threshold. However, EEG generates time-series data with meaningful detail at fine temporal resolution and with complex temporal dynamics. Human hearing has a much higher temporal resolution than human vision, and auditory displays do not require people to focus on a screen with their eyes open for extended periods of time – e.g. if they are engaged in some other task. Sonification of EEG could allow more rapid, contingent, salient and temporally detailed feedback. This could improve the efficiency of neurofeedback training and reduce the number and duration of sessions for successful neurofeedback. The same two deliberately simple sonification techniques were used in all three experiments of this research: Amplitude Modulation (AM) sonification, which maps the fluctuations in the power of the EEG to the volume of a pure tone; and Frequency Modulation (FM) sonification, which uses the changes in the EEG power to modify the frequency. Measures included, a listening task, NASA task load index; a measure of how much work it was to do the task, Pre & post measures of mood, and EEG. The first experiment used pre-recorded single channel EEG and participants were asked to listen to the sound of the sonified EEG and try and track the activity that they could hear by moving a slider on a computer screen using a computer mouse. This provided a quantitative assessment of how well people could perceive the sonified fluctuations in EEG level. The tracking accuracy scores were higher for the FM sonification but self-assessments of task load rated the AM sonification as easier to track. The second experiment used the same two sonifications, in a real neurofeedback task using participants own live EEG. Unbeknownst to the participants the neurofeedback task was designed to improve mood. A Pre-Post questionnaire showed that participants changed their self-rated mood in the intended direction with the EEG training, but there was no statistically significant change in EEG. Again the FM sonification showed a better performance but AM was rated as less effortful. The performance of sonifications in the tracking task in experiment 1 was found to predict their relative efficacy at blind self-rated mood modification in experiment 2. The third experiment used both the tracking as in experiment 1 and neurofeedback tasks as in experiment 2, but with modified versions of the AM and FM sonifications to allow two-channel EEG sonifications. This experiment introduced a physical slider as opposed to a mouse for the tracking task. Tracking accuracy increased, but this time no significant difference was found between the two sonification techniques on the tracking task. In the training task, once more the blind self-rated mood did improve in the intended direction with the EEG training, but as again there was no significant change in EEG, this cannot necessarily be attributed to the neurofeedback. There was only a slight difference between the two sonification techniques in the effort measure. In this way, a prototype method has been devised and validated for the quantitative assessment of real-time EEG sonifications. Conventional evaluations of neurofeedback techniques are expensive and time consuming. By contrast, this method potentially provides a rapid, objective and efficient method for evaluating the suitability of candidate sonifications for EEG neurofeedback

Rehabilitative devices for a top-down approach

In recent years, neurorehabilitation has moved from a "bottom-up" to a "top down" approach. This change has also involved the technological devices developed for motor and cognitive rehabilitation. It implies that during a task or during therapeutic exercises, new "top-down" approaches are being used to stimulate the brain in a more direct way to elicit plasticity-mediated motor re-learning. This is opposed to "Bottom up" approaches, which act at the physical level and attempt to bring about changes at the level of the central neural system. Areas covered: In the present unsystematic review, we present the most promising innovative technological devices that can effectively support rehabilitation based on a top-down approach, according to the most recent neuroscientific and neurocognitive findings. In particular, we explore if and how the use of new technological devices comprising serious exergames, virtual reality, robots, brain computer interfaces, rhythmic music and biofeedback devices might provide a top-down based approach. Expert commentary: Motor and cognitive systems are strongly harnessed in humans and thus cannot be separated in neurorehabilitation. Recently developed technologies in motor-cognitive rehabilitation might have a greater positive effect than conventional therapies

An introduction to interactive sonification

The research field of sonification, a subset of the topic of auditory display, has developed rapidly in recent decades. It brings together interests from the areas of data mining, exploratory data analysis, human–computer interfaces, and computer music. Sonification presents information by using sound (particularly nonspeech), so that the user of an auditory display obtains a deeper understanding of the data or processes under investigation by listening

Interpreting EEG by voice: Vocal EEG Sonification

Hermann T, Baier G. Interpreting EEG by voice: Vocal EEG Sonification. In: Rinnot M, ed. Proceedings of the 1st Sketching Sonic Interaction Design Workshop. Holon, Israel: COST IC0601 SID and HIT; 2009.Vocal EEG sonifications are presented as a method for complex time series sonification that is particularly tailored to address both humans' articulatory and auditory competences in order to improve the understanding and communication of the underlying data. In Vocal EEG sonification, the EEG data is represented in real-time by synthesized sound in a systematic, reproducible, task-centered way using an articulatory sound synthesizer capable of creating vowel transitions. Patterns such as 'EEG at rest', epileptic EEG, sleep EEG, etc. are thereby turned into characteristically different sonic gestalts that human listeners can discern from listening to the 'data babble'. In this paper, we emphasize the aspect of designing sonification particularly for the purpose of enhancing communication about sonic patterns, and we conduct a preliminary study about the human skill to use the own vocal tract to mimic or imitate patterns heard in the sonification. Our study will show to what degree humans are capable to recognize signal types correctly, both from the original sonifications and from vocal imitations performed by trained sonification users and naive users without extended previous experience in sonification

Prototyping a method for the assessment of real-time EEG sonifications

This paper presents a first step in the development of a methodology to compare the ability of different sonifications to convey the fine temporal detail of the Electroencephalography (EEG) brainwave signal in real time. In EEG neurofeedback a person‟s EEG activity is monitored and presented back to them, to help them to learn how to modify their brain activity. Learning theory suggests that the more rapidly and accurately the feedback follows behaviour the more efficient the learning will be. Therefore a critical issue is how to assess the ability of a sonification to convey rapid and temporally complex EEG data for neurofeedback. To allow for replication, this study used sonifications of pre-recorded EEG data and asked participants to try and track aspects of the signal in real time using a mouse. This study showed that, although imperfect, this approach is a practical way to compare the suitability of EEG sonifications for tracking detailed EEG signals in real time and that the combination of quantitative and qualitative data helped characterise the relative efficacy of different sonifications

Experiments of the sonification of the sleep electroencephalogram

It is becoming possible to perform sleep recordings at home with equipment targeted for the regular consumers. This alleviates the pressures to increase capacity in sleep clinics. The interpretation of the sleep recordings is not very easy for the laymen and alternative assisting methods should be sought for this. Sonification is a method by which a phenomenon is converted to a sound for human listeners. This paper describes experiments made for the sonification of the electric activity of the brain, the electroencephalography (EEG) for the purpose of recognizing the presence and absence of the necessary refreshing components of sleep, deep sleep and rapid eye movement (REM) sleep. The methods are based on the calculation of features of the EEG signal which are characteristic to the deep and REM sleep as well as wakefulness. The features are converted to amplitude modulation functions of artificial and musical instrument sounds by using mathematical transforms such as Principal Component Analysis and Linear Discriminant Analysis. The results indicate that modulated sinusoidal signals are not appropriate for the sonification of sleep EEG but that modulating the sound of musical instruments could be a viable option for making the recognition of good and bad sleep possible