Tangible auditory interfaces : combining auditory displays and tangible interfaces

Bovermann T. Tangible auditory interfaces : combining auditory displays and tangible interfaces. Bielefeld (Germany): Bielefeld University; 2009.Tangible Auditory Interfaces (TAIs) investigates into the capabilities of the interconnection of Tangible User Interfaces and Auditory Displays. TAIs utilise artificial physical objects as well as soundscapes to represent digital information. The interconnection of the two fields establishes a tight coupling between information and operation that is based on the human's familiarity with the incorporated interrelations. This work gives a formal introduction to TAIs and shows their key features at hand of seven proof of concept applications

Ways of Guided Listening: Embodied approaches to the design of interactive sonifications

This thesis presents three use cases for interactive feedback. In each case users interact with a system and receive feedback: the primary source of feedback is visual, while a second source of feedback is offered as sonification. The first use case comprised an interactive sonification system for use by pathologists in the triage stage of cancer diagnostics. Image features derived from computational homology are mapped to a soundscape with integrated auditory glance indicating potential regions of interests. The resulting prototype did not meet the requirements of a domain expert. In the second case this thesis presents an interactive sonification plug-in developed for a software package for interactive visualisation of macromolecular complexes. A framework for building different sonification methods in Python and an OSC-controlled sound producing software was established along with sonification methods and a general sonification plugin. It received generally positive feedback, but the mapping was deemed not very transparent. From these cases and ideas in sonification design literature, the Subject-Position-Based Sonification Design Framework (SPBDF) was developed. It explores an alternative conception of design: that working from a frame of reference encompassing a non-expert audience will lead towards sonifications that are more easily understood. A method for the analysis of sonifications according to its criteria is outlined and put into practice to evaluate a range of sonifications. This framework was evaluated in the third use case, a system for sonified feedback for an exercise device designed for back pain rehabilitation. Two different sonifications, one using SPBDF as basis of their design, were evaluated, indicating that interactive sonification can provide valuable feedback and improve task performance (decrease the mean speed) when the soundscape employed invokes an appropriate emotional response in the user

A technology-aided multi-modal training approach to assist abdominal palpation training and its assessment in medical education

Kinaesthetic Learning Activities (KLA) are techniques for enhancing the motor learning process to provide a deep understanding of fundamental skills in particular disciplines. With KLA learning takes place by carrying out a physical activity to transform empirical achievements into representative cognitive understanding. In disciplines such as medical education, frequent hands-on practice of certain motor skills plays a key role in the development of medical students' competency. Therefore it is essential that clinicians master these core skills early on in their educational journey as well as retain them for the entirety of their career. Transferring knowledge of performing dexterous motor skills, such as clinical examinations, from experts to novices demands a systematic approach to quantify relevant motor variables with the help of medical experts in order to form a reference best practice model for target skills. Additional information (augmented feedback) on certain aspects of movements could be extracted from this model and visualised via multi-modal sensory channels in order to enhance motor performance and learning processes. This thesis proposes a novel KLA methodology to significantly improve the quality of palpation training in medical students. In particular, it investigates whether it is possible to enhance the existing abdominal palpation skills acquisition process (motor performance and learning) with provision of instructional concurrent and terminal augmented feedback on applied forces by the learner's hand via an autonomous multimodal displays. This is achieved by considering the following: identifying key motor variables with help of medical experts; forming a gold standard model for target skills by collecting pre-defined motor variables with an innovative quantification technique; designing an assessment criteria by analysing the medical experts' data; and systematically evaluating the impact of instructional augmented feedback on medical students' motor performance with two distinct assessment approaches(a machine-based and a human-based). In addition, an evaluation of performance on a simpler task is carried out using a game-based training method, to compare feedback visualisation techniques, such as concurrent visual and auditory feedback as used in a serious games environment, with abstract visualisation of motor variables. A detailed between-participants study is presented to evaluate the effect of concurrent augmented feedback on participants' skills acquisition in the motor learning process. Significant improvement on medical students' motor performance was observed when augmented feedback on applied forces were visually presented (H(2) = 6:033, p < :05). Moreover, a positive correlation was reported between computer-generated scores and human-generated scores, r = :62, p (one-tailed) < :05. This indicates the potential of the computer-based assessment technique to assist the current assessment process in medical education. The same results were also achieved in a blind-folded (no-feedback) transfer test to evaluate performance and short-term retention of skills in the game-based training approach. The accuracy in the exerted target force for participants in the game-playing group, who were trained using the game approach (Mdn = 0:86), differed significantly from the participants in control group, who trained using the abstract visualisation of the exerted force value (Mdn = 1:56), U = 61, z = -2:137, p < :05, r = -0:36. Finally, the usability of both motor learning approaches were surveyed via feedback questionnaires and positive responses were achieved from users. The research presented shows that concurrent augmented feedback significantly improves the participants' motor control abilities. Furthermore, advanced visualisation techniques such as multi-modal displays increases the participants' motivation to engage in learning and to retain motor skills

Sound modeling issues in interactive sonification - From basic contact events to synthesis and manipulation tools

Questa tesi affronta una varietà di temi di ricerca, spaziando dalla interazione uomo-macchina alla modellizzazione fisica. Ciò che unisce queste ampie aree di interesse è l'idea di utilizzare simulazioni numeriche di fenomeni acustici basate sulla fisica, al fine di implementare interfacce uomo-macchina che offrano feedback sonoro coerente con l'interazione dell'utente. A questo proposito, negli ultimi anni sono nate numerose nuove discipline che vanno sotto il nome di -- per citarne alcune -- auditory display, sonificazione e sonic interaction design. In questa tesi vengono trattate la progettazione e la realizzazione di algoritmi audio efficienti per la sonificazione interattiva. A tale scopo si fa uso di tecniche di modellazione fisica di suoni ecologici (everyday sounds), ovvero suoni che non rientrano nelle famiglie del parlato e dei suoni musicali.The work presented in this thesis ranges over a variety of research topics, spacing from human-computer interaction to physical-modeling. What combines such broad areas of interest is the idea of using physically-based computer simulations of acoustic phenomena in order to provide human-computer interfaces with sound feedback which is consistent with the user interaction. In this regard, recent years have seen the emergence of several new disciplines that go under the name of -- to cite a few -- auditory display, sonification and sonic interaction design. This thesis deals with the design and implementation of efficient sound algorithms for interactive sonification. To this end, the physical modeling of everyday sounds is taken into account, that is sounds not belonging to the families of speech and musical sounds

Embodied geosensification-models, taxonomies and applications for engaging the body in immersive analytics of geospatial data

This thesis examines how we can use immersive multisensory displays and body-focused interaction technologies to analyze geospatial data. It merges relevant aspects from an array of interdisciplinary research areas, from cartography to the cognitive sciences, to form three taxonomies that describe the senses, data representations, and interactions made possible by these technologies. These taxonomies are then integrated into an overarching design model for such "Embodied Geosensifications". This model provides guidance for system specification and is validated with practical examples

Musical expectancy within movement sonification to overcome low self-efficacy

While engaging in physical activity is important for a healthy lifestyle, low self-efficacy, i.e. one's belief in one's own ability, can prevent engagement. Sound has been used in a variety of ways for physical activity: movement sonification to inform about movement, music to encourage and direct movement, and auditory illusions to adapt people's bodily representation and movement behaviour. However, no approach provides the whole picture when considering low self-efficacy. For example, sonification does not encourage movement past a person's expectation of their ability, music gives no information of one's capabilities, and auditory illusions do not direct changes in movement behaviour in a directed way. This thesis proposes a combined method that leverages the agency felt over sonification, our embodiment of music and movement altering feedback to design \textit{``musical expectancy sonifications''} which incorporate musical expectancy within sonification to alter movement perception and behaviour. This thesis proposes a Movement Sonification Expectation Model (MoSEM), which explores expectation within a movement sonification impact on people's perception of their abilities and the way they move. This MoSEM is then interrogated and developed in four initial control studies that investigate these sonifications for different types of movement as well as how they interact with one's expectation of a given movement. These findings led to an exploration of how the MoSEM can be applied to design sonification to support low-self efficacy in two case study populations: chronic pain rehabilitation, including one control study and one mixed methods study, and general well-being, including one interview study and two control studies. These studies show the impact of musical expectation on people's movement perception and behaviour. The findings from this thesis demonstrate not only how sonifications can be designed to use musical expectancy, but also shows a number of considerations that are needed when designing movement sonifications