Haptics for the development of fundamental rhythm skills, including multi-limb coordination

This chapter considers the use of haptics for learning fundamental rhythm skills, including skills that depend on multi-limb coordination. Different sensory modalities have different strengths and weaknesses for the development of skills related to rhythm. For example, vision has low temporal resolution and performs poorly for tracking rhythms in real-time, whereas hearing is highly accurate. However, in the case of multi-limbed rhythms, neither hearing nor sight are particularly well suited to communicating exactly which limb does what and when, or how the limbs coordinate. By contrast, haptics can work especially well in this area, by applying haptic signals independently to each limb. We review relevant theories, including embodied interaction and biological entrainment. We present a range of applications of the Haptic Bracelets, which are computer-controlled wireless vibrotactile devices, one attached to each wrist and ankle. Haptic pulses are used to guide users in playing rhythmic patterns that require multi-limb coordination. One immediate aim of the system is to support the development of practical rhythm skills and multi-limb coordination. A longer-term goal is to aid the development of a wider range of fundamental rhythm skills including recognising, identifying, memorising, retaining, analysing, reproducing, coordinating, modifying and creating rhythms – particularly multi-stream (i.e. polyphonic) rhythmic sequences. Empirical results are presented. We reflect on related work, and discuss design issues for using haptics to support rhythm skills. Skills of this kind are essential not just to drummers and percussionists but also to keyboards players, and more generally to all musicians who need a firm grasp of rhythm

Crossmodal audio and tactile interaction with mobile touchscreens

Touchscreen mobile devices often use cut-down versions of desktop user interfaces placing high demands on the visual sense that may prove awkward in mobile settings. The research in this thesis addresses the problems encountered by situationally impaired mobile users by using crossmodal interaction to exploit the abundant similarities between the audio and tactile modalities. By making information available to both senses, users can receive the information in the most suitable way, without having to abandon their primary task to look at the device. This thesis begins with a literature review of related work followed by a definition of crossmodal icons. Two icons may be considered to be crossmodal if and only if they provide a common representation of data, which is accessible interchangeably via different modalities. Two experiments investigated possible parameters for use in crossmodal icons with results showing that rhythm, texture and spatial location are effective. A third experiment focused on learning multi-dimensional crossmodal icons and the extent to which this learning transfers between modalities. The results showed identification rates of 92% for three-dimensional audio crossmodal icons when trained in the tactile equivalents, and identification rates of 89% for tactile crossmodal icons when trained in the audio equivalent. Crossmodal icons were then incorporated into a mobile touchscreen QWERTY keyboard. Experiments showed that keyboards with audio or tactile feedback produce fewer errors and greater speeds of text entry compared to standard touchscreen keyboards. The next study examined how environmental variables affect user performance with the same keyboard. The data showed that each modality performs differently with varying levels of background noise or vibration and the exact levels at which these performance decreases occur were established. The final study involved a longitudinal evaluation of a touchscreen application, CrossTrainer, focusing on longitudinal effects on performance with audio and tactile feedback, the impact of context on performance and personal modality preference. The results show that crossmodal audio and tactile icons are a valid method of presenting information to situationally impaired mobile touchscreen users with recognitions rates of 100% over time. This thesis concludes with a set of guidelines on the design and application of crossmodal audio and tactile feedback to enable application and interface designers to employ such feedback in all systems

"Tumbling through Space in a Gold Box" Reconceptualizing New Communication Technologies as Atmospheric

abstract: New communication technologies have undoubtedly altered the ways in which persons interact and have had a profound impact on public life. Engaging this impact, much of the scholarly literature has focused on how these interfaces mediate interaction however, less is known about technology's modulating effects. The current project moves beyond mediation, underscoring how social relations are not only activated by technology, but are actuated by these interfaces. Through an extended case study of Portals, gold shipping containers equipped with audio-visual technology that put persons in digital face-to-face interaction with others around the globe, the current project engages such actuation, highlighting how the co-mingling of affect and technology generate new ways of noticing, living and thinking through the complex relationships of public life. The human/technology relations mediated/modulated by the Portal produce unique atmospheres that activate/actuate public space and blur the boundaries between public and private. Additionally, the atmospheres of the Portal generate a digital co-presence that allows for user/participants to feel with their interlocutors. This “feeling with” suspends user/participants in atmospheres of human connection through the emergence of an imaginative dialogue, and the curating of such atmospheres leads to dialogic transformation. As such, the Portal operates as an atmospheric interface. Engaging the concept of atmosphere attunes those interested in new communication technologies to the complex gatherings these technologies create, and the potentialities and pitfalls of these emerging interfaces on public life.Dissertation/ThesisDoctoral Dissertation Communication Studies 201

The temporalities of tracking sitting time: an exploration of the influence of rhythms and biographies on behavioural change in chronically ill adults and office workers

This thesis explores how older adults with chronic obstructive pulmonary disease (COPD), and office workers, experienced sitting while wearing a self-tracking device that prompted them to break up and reduce their sitting time. My thesis draws on public health and social science research on self-tracking, as well as the temporality and rhythms literature, and I argue that sitting can be understood in relation to the wider social, personal, biographical and institutional contexts to which my participants related their experiences of the past, present and future, and their changing habitual routines. Findings were based on two studies, the motivations behind which were to encourage participants to reduce their sitting and to deduce whether wearing a self-tracking device would inspire them to do so. The first study was a qualitative nested study which was part of a multidisciplinary randomised control trial. This study investigated the feasibility of self-tracking and an educational booklet created to reduce sitting in older adults with COPD. The qualitative nested study interviewed 25 patients with COPD, both before and after the study, and the first interviews explored the contexts of their lives and sitting, while the second explored how they managed with the device, educational advice and the study as a whole. The second study interviewed 24 office workers about their experiences with a self-tracking device designed to reduce their sitting. Each participant was interviewed both before and after the two-week study period in interview 1, I explored their lives, their work and their experiences and associations with sitting, and in interview 2 I investigated their experiences with the device and the study as a whole. My four analytic chapters answer the following four questions: how do patients with chronic obstructive pulmonary disease (COPD), and office workers, use a device to self-track their sitting time? What kinds of meanings do patients with COPD and office workers attach to sitting? How do personal and social or institutional temporalities of the past and present, and the rhythms of everyday life, shape participants sitting and self-tracking? And what does the conceptual framework, focusing on meanings, temporalities and rhythms, add to our understanding of health, sitting and self-tracking? The findings of this thesis revealed that the meaning of sitting was different for my two participant groups, in that they were influenced heavily by their experiences with their past, present and future, as well as their daily routines and changes in pace. Therefore, in order to make sense of how these participants understood the meanings of sitting, I adopted a temporality and rhythms framework, which allowed me to make sense of how COPD participants either looked back on their previous lives and reminisced on happy memories, whereby they were mournful and sad about their current lives and changing behaviour, and sitting less was not important to them, or looked toward their futures in anticipation of a healthier life and the ability to do more. The concept of rhythms allowed me to make sense of how some of these participants felt that the self-tracking device and sitting interrupted or did not fit in with their lives and how they often felt that sitting had positive benefits, or where their existing rhythms had been interrupted by their illness and this prevented behavioural change and a reduction in sitting. The concept of rhythms also helped to make sense of those participants who adopted their existing habitual rhythms to encompass sitting less and self-tracking, or those who engaged when their habitual routines coincided with sitting less and self-tracking. In contrast, office workers sitting and self-tracking were related to the workplace, in that they looked back on previous work times when they would make time for their health and take breaks, thus the concept of temporality helped to make sense of this biographical and institutionally dictated time. The concept of rhythms helped to decipher how these participants did not have an issue with health but associated any negative well-being consequences to their increasingly fast-paced and stressful work lives. In addition, their free time was not considered problematic, and so they did not feel the need to change their behaviour or reduce their sitting or self-tracking during this time, as they saw it as an opportunity to gain some form of freedom and do what they wanted to do. Therefore the concept of rhythms provided a way of understanding the different routines of work and home and how the pace of these rhythms differed in speed and intensity. The thesis provides a new perspective on exploring sitting and highlights the importance of exploring both it and self-tracking in relation to the experiences of biographical time (past, present and future) and changing routines. I offer insights into how, by adopting a rhythms and temporality framework, we can make sense of people s experiences of reducing sitting and engaging with self-tracking in order to do so. The thesis brings together literature on public health, self-tracking and place and time, and it argues that by studying the meaning of sitting and adopting a temporality and rhythms framework, the complexity and experience of time and its relationship with chronic illness and work are illuminated, thereby highlighting how time, place and pace are fundamental in understanding sitting and self-tracking

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

Somatic ABC's: A Theoretical Framework for Designing, Developing and Evaluating the Building Blocks of Touch-Based Information Delivery

abstract: Situations of sensory overload are steadily becoming more frequent as the ubiquity of technology approaches reality--particularly with the advent of socio-communicative smartphone applications, and pervasive, high speed wireless networks. Although the ease of accessing information has improved our communication effectiveness and efficiency, our visual and auditory modalities--those modalities that today's computerized devices and displays largely engage--have become overloaded, creating possibilities for distractions, delays and high cognitive load; which in turn can lead to a loss of situational awareness, increasing chances for life threatening situations such as texting while driving. Surprisingly, alternative modalities for information delivery have seen little exploration. Touch, in particular, is a promising candidate given that it is our largest sensory organ with impressive spatial and temporal acuity. Although some approaches have been proposed for touch-based information delivery, they are not without limitations including high learning curves, limited applicability and/or limited expression. This is largely due to the lack of a versatile, comprehensive design theory--specifically, a theory that addresses the design of touch-based building blocks for expandable, efficient, rich and robust touch languages that are easy to learn and use. Moreover, beyond design, there is a lack of implementation and evaluation theories for such languages. To overcome these limitations, a unified, theoretical framework, inspired by natural, spoken language, is proposed called Somatic ABC's for Articulating (designing), Building (developing) and Confirming (evaluating) touch-based languages. To evaluate the usefulness of Somatic ABC's, its design, implementation and evaluation theories were applied to create communication languages for two very unique application areas: audio described movies and motor learning. These applications were chosen as they presented opportunities for complementing communication by offloading information, typically conveyed visually and/or aurally, to the skin. For both studies, it was found that Somatic ABC's aided the design, development and evaluation of rich somatic languages with distinct and natural communication units.Dissertation/ThesisPh.D. Computer Science 201

Sensing Place and Presence in an INTIMAL Long-Distance Improvisation

INTIMAL is an interactive system for relational listening, which integrates physical-virtual interfaces for people to sonically improvise between distant locations. The aim is to embrace two key aspects in the context of human migration: the sense of place and the sense of presence. This paper reflects on the use of INTIMAL in a long-distance improvisation between the cities of Oslo, Barcelona and London in May 2019. This improvisation was performed by nine Colombian migrant women, who had been involved in a research process using the Deep Listening® practice developed by Pauline Oliveros. Here we describe the performance setting and the implementation of the first two interfaces of the system: MEMENTO, an “embodied” navigator of an oral archive of Colombian women’s testimonies of conflict and migration; and RESPIRO, a sonification system that transmits and sonifies live, breathing signals between distant locations. We reflect on how the two interfaces facilitated and challenged the improvisers’ listening experiences and connections

Design and Effect of Continuous Wearable Tactile Displays

Our sense of touch is one of our core senses and while not as information rich as sight and hearing, it tethers us to reality. Our skin is the largest sensory organ in our body and we rely on it so much that we don\u27t think about it most of the time. Tactile displays - with the exception of actuators for notifications on smartphones and smartwatches - are currently understudied and underused. Currently tactile cues are mostly used in smartphones and smartwatches to notify the user of an incoming call or text message. Specifically continuous displays - displays that do not just send one notification but stay active for an extended period of time and continuously communicate information - are rarely studied. This thesis aims at exploring the utilization of our vibration perception to create continuous tactile displays. Transmitting a continuous stream of tactile information to a user in a wearable format can help elevate tactile displays from being mostly used for notifications to becoming more like additional senses enabling us to perceive our environment in new ways. This work provides a serious step forward in design, effect and use of continuous tactile displays and their use in human-computer interaction. The main contributions include: Exploration of Continuous Wearable Tactile Interfaces This thesis explores continuous tactile displays in different contexts and with different types of tactile information systems. The use-cases were explored in various domains for tactile displays - Sports, Gaming and Business applications. The different types of continuous tactile displays feature one- or multidimensional tactile patterns, temporal patterns and discrete tactile patterns. Automatic Generation of Personalized Vibration Patterns In this thesis a novel approach of designing vibrotactile patterns without expert knowledge by leveraging evolutionary algorithms to create personalized vibration patterns - is described. This thesis presents the design of an evolutionary algorithm with a human centered design generating abstract vibration patterns. The evolutionary algorithm was tested in a user study which offered evidence that interactive generation of abstract vibration patterns is possible and generates diverse sets of vibration patterns that can be recognized with high accuracy. Passive Haptic Learning for Vibration Patterns Previous studies in passive haptic learning have shown surprisingly strong results for learning Morse Code. If these findings could be confirmed and generalized, it would mean that learning a new tactile alphabet could be made easier and learned in passing. Therefore this claim was investigated in this thesis and needed to be corrected and contextualized. A user study was conducted to study the effects of the interaction design and distraction tasks on the capability to learn stimulus-stimulus-associations with Passive Haptic Learning. This thesis presents evidence that Passive Haptic Learning of vibration patterns induces only a marginal learning effect and is not a feasible and efficient way to learn vibration patterns that include more than two vibrations. Influence of Reference Frames for Spatial Tactile Stimuli Designing wearable tactile stimuli that contain spatial information can be a challenge due to the natural body movement of the wearer. An important consideration therefore is what reference frame to use for spatial cues. This thesis investigated allocentric versus egocentric reference frames on the wrist and compared them for induced cognitive load, reaction time and accuracy in a user study. This thesis presents evidence that using an allocentric reference frame drastically lowers cognitive load and slightly lowers reaction time while keeping the same accuracy as an egocentric reference frame, making a strong case for the utilization of allocentric reference frames in tactile bracelets with several tactile actuators

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc