Data Driven Analysis of Tiny Touchscreen Performance with MicroJam

The widespread adoption of mobile devices, such as smartphones and tablets, has made touchscreens a common interface for musical performance. New mobile musical instruments have been designed that embrace collaborative creation and that explore the affordances of mobile devices, as well as their constraints. While these have been investigated from design and user experience perspectives, there is little examination of the performers' musical outputs. In this work, we introduce a constrained touchscreen performance app, MicroJam, designed to enable collaboration between performers, and engage in a novel data-driven analysis of more than 1600 performances using the app. MicroJam constrains performances to five seconds, and emphasises frequent and casual music making through a social media-inspired interface. Performers collaborate by replying to performances, adding new musical layers that are played back at the same time. Our analysis shows that users tend to focus on the centre and diagonals of the touchscreen area, and tend to swirl or swipe rather than tap. We also observe that while long swipes dominate the visual appearance of performances, the majority of interactions are short with limited expressive possibilities. Our findings are summarised into a set of design recommendations for MicroJam and other touchscreen apps for social musical interaction

Noise Music, Environment and Peripheral Patterns

The development of the project Peripheral Patterns aims toward an autonomous software system for generative music, that takes as a point of departure, how sound is implicated in our experience of the everyday world in the city. This investigation began with the analysis and consideration of the contrasting soundscapes of the city and the natural settings. The analysis was found to show some correlation with the arrangement and frequencies of sounds in soundscapes and post-industrial noise music works. The above observations led me to the further analysis of soundscapes and noise music. My position on the current city soundscape is as follows: As the human is both a creature from the natural and urban environments, these contrasting vistas will create modern sonic topographies which are exhibited in noise music. Peripheral Patterns explores these convergent spaces to investigate the potential of the sound that is at the limits of human hearing. Whereas once animals occupied small spectral niches in the soundfield, a sky train can be heard passing over Commercial Drive as the constant white noise roars from the city. Conversely, there is a desire for something that resembles peace, a desire to return to an experience of natural soundscape. This thesis looks at a possible convergence of urban and natural soundscapes

Tools in and out of sight : an analysis informed by Cultural-Historical Activity Theory of audio-haptic activities involving people with visual impairments supported by technology

The main purpose of this thesis is to present a Cultural-Historical Activity Theory (CHAT) based analysis of the activities conducted by and with visually impaired users supported by audio-haptic technology.This thesis covers several studies conducted in two projects. The studies evaluate the use of audio-haptic technologies to support and/or mediate the activities of people with visual impairment. The focus is on the activities involving access to two-dimensional information, such as pictures or maps. People with visual impairments can use commercially available solutions to explore static information (raised lined maps and pictures, for example). Solu-tions for dynamic access, such as drawing a picture or using a map while moving around, are more scarce. Two distinct projects were initiated to remedy the scarcity of dynamic access solutions, specifically focusing on two separate activities.The first project, HaptiMap, focused on pedestrian outdoors navigation through audio feedback and gestures mediated by a GPS equipped mobile phone. The second project, HIPP, focused on drawing and learning about 2D representations in a school setting with the help of haptic and audio feedback. In both cases, visual feedback was also present in the technology, enabling people with vision to take advantage of that modality too.The research questions addressed are: How can audio and haptic interaction mediate activ-ities for people with visual impairment? Are there features of the programming that help or hinder this mediation? How can CHAT, and specifically the Activity Checklist, be used to shape the design process, when designing audio haptic technology together with persons with visual impairments?Results show the usefulness of the Activity Checklist as a tool in the design process, and provide practical application examples. A general conclusion emphasises the importance of modularity, standards, and libre software in rehabilitation technology to support the development of the activities over time and to let the code evolve with them, as a lifelong iterative development process. The research also provides specific design recommendations for the design of the type of audio haptic systems involved

Hybrid Human Agency: A Teleodynamic Socio-Spatial Interaction Model for Emergent Human Agency Architecture

People relate with one another in space and through imagined and technologically mediated networks. This thesis is concerned with the relationship between these two types of social connections – spatial and network. Spatial connections structure collectives of people in the same place at the same time. Network connections structure relations between people without regard to place or time. Spatial connections are complex, but rigid by nature, while network connections are simple, but flexible. Essential articulations emerge between these two connection types. These articulations create and evolve contemporary socio-spatial systems such as the city, its many places, and groups of people therein. However, the basic human experience of these systems remains largely polarized between spatial and network social practices to the disadvantage of human agency. This thesis proposes a teleodynamic, socio-spatial interaction model for the articulation of these social practices in human agency architecture. The model is a mobile experience design that functions through people with ‘smart’ mobile devices. It connects them with one another in public place and to global information and communication networks simultaneously. Sociological study informs the model’s design – constraints and conditions for the connection extents and integrity of social interaction. The model supports self-organizing circular relationships between human interaction dynamics and their trace structures based on a methodology for emergence in complex systems. It effects the emergence of the aforementioned socio-spatial, human agency architecture, with great flexibility. The model and architecture together serve to better articulate contemporary spatial and network social practices to the benefit of human agency in urban space

Mobile sound: media art in hybrid spaces

The thesis explores the relationships between sound and mobility through an examination of sound art. The research engages with the intersection of sound, mobility and art through original empirical work and theoretically through a critical engagement with sound studies. In dialogue with the work of De Certeau, Lefebvre, Huhtamo and Habermas in terms of the poetics of walking, rhythms, media archeology and questions of publicness, I understand sound art as an experimental mobile and public space. The thesis establishes and situates the emerging field of mobile sound art by mapping three key traditions of mobile sound art - locative art, sound art and public art - and creates a taxonomy of mobile sound art by defining four categories: 'placing sounds', 'sound platforms', 'sonifying mobility' and 'musical instruments' (each represented by one case study). In doing so it develops a methodology that is attentive to the specifics of the sonic and mobile of media experience. I demonstrate how sonic interactions and embodied mobility are designed and experienced in specific ways in each of the four case studies - 'Aura' by Symons (UK), 'Pophorns' by Torstensson and Sandelin (Sweden), 'SmSage' by Redfern and Borland (US) and 'Core Sample' by Rueb (US) (all 2007). In tracing the topos of the musical telephone, discussing the making and breaking of relevant micro publics, accounting for the polyphonies of footsteps and unwrapping bundles of rhythms, this thesis contributes to understanding complex media experiences in hybrid spaces. In doing so it critically sheds light on the quality of sonic artistic experiences, the audience engagement with urban, public and networked spaces and the relationship between sound art and everyday media experience. My thesis provides valuable insight into auditory ways of mobilising and making public spaces, non-verbal and embodied media practices, and rhythms and scales of mobile media experiences

Integrative Sonic Urbanism: Artist-Led Strategies for Urban Sound Design in the Contemporary City

This doctoral research advances the fields of urban sound design and acoustic planning, presenting new ways of exploring the interrelationship between individual and collective sonic experience, the dynamic potential of the urban sound environment and the complex evolution of the contemporary cityscape. It links urban sound art practices with larger urban design processes, revealing how sound contributes to the production of urban space. The research progresses by crafting a dynamic, integrative methodology that activates contrasting sonic perspectives to critically reassess the role of sound in the public realm. As it discloses this methodology, the research navigates the tension between new modes of urban sound design guided by critical artistic practice and more conventional strategies rooted in the paradigm of environmental noise. Efforts to address urban sonic conditions through quantifiable metrics are contextualised within a wider transition in which urban form is increasingly influenced by data capture, analysis and governance. Within this transition, the critical potential of sound as an active component of urban space is obscured by remedial strategies established to improve what are construed as unfavourable conditions. This research analyses the relationship between these remedial strategies, the emergence of the ISO soundscape standard and the concepts of urban ambiances, urban atmospheres and acoustic territories. It postulates that these centralising conceptual models can serve to limit as well as to advance the critical potential of this field, pursuing instead a more tactical, performative and pluralistic methodology. The articulation of this methodology is substantiated through the exposition of three major public artworks developed by the author, including: Continuous Drift (2015–), a permanent sound installation in a public urban square; The Manual for Acoustic Planning and Urban Sound Design (2013–2020), an artist placement exploring the role of the acoustic planner within a local authority; and The Office for Common Sound (2016–), a project space that fosters dialogue concerning sound within specific regional and institutional contexts. These projects expand the role of artistic practice within the context of urban design and spatial planning by activating the field of urban sound design within diverse spatial, administrative and social contexts. These projects extend established methodologies drawn from sound art, site-specific art and sound installation practices with tactics inherited from public, participatory and socially engaged art, demonstrating how artist-led strategies for urban sound design can advance new forms of spatial production through collaboration with diverse urban actors

Integrating passive ubiquitous surfaces into human-computer interaction

Mobile technologies enable people to interact with computers ubiquitously. This dissertation investigates how ordinary, ubiquitous surfaces can be integrated into human-computer interaction to extend the interaction space beyond the edge of the display. It turns out that acoustic and tactile features generated during an interaction can be combined to identify input events, the user, and the surface. In addition, it is shown that a heterogeneous distribution of different surfaces is particularly suitable for realizing versatile interaction modalities. However, privacy concerns must be considered when selecting sensors, and context can be crucial in determining whether and what interaction to perform.Mobile Technologien ermöglichen den Menschen eine allgegenwärtige Interaktion mit Computern. Diese Dissertation untersucht, wie gewöhnliche, allgegenwärtige Oberflächen in die Mensch-Computer-Interaktion integriert werden können, um den Interaktionsraum über den Rand des Displays hinaus zu erweitern. Es stellt sich heraus, dass akustische und taktile Merkmale, die während einer Interaktion erzeugt werden, kombiniert werden können, um Eingabeereignisse, den Benutzer und die Oberfläche zu identifizieren. Darüber hinaus wird gezeigt, dass eine heterogene Verteilung verschiedener Oberflächen besonders geeignet ist, um vielfältige Interaktionsmodalitäten zu realisieren. Bei der Auswahl der Sensoren müssen jedoch Datenschutzaspekte berücksichtigt werden, und der Kontext kann entscheidend dafür sein, ob und welche Interaktion durchgeführt werden soll

An investigation of eyes-free spatial auditory interfaces for mobile devices: supporting multitasking and location-based information

Auditory interfaces offer a solution to the problem of effective eyes-free mobile interactions. However, a problem with audio, as opposed to visual displays, is dealing with multiple simultaneous information streams. Spatial audio can be used to differentiate between different streams by locating them into separate spatial auditory streams. In this thesis, we consider which spatial audio designs might be the most effective for supporting multiple auditory streams and the impact such spatialisation might have on the users' cognitive load. An investigation is carried out to explore the extent to which 3D audio can be effectively incorporated into mobile auditory interfaces to offer users eyes-free interaction for both multitasking and accessing location-based information. Following a successful calibration of the 3D audio controls on the mobile device of choice for this work (the Nokia N95 8GB), a systematic evaluationof 3D audio techniques is reported in the experimental chapters of this thesis which considered the effects of multitasking, multi-level displays, as well as differences between egocentric and exocentric designs. One experiment investigates the implementation and evaluation of a number of different spatial (egocentric) and non-spatial audio techniques for supporting eyes-free mobile multitasking that included spatial minimisation. The efficiency and usability of these techniques was evaluated under varying cognitive load. This evaluation showed an important interaction between cognitive load and the method used to present multiple auditory streams. The spatial minimisation technique offered an effective means of presenting and interacting with multiple auditory streams simultaneously in a selective-attention task (low cognitive load) but it was not as effective in a divided-attention task (high cognitive load), in which the interaction benefited significantly from the interruption of one of the stream. Two further experiments examine a location-based approach to supporting multiple information streams in a realistic eyes-free mobile environment. An initial case study was conducted in an outdoor mobile audio-augmented exploratory environment that allowed for the analysis and description of user behaviour in a purely exploratory environment. 3D audio was found to be an effective technique to disambiguate multiple sound sources in a mobile exploratory environment and to provide a more engaging and immersive experience as well as encouraging an exploratory behaviour. A second study extended the work of the previous case study by evaluating a number of complex multi-level spatial auditory displays that enabled interaction with multiple location-based information in an indoor mobile audio-augmented exploratory environment. It was found that a consistent exocentric design across levels failed to reduce workload or increase user satisfaction, so this design was widely rejected by users. However, the rest of spatial auditory displays tested in this study encouraged an exploratory behaviour similar to that described in the previous case study, here further characterised by increased user satisfaction and low perceived workload