Integrating passive ubiquitous surfaces into human-computer interaction

Mobile technologies enable people to interact with computers ubiquitously. This dissertation investigates how ordinary, ubiquitous surfaces can be integrated into human-computer interaction to extend the interaction space beyond the edge of the display. It turns out that acoustic and tactile features generated during an interaction can be combined to identify input events, the user, and the surface. In addition, it is shown that a heterogeneous distribution of different surfaces is particularly suitable for realizing versatile interaction modalities. However, privacy concerns must be considered when selecting sensors, and context can be crucial in determining whether and what interaction to perform.Mobile Technologien ermöglichen den Menschen eine allgegenwärtige Interaktion mit Computern. Diese Dissertation untersucht, wie gewöhnliche, allgegenwärtige Oberflächen in die Mensch-Computer-Interaktion integriert werden können, um den Interaktionsraum über den Rand des Displays hinaus zu erweitern. Es stellt sich heraus, dass akustische und taktile Merkmale, die während einer Interaktion erzeugt werden, kombiniert werden können, um Eingabeereignisse, den Benutzer und die Oberfläche zu identifizieren. Darüber hinaus wird gezeigt, dass eine heterogene Verteilung verschiedener Oberflächen besonders geeignet ist, um vielfältige Interaktionsmodalitäten zu realisieren. Bei der Auswahl der Sensoren müssen jedoch Datenschutzaspekte berücksichtigt werden, und der Kontext kann entscheidend dafür sein, ob und welche Interaktion durchgeführt werden soll

UNCAGED: a novel, ‘telesymbiotic’ approach to bridge the divide between the physical world and the virtual world of computers?

The main subject of this thesis is my artistic project UNCAGED, which explores interrelationships and transitions between computer-based virtual environments and their immediate physical surroundings. The underlying motivation behind my approach was to ‘uncage’ screen-based realities from the confines of their digital existence and to bring the remote computer world closer to our human experience. In particular, my work was opposed to the notion of immersive ‘virtual reality’ where the physical world is more or less excluded from the participants, but instead attempted to situate the virtual domain within the physical world. Initially, I will discuss the theoretical framework behind UNCAGED, ranging from aesthetic considerations, the particular role of sound, human computer interaction (HCI) to technical issues, and afterwards describe the creation process of UNCAGED. Based on a study of audience behaviour with UNCAGED at a major London museum, I will claim that the work’s popularity seems to relate to its perceptually intriguing fusion between the virtual domain and the physical world, and in this respect my project can be deemed successful. Furthermore, on the basis of an extended review and analysis of related work in the broad area of ‘mixed reality’, I will suggest that my own approachcan indeed be viewed as a novel way to bridge the divide between the physical world and the virtual world of computers. The innovation relates, in particular, to its unique balance of formal simplicity and technical sophistication. In the last chapter, I will provide a more critical evaluation of UNCAGED, largely informed by Jean Baudrillard’s conception of the ‘real’ and the ‘virtual’, which raises questions about the very idea of integrating digital technology in our lives in a meaningful and satisfying way. Finally, I will present my subsequent practical work, which strongly engages with my critical reflections on UNCAGED. In particular, it is informed by a new heightened sensitivity regarding the role of digital technology in my artistic practice

Computed fingertip touch for the instrumental control of musical sound with an excursion on the computed retinal afterimage

In this thesis, we present an articulated, empirical view on what human music making is, and on how this fundamentally relates to computation. The experimental evidence which we obtained seems to indicate that this view can be used as a tool, to systematically generate models, hypotheses and new technologies that enable an ever more complete answer to the fundamental question as to what forms of instrumental control of musical sound are possible to implement. This also entails the development of two novel transducer technologies for computed fingertip touch: The cyclotactor (CT) system, which provides fingerpad-orthogonal force output while tracking surface-orthogonal fingertip movement; and the kinetic surface friction transducer (KSFT) system, which provides fingerpad-parallel force output while tracking surface-parallel fingertip movement. In addition to the main research, the thesis also contains two research excursions, which are due to the nature of the Ph.D. position. The first excursion shows how repeated and varying pressing movements on the already held-down key of a computer keyboard can be used both to simplify existing user interactions and to implement new ones, that allow the rapid yet detailed navigation of multiple possible interaction outcomes. The second excursion shows that automated computational techniques can display shape specifically in the retinal afterimage, a well-known effect in the human visual system.Computer Systems, Imagery and Medi

Interaction design for live performance

PhD Thesis Multimedia item accompanying this thesis to be consulted at Robinson LibraryThe role of interactive technology in live performance has increased substantially in recent years. Practices and experiences of existing forms of live performance have been transformed and new genres of technology-­‐mediated live performance have emerged in response to novel technological opportunities. Consequently, designing for live performance is set to become an increasingly important concern for interaction design researchers and practitioners. However, designing interactive technology for live performance is a challenging activity, as the experiences of both performers and their audiences are shaped and influenced by a number of delicate and interconnected issues, which relate to different forms and individual practices of live performance in varied and often conflicting ways. The research presented in this thesis explores how interaction designers might be better supported in engaging with this intricate and multifaceted design space. This is achieved using a practice-­‐led methodology, which involves the researcher’s participation in both the investigation of, and design response to, issues of live performance as they are embodied in the lived and felt experiences of individual live performers’ practices during three interaction design case studies. This research contributes to the field of interaction design for live performance in three core areas. Understandings of the relationships between key issues of live performance and individual performers’ lived and felt experiences are developed, approaches to support interaction designers in engaging individual live performers’ lived and felt experiences in design are proposed and innovative interfaces and interaction techniques for live performance are designed. It is anticipated that these research outcomes will prove directly applicable or inspiring to the practices of interaction designers wishing to address live performance and will contribute to the ongoing academic discourse around the experience of, and design for, live performance.Engineering and Physical Sciences Research Council

The Seaboard: discreteness and continuity in musical interface design

The production of acoustic music bridges two senses—touch and hearing—by connecting physical movements, gestures, and tactile interactions with the creation of sound. Mastery of acoustic music depends on the development and refinement of muscle memory and ear training in concert. This process leads to a capacity for great depth of expression even though the actual timbral palette of each given acoustic instrument is relatively limited. By contrast, modern modes of music creation involving recorded music and digital sound manipulation sacrifice this immediate bridge and substitute more abstract processes that enable sonic possibilities extending far beyond the acoustic palette. Mastery in abstract approaches to music making doesn’t necessarily rely on muscle memory or ear training, as many key processes do not need to happen in realtime. This freedom from the limits of time and practiced physical manipulation radically increases the range of achievable sounds, rhythms and effects, but sometimes results in a loss of subtlety of expressiveness. This practice-based PhD asks whether it is possible, and if so how, to achieve an integration of relevant sensor technologies, design concepts, and formation techniques to create a new kind of musical instrument and sound creation tool that bridges this gap with a satisfying result for musicians and composers. In other words, can one create new, multi-dimensional interfaces which provide more effective ways to control the expressive capabilities of digital music creation in real-time? In particular, can one build on the intuitive, logical, and well-known layout of the piano keyboard to create a new instrument that more fully enables both continuous and discrete approaches to music making? My research practice proposes a new musical instrument called the Seaboard, documents its invention, development, design, and refinement, and evaluates the extent to which it positively answers the above question. The Seaboard is a reinterpretation of the piano keyboard as a soft, continuous wavelike surface that places polyphonic pitch bend, vibrato and continuous touch right at the musician’s fingertips. The addition of new realtime parameters to a familiar layout means it combines the intuitiveness of the traditional instrument with some of the versatility of digital technology. Designing and prototyping the Seaboard to the point of successfully proving that a new synthesis between acoustic techniques and digital technologies is possible is shown to require significant coordination and integration of a range of technical disciplines. The research approach has been to build and refine a series of prototypes that successively grapple with the integration of these elements, whilst rigorously documenting the design issues, engineering challenges, and ultimate decisions that determine whether an intervention in the field of musical instrumentation is fruitful

Experimental turntablism - live performances with second hand technology: Analysis and methodological considerations

In experimental turntablism, sound artists and musicians encounter not only the pre-recorded sound of the vinyl records, as is common in DJ culture and hip hop turntablism, but also accentuate the materiality of the records and turntables themselves. The thesis shows that the record player is itself the key concept within which each experimental turntablist unfolds an intricate dialogue between mediation and materiality. Through these media-specific practices, these sound artists raise to the surface the fact that our listening habits tend to dissolve the reproduction medium from our awareness. This thesis explores experimental turntablism in live performance and presents an innovative methodology that establishes the ideas and tools for a potentially generalisable approach to performance analysis for concerts using live electronics. The analytical framework, disclosing the medial and sensual significance of experimental turntablism performances in a digital era, broadens the perspective on sound with theories of performativity, materiality, mediality and instrumentality in electronic music. The thesis methodology includes performance analysis, artist interviews, video and audio recordings and interactive graphical transcriptions based on the current music analysis software EAnalysis. Three case studies examine three distinct artistic approaches: the specific focus of each experimental turntablist varies from playing techniques, to sculptural objects, to mechanical operations. Joke Lanz’s direct and embodied playing negotiates a sound production between signal and noise, musicalises samples, and leads to spontaneous acts with site-specific aspects. Vinyl -terror & -horror destruct playback devices and vinyl records to re-structure samples in chance processes; the duo accompany their sculptural objects with movie soundtracks and ‘unfinished compositions’ from their own records to engender cinematic soundscapes and imaginary scenes. Graham Dunning’s turntable construction sequences patterned discs, which trigger auxiliary instruments through the turntable’s rotary motor operations. These mechanical movements embody rhythmic loop structures with temporal inconsistencies, creating a mechanical techno. Having been considered redundant following the introduction of digital media, the vinyl record has recently witnessed a revival. As a post-digital tendency, contemporary musicians using live electronics seek to recover tactile and physical actions in performance. This thesis shows the ways in which the turntable allows artists to develop personal instruments from ready-made products and to emphasise specific sensual-bodily aspects

3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance

This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil

Extending physical instruments using sampled acoustics

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2007.Includes bibliographical references (p. 133-138).This thesis presents a system architecture for creating hybrid digital-acoustic percussion instruments by combining extensions of existing signal processing techniques with specially-designed semi-acoustic physical controllers. This work aims to provide greater realism to digital percussion, gaining much of the richness and understandability of acoustic instruments while preserving the flexibility of digital systems. For this thesis, I have collaborated with percussionists to develop a range of instruments, to refine and extend the algorithmic and physical designs, and to determine successful models of interaction. Conventional percussion controllers measure and discretize the intensity of strikes into discrete trigger messages, but they also ignore the timbre of the hits and fail to track more ambiguous input. In this work, the continuous acoustic output of a struck physical object is processed to add the resonance of a sampled instrument. This is achieved by employing existing low-latency convolution algorithms which have been extended to give the player control over features such as damping, spectral flattening, nonlinear effects, and pitch.(cont.) One of the advantages of this approach is that light taps, scrapes, rubs, or stirring with brushes all take on a hybrid timbre of the real and sampled sound that is surprisingly realistic and controllable. Since part of its behavior is inherently acoustic, a player's intuition about interacting with physical objects can be applied to controlling it. The ability to transform the apparent acoustic properties of objects also suggests applications to HCI and product design contexts.by Roberto Mario Aimi.Ph.D