Analysis of conceptual metaphors to improve music software: the role of prior experience in inclusive music interaction

An open challenge for interaction designers is to find ways of designing software solutions capable of enhancing the ability of users with differing levels of knowledge and capabilities to perform tasks that normally require specialised domain expertise. This challenge is particularly demanding in areas such as music analysis, where complex, abstract, domain-specific concepts and notations occur. One promising theoretical foundation for this work involves the identification of what are assumed to be conceptual metaphors and image schemas. These constructs are grounded in prior sensory-motor experiences of space, force, orientation and interactions with other bodies and are identified through analysing discourse. These theories have already been applied with some success both to musical concepts and, separately, to user interface design with the intention of creating more intuitive designs through exploiting these prior sensory-motor experiences. However, the present work appears to be the first to combine these hitherto distinct bodies of research. We hypothesise that if we can identify the conceptual metaphors and image schemas used by musical experts to structure their understanding of music, we can utilise this information to develop a general theoretical framework to aid designers in systematically improving their designs, providing a foundation for inclusive music interaction design and thus opening up software normally restricted to domain experts to a wider and more varied user population. A possible methodology for the development of such a framework is discussed along with details of a study to apply the conceptual metaphors and image schemas used by a small group of experienced musicians to an evaluation of Harmony Space

Digitization, innovation, and participation: digital conviviality of the Google Cultural Institute

2018 Summer.Includes bibliographical references.To view the abstract, please see the full text of the document

Song walker harmony space: embodied interaction design for complex musical skills

Tonal harmony is widely considered to be the most technical and complex part of music theory. Consequently harmonic skills can be hard to acquire. Furthermore, experience of the flexible manipulation of harmony in real time generally requires the ability to play an instrument. Even for those with instrumental skills, it can be difficult to gain clear insight into harmonic abstractions. The above state of affairs gives rise to substantial barriers not only for beginners but also for many experienced musicians. To address these problems, Harmony Space (Holland et al. 2009) is an interactive digital music system designed to give insight into a wide range of musical tasks in tonal harmony, ranging from performance and composition to analysis. Harmony Space employs a principled set of spatial mappings to offer fluid, precise, intuitive control of harmony. These mappings give rise to sensory-motor and music-theoretic affordances that are hard to obtain in any other way. As a result, harmonic abstractions are rendered amenable to concrete, visible control by simple spatial manipulation. In the language of conceptual metaphor theory, many relationships in tonal harmony become accessible to rapid, universal, low-level, robust human inference mechanisms using image schemata such as containment, contact, centre-periphery, and source-path-goal. This process is more rapid, and imposes far less cognitive load, than slow, abstract symbolic reasoning. Using the above principles, several versions of Harmony Space have been de-signed to exploit specific interaction styles for different purposes. We note some key variants, such as the desktop version (Holland 1994), the camera tracked version (Holland et al. 2009), while focusing principally on the most recent version, Song Walker (Holland et al. 2011), which employs whole body interaction. Preliminary results from a study of the Song Walker system are outlined, in which both beginners and expert musicians undertook a range of musical tasks involving the performance, composition and analysis of music. Finally, we offer a discussion of the limitations of the current system, and outline directions for future wor

CONCRETE ABSTRACT EXPLORING TACTILITY IN ABSTRACT ANIMATIONS FROM EARLY AVANT-GARDE FILMS TO CONTEMPORARY ARTWORKS

After witnessing social chaos and the collapse of values at the beginning of the twentieth century, avant-garde artists insert new thought patterns and progressive aesthetic into the traditional perception of art. Being enthralled by the new film medium, former painters like Viking Eggeling, Walther Ruttman and Hans Richter start to experiment with light in two-dimensional film formats, they animate lines, stripes, basic shapes, play with the foreground and the background, and, most important of all, they construct a temporality within the visual order of the screen. Viking Eggeling’s Symphonie Diagonale (1921-24), Walther Ruttman’s Opus I (1921) and Hans Richter’s Rhythmus 21 (1921) show such temporality built in, which is caught by the idea of music as their titles suggest. These short abstract animation films attempt to discover the artistic possibilities of the new developing medium, film. Like the pioneer avant-garde abstract filmmakers, today’s artists still seek to stimulate a new perception for a possible embodiment that will activate the sense of touch in the audience. Tactility, enhanced by the material, opens up a new network of spa-tio-temporal relationships in the viewer’s consciousness and subjecthood. This essay aims to bring a historical perspective to the abstract moving images of which the tactile or haptic experience is a defining characteristic. Through a selection of abstract animations, the materiality of the film image and the screening site will be elaborated upon according to the haptic features that are corporally embodied by the viewers. In the light of historical abstract animation, the aim is to dwell upon the dynamics of a continuous tendency to capture tactile instances to help bring forth the spatial resonances as well as visualize and reedify the rhythmic passing of time. © 2021 BY-NC

Song Walker: Embodied interaction design for harmony

Tonal Harmony is widely considered to be the most technical and complex part of music theory, and harmonic skills can be hard to acquire. Experience of the precise and flexible manipulation of harmony in real time generally requires hard-won instrumental skill. Even with instrumental skills, it can be hard to gain clear insight into harmonic abstractions. The above state of affairs gives rise to substantial barriers not only for beginners but also for many musicians. To address these problems, Harmony Space [Holland et al, 2009] is an interactive digital music system designed to give insight into a wide range of musical tasks in tonal harmony ranging from performance and composition to analysis. Harmony Space employs a principled set of spatial mappings to offer fluid, precise, intuitive control of harmony. These mappings give rise to sensory-motor, music-theoretic and information-theoretic affordances that are not readily obtainable in any other way. The result is that a wide range of harmonic abstractions are rendered amenable to concrete, visible manipulation by simple spatial means. In the language of conceptual metaphor theory, most relationships in tonal harmony become accessible, to rapid, universal, low-level, robust human inference mechanisms using image schema, such as containment, contact, centre-periphery, and source-path-goal, in place of slow, abstract symbolic reasoning. While keeping the above principles invariant, different versions of Harmony Space have been designed to exploit different detailed interaction styles for different purposes. We note some key variants, such as the desktop version [Holland, 1994], the camera tracked version [Holland et al., 2009], and the most recent whole body version, Song Walker [Holland et al., 2011]. Preliminary results from a recent study of the Song Walker system are outlined, in which both beginners and expert musicians undertook a range of solo and collaborative musical tasks involving the performance, composition and analysis of music. Finally, we offer a discussion of the limitations of the current system, and outline directions for future work.Paper presented at the workshop: When Words Fail: What can Music Interaction tell us about HCI? The 25th British Computer Society Conference on Human-Computer Interaction

Concrete Abstract: Exploring Tactility in Abstract Animations from Early Avant-garde Films to Contemporary Artworks

After witnessing social chaos and the collapse of values at the beginning of the twentieth century, avant-garde artists insert new thought patterns and progressive aesthetic into the traditional perception of art. Being enthralled by the new film medium, former painters like Viking Eggeling, Walther Ruttman and Hans Richter start to experiment with light in two-dimensional film formats, they animate lines, stripes, basic shapes, play with the foreground and the background, and, most important of all, they construct a temporality within the visual order of the screen. Viking Eggeling’s Symphonie Diagonale (1921-24), Walther Ruttman’s Opus I (1921) and Hans Richter’s Rhythmus 21 (1921) show such temporality built in, which is caught by the idea of music as their titles suggest. These short abstract animation films attempt to discover the artistic possibilities of the new developing medium, film. Like the pioneer avant-garde abstract filmmakers, today’s artists still seek to stimulate a new perception for a possible embodiment that will activate the sense of touch in the audience. Tactility, enhanced by the material, opens up a new network of spatio-temporal relationships in the viewer's consciousness and subjecthood. This essay aims to bring a historical perspective to the abstract moving images of which the tactile or haptic experience is a defining characteristic. Through a selection of abstract animations, the materiality of the film image and the screening site will be elaborated upon according to the haptic features that are corporally embodied by the viewers. In the light of historical abstract animation, the aim is to dwell upon the dynamics of a continuous tendency to capture tactile instances to help bring forth the spatial resonances as well as visualize and reedify the rhythmic passing of time

When technology goes out of control

This paper uses the example of software and electronic devices used in musical improvisation to develop a critique of the dominant view of technology, specified by function and input–output behaviour, and optimized so that it is as domesticated as a faithful dog. The optimization in question attempts to avoid discontinuity and, more generally, unforeseen responses from a system, assuming a human being’s need for an interface is purely functional. Against this, we argue that some devices are, by their nature, complex and chaotic, and also that, because of this complexity, we can form deep attachments to them. These interspecies forms of affection are rooted in the sense of incompleteness of the human, its uncertainty in relation to an other and the reasons why, while a synthetic companion can be desirable because more predictable, in the case of improvisational interaction we desire our machinic counterparts to surprise us

Vernacular Posthumanism: Visual Culture and Material Imagination

Vernacular Posthumanism: Visual Culture and Material Imagination uses a theory of image vernaculars in order to explore the ways in which contemporary visual culture both reflects on and constructs 21st century cultural attitudes toward the human and the nonhuman. This project argues that visual culture manifests a vernacular posthumanism that expresses a fundamental contradiction: the desire to transcend the human while at the same time reasserting the importance of the flesh and the materiality of lived experience. This contradiction is based in a biodeterminist desire, one that fantasizes about reducing all actants, both human and nonhuman, to functions of code. Within this framework, actants become fundamentally exchangeable, able to be combined, manipulated, and understood as variations of digital code. Visual culture – and its expression of vernacular posthumanism – thus functions as a reflection on contemporary conceptualizations of the human, a rehearsal of the posthuman, and a staging ground for encounters between the human and the nonhuman. Each chapter of this project begins in the field of film studies and then moves out toward a broader analysis of visual culture and nonhumanist theory. This project relies on the theories and methodologies of phenomenology, materialism, posthumanism, object-oriented ontology, actor-network theory, film and media studies, and visual culture studies. Visual objects analyzed include: the films of Stanley Kubrick, David Cronenberg, and Krzysztof Kieślowski; Fast, Cheap & Out of Control (1997); the film 300 (2006); the TV series Planet Earth (2006); DNA portraits, the art of Damien Hirst; Body Worlds; human migration maps; and remote surgical machinery