Supporting virtuosity and flow in computer music

As we begin to realise the sonic and expressive potential of the computer, HCI researchers face the challenge of designing rewarding and accessible user experiences that enable individuals to explore complex creative domains such as music. In performance-based music systems such as sequencers, a disjunction exists between the musician’s specialist skill with performance hardware and the generic usability techniques applied in the design of the software. The creative process is not only fragmented across multiple physical (and virtual) devices, but divided across creativity and productivity phases separated by the act of recording. Integrating psychologies of expertise and intrinsic motivation, this thesis proposes a design shift from usability to virtuosity, using theories of “flow” (Csikszentmihalyi, 1996) and feedback “liveness” (Tanimoto, 1990) to identify factors that facilitate learning and creativity in digital notations and interfaces, leading to a set of design heuristics to support virtuosity in notation use. Using the cognitive dimensions of notations framework (Green, 1996), models of the creative user experience are developed, working towards a theoretical framework for HCI in music systems, and specifically computer-aided composition. Extensive analytical methods are used to look at corollaries of virtuosity and flow in real-world computer music interaction, notably in soundtracking, a software-based composing environment offering a rapid edit-audition feedback cycle, enabled by the user’s skill in manipulating the text-based notation (and program) through the computer keyboard. The interaction and development of more than 1,000 sequencer and tracker users was recorded over a period of 2 years, to investigate the nature and development of skill and technique, look for evidence of flow experiences, and establish the use and role of both visual and musical feedback in music software. Quantitative analyses of interaction data are supplemented with a detailed video study of a professional tracker composer, and a user survey that draws on psychometric methods to evaluate flow experiences in the use of digital music notations, such as sequencers and trackers. Empirical findings broadly support the proposed design heuristics, and enable the development of further models of liveness and flow in notation use. Implications for UI design are discussed in the context of existing music systems, and supporting digitally-mediated creativity in other domains based on notation use

A white paper: NASA virtual environment research, applications, and technology

Research support for Virtual Environment technology development has been a part of NASA's human factors research program since 1985. Under the auspices of the Office of Aeronautics and Space Technology (OAST), initial funding was provided to the Aerospace Human Factors Research Division, Ames Research Center, which resulted in the origination of this technology. Since 1985, other Centers have begun using and developing this technology. At each research and space flight center, NASA missions have been major drivers of the technology. This White Paper was the joint effort of all the Centers which have been involved in the development of technology and its applications to their unique missions. Appendix A is the list of those who have worked to prepare the document, directed by Dr. Cynthia H. Null, Ames Research Center, and Dr. James P. Jenkins, NASA Headquarters. This White Paper describes the technology and its applications in NASA Centers (Chapters 1, 2 and 3), the potential roles it can take in NASA (Chapters 4 and 5), and a roadmap of the next 5 years (FY 1994-1998). The audience for this White Paper consists of managers, engineers, scientists and the general public with an interest in Virtual Environment technology. Those who read the paper will determine whether this roadmap, or others, are to be followed

Design Fiction Diegetic Prototyping: A Research Framework for Visualizing Service Innovations

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Purpose: This paper presents a design fiction diegetic prototyping methodology and research framework for investigating service innovations that reflect future uses of new and emerging technologies. Design/methodology/approach: Drawing on speculative fiction, we propose a methodology that positions service innovations within a six-stage research development framework. We begin by reviewing and critiquing designerly approaches that have traditionally been associated with service innovations and futures literature. In presenting our framework, we provide an example of its application to the Internet of Things (IoT), illustrating the central tenets proposed and key issues identified. Findings: The research framework advances a methodology for visualizing future experiential service innovations, considering how realism may be integrated into a designerly approach. Research limitations/implications: Design fiction diegetic prototyping enables researchers to express a range of ‘what if’ or ‘what can it be’ research questions within service innovation contexts. However, the process encompasses degrees of subjectivity and relies on knowledge, judgment and projection. Practical implications: The paper presents an approach to devising future service scenarios incorporating new and emergent technologies in service contexts. The proposed framework may be used as part of a range of research designs, including qualitative, quantitative and mixed method investigations. Originality: Operationalizing an approach that generates and visualizes service futures from an experiential perspective contributes to the advancement of techniques that enables the exploration of new possibilities for service innovation research

Inclusive improvisation: exploring the line between listening and playing music

The field of Accessible Digital Musical Instruments (ADMIs) is growing rapidly, with instrument designers recognising that adaptations to existing Digital Musical Instruments (DMIs) can foster inclusive music making. ADMIs offer opportunities to engage with a wider range of sounds than acoustic instruments. Furthermore, gestural ADMIs free the music maker from relying on screen, keyboard, and mouse-based interfaces for engaging with these sounds. This brings greater opportunities for exploration, improvisation, empowerment, and flow through music making for people with disability and the communities of practice they are part of. This article argues that developing ADMIs from existing DMIs can speed up the process and allow for more immediate access for those with diverse needs. It presents three case studies of a gestural DMI, originally designed by the first author for his own creative practice, played by people with disability in diverse contexts. The article shows that system-based considerations that enabled an expert percussionist to achieve virtuoso performances with the instrument required minimal hardware and software changes to facilitate greater inclusivity. Understanding the needs of players and customising the system-based movement to sound mappings was of far greater importance in making the instrument accessible

Intelligent Interfaces to Empower People with Disabilities

Severe motion impairments can result from non-progressive disorders, such as cerebral palsy, or degenerative neurological diseases, such as Amyotrophic Lateral Sclerosis (ALS), Multiple Sclerosis (MS), or muscular dystrophy (MD). They can be due to traumatic brain injuries, for example, due to a traffic accident, or to brainste

Expanding tangible tabletop interfaces beyond the display

L’augment de popularitat de les taules i superfícies interactives està impulsant la recerca i la innovació en una gran varietat d’àrees, incloent-­‐hi maquinari, programari, disseny de la interacció i noves tècniques d’interacció. Totes, amb l’objectiu de promoure noves interfícies dotades d’un llenguatge més ric, potent i natural. Entre totes aquestes modalitats, la interacció combinada a sobre i per damunt de la superfície de la taula mitjançant tangibles i gestos és actualment una àrea molt prometedora. Aquest document tracta d’expandir les taules interactives més enllà de la superfície per mitjà de l’exploració i el desenvolupament d’un sistema o dispositiu enfocat des de tres vessants diferents: maquinari, programari i disseny de la interacció. Durant l’inici d’aquest document s’estudien i es resumeixen els diferents trets característics de les superfícies interactives tangibles convencionals o 2D i es presenten els treballs previs desenvolupats per l’autor en solucions de programari que acaben resultant en aplicacions que suggereixen l’ús de la tercera dimensió a les superfícies tangibles. Seguidament, es presenta un repàs del maquinari existent en aquest tipus d’interfícies per tal de concebre un dispositiu capaç de detectar gestos i generar visuals per sobre de la superfície, per introduir els canvis realitzats a un dispositiu existent, desenvolupat i cedit per Microsoft Reseach Cambridge. Per tal d’explotar tot el potencial d’aquest nou dispositiu, es desenvolupa un nou sistema de visió per ordinador que estén el seguiment d’objectes i mans en una superfície 2D a la detecció de mans, dits i etiquetes amb sis graus de llibertat per sobre la superfície incloent-­‐hi la interacció tangible i tàctil convencional a la superfície. Finalment, es presenta una eina de programari per a generar aplicacions per al nou sistema i es presenten un seguit d’aplicacions per tal de provar tot el desenvolupament generat al llarg de la tesi que es conclou presentant un seguit de gestos tant a la superfície com per sobre d’aquesta i situant-­‐los en una nova classificació que alhora recull la interacció convencional 2D i la interacció estesa per damunt de la superfície desenvolupada.The rising popularity of interactive tabletops and surfaces is spawning research and innovation in a wide variety of areas, including hardware and software technologies, interaction design and novel interaction techniques, all of which seek to promote richer, more powerful and more natural interaction modalities. Among these modalities, combined interaction on and above the surface, both with gestures and with tangible objects, is a very promising area. This dissertation is about expanding tangible and tabletops surfaces beyond the display by exploring and developing a system from the three different perspectives: hardware, software, and interaction design. This dissertation, studies and summarizes the distinctive affordances of conventional 2D tabletop devices, with a vast literature review and some additional use cases developed by the author for supporting these findings, and subsequently explores the novel and not yet unveiled potential affordances of 3D-­‐augmented tabletops. It overviews the existing hardware solutions for conceiving such a device, and applies the needed hardware modifications to an existing prototype developed and rendered to us by Microsoft Research Cambridge. For accomplishing the interaction purposes, it is developed a vision system for 3D interaction that extends conventional 2D tabletop tracking for the tracking of hand gestures, 6DoF markers and on-­‐surface finger interaction. It finishes by conceiving a complete software framework solution, for the development and implementation of such type of applications that can benefit from these novel 3D interaction techniques, and implements and test several software prototypes as proof of concepts, using this framework. With these findings, it concludes presenting continuous tangible interaction gestures and proposing a novel classification for 3D tangible and tabletop gestures