Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable

Interactive Machine Learning for User-Innovation Toolkits – An Action Design Research approach

Machine learning offers great potential to developers and end users in the creative industries. However, to better support creative software developers' needs and empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. This thesis asks the following research questions: How can we apply a user-centred approach to the design of developer tools for rapid prototyping with Interactive Machine Learning? In what ways can we design better developer tools to accelerate and broaden innovation with machine learning? This thesis presents a three-year longitudinal action research study that I undertook within a multi-institutional consortium leading the EU H2020 -funded Innovation Action RAPID-MIX. The scope of the research presented here was the application of a user-centred approach to the design and evaluation of developer tools for rapid prototyping and product development with machine learning. This thesis presents my work in collaboration with other members of RAPID-MIX, including design and deployment of a user-centred methodology for the project, interventions for gathering requirements with RAPID-MIX consortium stakeholders and end users, and prototyping, development and evaluation of a software development toolkit for interactive machine learning. This thesis contributes with new understanding about the consequences and implications of a user-centred approach to the design and evaluation of developer tools for rapid prototyping of interactive machine learning systems. This includes 1) new understanding about the goals, needs, expectations, and challenges facing creative machine-learning non-expert developers and 2) an evaluation of the usability and design trade-offs of a toolkit for rapid prototyping with interactive machine learning. This thesis also contributes with 3) a methods framework of User-Centred Design Actions for harmonising User-Centred Design with Action Research and supporting the collaboration between action researchers and practitioners working in rapid innovation actions, and 4) recommendations for applying Action Research and User-Centred Design in similar contexts and scale

Interactive Machine Learning for End-User

User interaction with intelligent systems need not be limited to interaction where pre-trained software has intelligence “baked in.” End-user training, including interactive machine learning (IML) approaches, can enable users to create and customise systems themselves. We propose that the user experience of these users is worth considering. Furthermore, the user experience of system developers—people who may train and configure both learning algorithms and their user interfaces—also deserves attention. We additionally propose that IML can improve user experiences by supporting usercentred design processes, and that there is a further role for user-centred design in improving interactive and classical machine learning systems. We are developing this approach and embodying it through the design of a new User Innovation Toolkit, in the context of the European Commission-funded project RAPID-MIX

User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit

Machine learning offers great potential to developers and end users in the creative industries. For example, it can support new sensor-based interactions, procedural content generation and end-user product customisation. However, designing machine learning toolkits for adoption by creative developers is still a nascent effort. This work focuses on the application of user-centred design with creative end-user developers for informing the design of an interactive machine learning toolkit. We introduce a framework for user-centred design actions that we developed within the context of an EU innovation project, RAPID-MIX. We illustrate the application of the framework with two actions for lightweight formative evaluation of our toolkit—the JUCE Machine Learning Hackathon and the RAPID-MIX API workshop at eNTERFACE’17. We describe how we used these actions to uncover conceptual and technical limitations. We also discuss how these actions provided us with a better understanding of users, helped us to refine the scope of the design space, and informed improvements to the toolkit. We conclude with a reflection about the knowledge we obtained from applying user-centred design to creative technology, in the context of an innovation project in the creative industries

Interactive Machine Learning for End-User Innovation

User interaction with intelligent systems need not be limited to interaction where pre-trained software has intelligence “baked in.” End-user training, including interactive machine learning (IML) approaches, can enable users to create and customise systems themselves. We propose that the user experience of these users is worth considering. Furthermore, the user experience of system developers—people who may train and configure both learning algorithms and their user interfaces—also deserves attention. We additionally propose that IML can improve user experiences by supporting user-centred design processes, and that there is a further role for user-centred design in improving interactive and classical machine learning systems. We are developing this approach and embodying it through the design of a new User Innovation Toolkit, in the context of the European Commission-funded project RAPID-MIX

Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

Improving User Involvement Through Live Collaborative Creation

Creating an artifact - such as writing a book, developing software, or performing a piece of music - is often limited to those with domain-specific experience or training. As a consequence, effectively involving non-expert end users in such creative processes is challenging. This work explores how computational systems can facilitate collaboration, communication, and participation in the context of involving users in the process of creating artifacts while mitigating the challenges inherent to such processes. In particular, the interactive systems presented in this work support live collaborative creation, in which artifact users collaboratively participate in the artifact creation process with creators in real time. In the systems that I have created, I explored liveness, the extent to which the process of creating artifacts and the state of the artifacts are immediately and continuously perceptible, for applications such as programming, writing, music performance, and UI design. Liveness helps preserve natural expressivity, supports real-time communication, and facilitates participation in the creative process. Live collaboration is beneficial for users and creators alike: making the process of creation visible encourages users to engage in the process and better understand the final artifact. Additionally, creators can receive immediate feedback in a continuous, closed loop with users. Through these interactive systems, non-expert participants help create such artifacts as GUI prototypes, software, and musical performances. This dissertation explores three topics: (1) the challenges inherent to collaborative creation in live settings, and computational tools that address them; (2) methods for reducing the barriers of entry to live collaboration; and (3) approaches to preserving liveness in the creative process, affording creators more expressivity in making artifacts and affording users access to information traditionally only available in real-time processes. In this work, I showed that enabling collaborative, expressive, and live interactions in computational systems allow the broader population to take part in various creative practices.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/145810/1/snaglee_1.pd

Designing a Sensor-Based Wearable Computing System for Custom Hand Gesture Recognition Using Machine Learning

This thesis investigates how assistive technology can be made to facilitate communication for people that are unable to or have difficulty communicating via vocal speech, and how this technology can be made more universal and compatible with the many different types of sign language that they use. Through this research, a fully customisable and stand-alone wearable device was developed, that employs machine learning techniques to recognise individual hand gestures and translate them into text, images and speech. The device can recognise and translate custom hand gestures by training a personal classifier for each user, relying on a small training sample size, that works online on an embedded system or mobile device, with a classification accuracy rate of up to 99%. This was achieved through a series of iterative case studies, with user testing carried out by real users in their every day environments and in public spaces

Designing AI Experiences: Boundary Representations, Collaborative Processes, and Data Tools

Artificial Intelligence (AI) has transformed our everyday interactions with technology through automation, intelligence augmentation, and human-machine partnership. Nevertheless, we regularly encounter undesirable and often frustrating experiences due to AI. A fundamental challenge is that existing software practices for coordinating system and experience designs fall short when creating AI for diverse human needs, i.e., ``human-centered AI'' or HAI. ``AI-first'' development workflows allow engineers to first develop the AI components, and then user experience (UX) designers create end-user experiences around the AI's capabilities. Consequently, engineers encounter end-user blindness when making critical decisions about AI training data needs, implementation logic, behavior, and evaluation. In the conventional ``UX-first'' process, UX designers lack the needed technical understanding of AI capabilities (technological blindness) that limits their ability to shape system design from the ground up. Human-AI design guidelines have been offered to help but neither describe nor prescribe ways to bridge the gaps in needed expertise in creating HAI. In this dissertation, I investigate collaboration approaches between designers and engineers to operationalize the vision for HAI as technology inspired by human intelligence that augments human abilities while addressing societal needs. In a series of studies combining technical HCI research with qualitative studies of AI production in practice, I contribute (1) an approach to software development that blurs rigid design-engineering boundaries, (2) a process model for co-designing AI experiences, and (3) new methods and tools to empower designers by making AI accessible to UX designers. Key findings from interviews with industry practitioners include the need for ``leaky'' abstractions shared between UX and AI designers. Because modular development and separation of concerns fail with HAI design, leaky abstractions afford collaboration across expertise boundaries and support human-centered design solutions through vertical prototyping and constant evaluation. Further, by observing how designers and engineers collaborate on HAI design in an in-lab study, I highlight the role of design `probes' with user data to establish common ground between AI system and UX design specifications, providing a critical tool for shaping HAI design. Finally, I offer two design methods and tool implementations --- Data-Assisted Affinity Diagramming and Model Informed Prototyping --- for incorporating end-user data into HAI design. HAI is necessarily a multidisciplinary endeavor, and human data (in multiple forms) is the backbone of AI systems. My dissertation contributions inform how stakeholders with differing expertise can collaboratively design AI experiences by reducing friction across expertise boundaries and maintaining agency within team roles. The data-driven methods and tools I created provide direct support for software teams to tackle the novel challenges of designing with data. Finally, this dissertation offers guidance for imagining future design tools for human-centered systems that are accessible to diverse stakeholders.PHDInformationUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169917/1/harihars_1.pd