Interactive Machine Learning for End-User Innovation

User interaction with intelligent systems need not be limited to interaction where pre-trained software has intelligence “baked in.” End-user training, including interactive machine learning (IML) approaches, can enable users to create and customise systems themselves. We propose that the user experience of these users is worth considering. Furthermore, the user experience of system developers—people who may train and configure both learning algorithms and their user interfaces—also deserves attention. We additionally propose that IML can improve user experiences by supporting user-centred design processes, and that there is a further role for user-centred design in improving interactive and classical machine learning systems. We are developing this approach and embodying it through the design of a new User Innovation Toolkit, in the context of the European Commission-funded project RAPID-MIX

Interactive Machine Learning for User-Innovation Toolkits – An Action Design Research approach

Machine learning offers great potential to developers and end users in the creative industries. However, to better support creative software developers' needs and empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. This thesis asks the following research questions: How can we apply a user-centred approach to the design of developer tools for rapid prototyping with Interactive Machine Learning? In what ways can we design better developer tools to accelerate and broaden innovation with machine learning? This thesis presents a three-year longitudinal action research study that I undertook within a multi-institutional consortium leading the EU H2020 -funded Innovation Action RAPID-MIX. The scope of the research presented here was the application of a user-centred approach to the design and evaluation of developer tools for rapid prototyping and product development with machine learning. This thesis presents my work in collaboration with other members of RAPID-MIX, including design and deployment of a user-centred methodology for the project, interventions for gathering requirements with RAPID-MIX consortium stakeholders and end users, and prototyping, development and evaluation of a software development toolkit for interactive machine learning. This thesis contributes with new understanding about the consequences and implications of a user-centred approach to the design and evaluation of developer tools for rapid prototyping of interactive machine learning systems. This includes 1) new understanding about the goals, needs, expectations, and challenges facing creative machine-learning non-expert developers and 2) an evaluation of the usability and design trade-offs of a toolkit for rapid prototyping with interactive machine learning. This thesis also contributes with 3) a methods framework of User-Centred Design Actions for harmonising User-Centred Design with Action Research and supporting the collaboration between action researchers and practitioners working in rapid innovation actions, and 4) recommendations for applying Action Research and User-Centred Design in similar contexts and scale

Interactive Machine Learning for End-User

User interaction with intelligent systems need not be limited to interaction where pre-trained software has intelligence “baked in.” End-user training, including interactive machine learning (IML) approaches, can enable users to create and customise systems themselves. We propose that the user experience of these users is worth considering. Furthermore, the user experience of system developers—people who may train and configure both learning algorithms and their user interfaces—also deserves attention. We additionally propose that IML can improve user experiences by supporting usercentred design processes, and that there is a further role for user-centred design in improving interactive and classical machine learning systems. We are developing this approach and embodying it through the design of a new User Innovation Toolkit, in the context of the European Commission-funded project RAPID-MIX

User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit

Machine learning offers great potential to developers and end users in the creative industries. For example, it can support new sensor-based interactions, procedural content generation and end-user product customisation. However, designing machine learning toolkits for adoption by creative developers is still a nascent effort. This work focuses on the application of user-centred design with creative end-user developers for informing the design of an interactive machine learning toolkit. We introduce a framework for user-centred design actions that we developed within the context of an EU innovation project, RAPID-MIX. We illustrate the application of the framework with two actions for lightweight formative evaluation of our toolkit—the JUCE Machine Learning Hackathon and the RAPID-MIX API workshop at eNTERFACE’17. We describe how we used these actions to uncover conceptual and technical limitations. We also discuss how these actions provided us with a better understanding of users, helped us to refine the scope of the design space, and informed improvements to the toolkit. We conclude with a reflection about the knowledge we obtained from applying user-centred design to creative technology, in the context of an innovation project in the creative industries

On the challenges and opportunities in visualization for machine learning and knowledge extraction: A research agenda

We describe a selection of challenges at the intersection of machine learning and data visualization and outline a subjective research agenda based on professional and personal experience. The unprecedented increase in the amount, variety and the value of data has been significantly transforming the way that scientific research is carried out and businesses operate. Within data science, which has emerged as a practice to enable this data-intensive innovation by gathering together and advancing the knowledge from fields such as statistics, machine learning, knowledge extraction, data management, and visualization, visualization plays a unique and maybe the ultimate role as an approach to facilitate the human and computer cooperation, and to particularly enable the analysis of diverse and heterogeneous data using complex computational methods where algorithmic results are challenging to interpret and operationalize. Whilst algorithm development is surely at the center of the whole pipeline in disciplines such as Machine Learning and Knowledge Discovery, it is visualization which ultimately makes the results accessible to the end user. Visualization thus can be seen as a mapping from arbitrarily high-dimensional abstract spaces to the lower dimensions and plays a central and critical role in interacting with machine learning algorithms, and particularly in interactive machine learning (iML) with including the human-in-the-loop. The central goal of the CD-MAKE VIS workshop is to spark discussions at this intersection of visualization, machine learning and knowledge discovery and bring together experts from these disciplines. This paper discusses a perspective on the challenges and opportunities in this integration of these discipline and presents a number of directions and strategies for further research

User producer interaction in context: a classification

Science, Technology and Innovation Studies show that intensified user producer interaction (UPI) increases chances for successful innovations, especially in the case of emerging technology. It is not always clear, however, what type of interaction is necessary in a particular context. This paper proposes a conceptualization of contexts in terms of three dimensions – the phase of technology development, the flexibility of the technology, and the heterogeneity of user populations – resulting in a classification scheme with eight different contextual situations. The paper identifies and classifies types of interaction, like demand articulation, interactive learning, learning by using and domestication. It appears that each contextual situation demands a different set of UPI types. To illustrate the potential value of the classification scheme, four examples of innovations with varying technological and user characteristics are explored: the refrigerator, clinical anaesthesia, video cassette recording, and the bicycle. For each example the relevant UPI types are discussed and it is shown how these types highlight certain activities and interactions during key events of innovation processes. Finally, some directions for further research are suggested alongside a number of comments on the utility of the classification

Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable