Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable

Methodological development

Book description: Human-Computer Interaction draws on the fields of computer science, psychology, cognitive science, and organisational and social sciences in order to understand how people use and experience interactive technology. Until now, researchers have been forced to return to the individual subjects to learn about research methods and how to adapt them to the particular challenges of HCI. This is the first book to provide a single resource through which a range of commonly used research methods in HCI are introduced. Chapters are authored by internationally leading HCI researchers who use examples from their own work to illustrate how the methods apply in an HCI context. Each chapter also contains key references to help researchers find out more about each method as it has been used in HCI. Topics covered include experimental design, use of eyetracking, qualitative research methods, cognitive modelling, how to develop new methodologies and writing up your research

Learning Dimensions: Lessons from Field Studies

In this paper, we describe work to investigate the creation of engaging programming learning experiences. Background research informed the design of four fieldwork studies involving a range of age groups to explore how programming tasks could best be framed to motivate learners. Our empirical findings from these four studies, described here, contributed to the design of a set of programming "Learning Dimensions" (LDs). The LDs provide educators with insights to support key design decisions for the creation of engaging programming learning experiences. This paper describes the background to the identification of these LDs and how they could address the design and delivery of highly engaging programming learning tasks. A web application has been authored to support educators in the application of the LDs to their lesson design

ICS Materials. Towards a re-Interpretation of material qualities through interactive, connected, and smart materials.

The domain of materials for design is changing under the influence of an increased technological advancement, miniaturization and democratization. Materials are becoming connected, augmented, computational, interactive, active, responsive, and dynamic. These are ICS Materials, an acronym that stands for Interactive, Connected and Smart. While labs around the world are experimenting with these new materials, there is the need to reflect on their potentials and impact on design. This paper is a first step in this direction: to interpret and describe the qualities of ICS materials, considering their experiential pattern, their expressive sensorial dimension, and their aesthetic of interaction. Through case studies, we analyse and classify these emerging ICS Materials and identified common characteristics, and challenges, e.g. the ability to change over time or their programmability by the designers and users. On that basis, we argue there is the need to reframe and redesign existing models to describe ICS materials, making their qualities emerge

Proficiency-aware systems

In an increasingly digital world, technological developments such as data-driven algorithms and context-aware applications create opportunities for novel human-computer interaction (HCI). We argue that these systems have the latent potential to stimulate users and encourage personal growth. However, users increasingly rely on the intelligence of interactive systems. Thus, it remains a challenge to design for proficiency awareness, essentially demanding increased user attention whilst preserving user engagement. Designing and implementing systems that allow users to become aware of their own proficiency and encourage them to recognize learning benefits is the primary goal of this research. In this thesis, we introduce the concept of proficiency-aware systems as one solution. In our definition, proficiency-aware systems use estimates of the user's proficiency to tailor the interaction in a domain and facilitate a reflective understanding for this proficiency. We envision that proficiency-aware systems leverage collected data for learning benefit. Here, we see self-reflection as a key for users to become aware of necessary efforts to advance their proficiency. A key challenge for proficiency-aware systems is the fact that users often have a different self-perception of their proficiency. The benefits of personal growth and advancing one's repertoire might not necessarily be apparent to users, alienating them, and possibly leading to abandoning the system. To tackle this challenge, this work does not rely on learning strategies but rather focuses on the capabilities of interactive systems to provide users with the necessary means to reflect on their proficiency, such as showing calculated text difficulty to a newspaper editor or visualizing muscle activity to a passionate sportsperson. We first elaborate on how proficiency can be detected and quantified in the context of interactive systems using physiological sensing technologies. Through developing interaction scenarios, we demonstrate the feasibility of gaze- and electromyography-based proficiency-aware systems by utilizing machine learning algorithms that can estimate users' proficiency levels for stationary vision-dominant tasks (reading, information intake) and dynamic manual tasks (playing instruments, fitness exercises). Secondly, we show how to facilitate proficiency awareness for users, including design challenges on when and how to communicate proficiency. We complement this second part by highlighting the necessity of toolkits for sensing modalities to enable the implementation of proficiency-aware systems for a wide audience. In this thesis, we contribute a definition of proficiency-aware systems, which we illustrate by designing and implementing interactive systems. We derive technical requirements for real-time, objective proficiency assessment and identify design qualities of communicating proficiency through user reflection. We summarize our findings in a set of design and engineering guidelines for proficiency awareness in interactive systems, highlighting that proficiency feedback makes performance interpretable for the user.In einer zunehmend digitalen Welt schaffen technologische Entwicklungen - wie datengesteuerte Algorithmen und kontextabhängige Anwendungen - neuartige Interaktionsmöglichkeiten mit digitalen Geräten. Jedoch verlassen sich Nutzer oftmals auf die Intelligenz dieser Systeme, ohne dabei selbst auf eine persönliche Weiterentwicklung hinzuwirken. Wird ein solches Vorgehen angestrebt, verlangt dies seitens der Anwender eine erhöhte Aufmerksamkeit. Es ist daher herausfordernd, ein entsprechendes Design für Kompetenzbewusstsein (Proficiency Awareness) zu etablieren. Das primäre Ziel dieser Arbeit ist es, eine Methodik für das Design und die Implementierung von interaktiven Systemen aufzustellen, die Nutzer dabei unterstützen über ihre eigene Kompetenz zu reflektieren, um dadurch Lerneffekte implizit wahrnehmen können. Diese Arbeit stellt ein Konzept für fähigkeitsbewusste Systeme (proficiency-aware systems) vor, welche die Fähigkeiten von Nutzern abschätzen, die Interaktion entsprechend anpassen sowie das Bewusstsein der Nutzer über deren Fähigkeiten fördern. Hierzu sollten die Systeme gesammelte Daten von Nutzern einsetzen, um Lerneffekte sichtbar zu machen. Die Möglichkeit der Anwender zur Selbstreflexion ist hierbei als entscheidend anzusehen, um als Motivation zur Verbesserung der eigenen Fähigkeiten zu dienen. Eine zentrale Herausforderung solcher Systeme ist die Tatsache, dass Nutzer - im Vergleich zur Abschätzung des Systems - oft eine divergierende Selbstwahrnehmung ihrer Kompetenz haben. Im ersten Moment sind daher die Vorteile einer persönlichen Weiterentwicklung nicht unbedingt ersichtlich. Daher baut diese Forschungsarbeit nicht darauf auf, Nutzer über vorgegebene Lernstrategien zu unterrichten, sondern sie bedient sich der Möglichkeiten interaktiver Systeme, die Anwendern die notwendigen Hilfsmittel zur Verfügung stellen, damit diese selbst über ihre Fähigkeiten reflektieren können. Einem Zeitungseditor könnte beispielsweise die aktuelle Textschwierigkeit angezeigt werden, während einem passionierten Sportler dessen Muskelaktivität veranschaulicht wird. Zunächst wird herausgearbeitet, wie sich die Fähigkeiten der Nutzer mittels physiologischer Sensortechnologien erkennen und quantifizieren lassen. Die Evaluation von Interaktionsszenarien demonstriert die Umsetzbarkeit fähigkeitsbewusster Systeme, basierend auf der Analyse von Blickbewegungen und Muskelaktivität. Hierbei kommen Algorithmen des maschinellen Lernens zum Einsatz, die das Leistungsniveau der Anwender für verschiedene Tätigkeiten berechnen. Im Besonderen analysieren wir stationäre Aktivitäten, die hauptsächlich den Sehsinn ansprechen (Lesen, Aufnahme von Informationen), sowie dynamische Betätigungen, die die Motorik der Nutzer fordern (Spielen von Instrumenten, Fitnessübungen). Der zweite Teil zeigt auf, wie Systeme das Bewusstsein der Anwender für deren eigene Fähigkeiten fördern können, einschließlich der Designherausforderungen , wann und wie das System erkannte Fähigkeiten kommunizieren sollte. Abschließend wird die Notwendigkeit von Toolkits für Sensortechnologien hervorgehoben, um die Implementierung derartiger Systeme für ein breites Publikum zu ermöglichen. Die Forschungsarbeit beinhaltet eine Definition für fähigkeitsbewusste Systeme und veranschaulicht dieses Konzept durch den Entwurf und die Implementierung interaktiver Systeme. Ferner werden technische Anforderungen objektiver Echtzeitabschätzung von Nutzerfähigkeiten erforscht und Designqualitäten für die Kommunikation dieser Abschätzungen mittels Selbstreflexion identifiziert. Zusammengefasst sind die Erkenntnisse in einer Reihe von Design- und Entwicklungsrichtlinien für derartige Systeme. Insbesondere die Kommunikation, der vom System erkannten Kompetenz, hilft Anwendern, die eigene Leistung zu interpretieren

Adaptive dashboard for IoT environments: application for senior residences

Les tableaux de bord sont de puissants outils électroniques qui peuvent fournir des informations exploitables et utiles pour une intervention rapide et une prise de décision éclairée. Ils peuvent être particulièrement bénéfiques pour favoriser un vieillissement en bonne santé en fournissant aux professionnels de la santé un aperçu en un coup d'œil des conditions du patient (par exemple, les personnes âgées). Alors que la population des personnes âgées augmente dans plusieurs pays, dont le Canada, un grand nombre d'entre eux seront forcés de déménager dans des résidences pour personnes âgées pour des raisons telles que la fragilité, la démence ou le sentiment de solitude. Cette population importante de personnes âgées augmentera la charge de travail des infirmières et des professionnels de la santé travaillant dans ces lieux, en raison du fait que les personnes âgées ont besoin de visites fréquentes et d'une surveillance en raison de leur état de santé. Ce problème a le potentiel de mettre plus de pression sur le système de santé déjà tendu dans les prochaines années. La pénurie d'infirmières et de main-d'œuvre rend la situation plus grave, en particulier dans les pays développés. Il faudrait donc prendre des initiatives pour soutenir les soignants de ces résidences. Le tableau de bord peut jouer un rôle clé pour aider les professionnels de la santé dans leurs tâches car il peut fournir des informations en un coup d'œil et en temps réel sur la situation actuelle. De nos jours, avec les progrès technologiques dans les dispositifs de détection et l'infrastructure IoT ainsi qu'un accès Internet élargi, la surveillance des patients à distance est devenue une option réalisable. Par ailleurs, en utilisant un tableau de bord, les professionnels de la santé peuvent visualiser les informations collectées à distance pour surveiller les personnes âgées vivant dans des résidences, ce qui fera gagner un temps considérable aux professionnels de la santé et les aidera à servir plus de patients. Cependant, il est important de considérer que les résidences pour personnes âgées accueillent généralement un grand nombre de résidents et les professionnels de la santé qui les desservent. Chaque professionnel de la santé est motivé par certains objectifs et exécute des tâches précises selon des priorités différentes. Cette différence change la façon dont chaque fournisseur de soins de santé utilisera le tableau de bord, car ils ont besoin d'informations qui les aident dans leurs tâches principales. Les informations qu'un groupe de professionnels de la santé trouve bénéfiques peuvent ne pas être utiles pour un autre groupe. Ainsi, la méthode de visualisation utilisée pour un individu peut ne pas être significative pour un autre. Par conséquence, les informations doivent être présentées de manière personnalisée et adaptée à un utilisateur ciblé. Il est important de souligner que la visualisation appropriée des informations dans les tableaux de bord est un facteur clé pour offrir une valeur réelle aux utilisateurs. Cette diversité de besoins, de préférences et de priorités doit être prise en compte tout au long de l'élaboration du tableau de bord. En raison de la diversité des rôles et des intérêts existant dans les résidences pour personnes âgées, et compte tenu du coût élevé du développement du tableau de bord, il est très difficile de développer des tableaux de bord séparés pour chaque partie. Cependant, les solutions existantes dans la littérature sont développées à l'aide de méthodes statiques et se concentrent sur la satisfaction des besoins d'un groupe particulier. Ces approches limitent les capacités des tableaux de bord existants à s'adapter aux besoins des différentes personnes. Dans cette étude, nous présentons AMI-Dash comme une tentative de réalisation d'une solution de tableau de bord qui permet une conception dynamique et une visualisation appropriée des informations pour plusieurs groupes. Notre solution vise à fournir les bonnes informations aux bonnes personnes en minimisant le temps nécessaire pour fournir un tableau de bord aux professionnels la santé, afin de les aider dans l'exercice de leurs fonctions en accédant à des informations exploitables. Nous avons également évalué notre solution sous deux aspects : l'évaluation de l'interaction homme-machine et l'évaluation technique. Le résultat de notre évaluation montre que la solution proposée peut satisfaire à la fois les exigences de l'utilisateur final et les exigences techniques tout en maintenant un haut niveau de satisfaction.Abstract: Dashboards are powerful electronic tools that can provide actionable insights for timely intervention and wise decision-making. They can be particularly beneficial to support healthy aging by providing healthcare professionals with at-a-glance overview of health conditions of patients (e.g., older adults). As the population of older adults is increasing in several countries including Canada, a large number of them will be forced to move to Senior Residences due to reasons like frailty, dementia or loneliness. This swelled senior population will increase the workload of nurses and health professionals working in these places, due to the fact that older adults need frequent visits and monitoring because of their health condition. This issue has the potential to put more pressure on the already stretched healthcare system in the next years. The situation is aggravated when it is coincided with the shortage of nurses and workforce especially in developed countries. Therefore, initiative should be taken to support healthcare professionals in these residences. Dashboard can play a key role to support healthcare professionals in their tasks as it can provide real-time information about the current situation in more helpful visualization form. Nowadays, with technological advancements in sensing devices and IoT infrastructure along with broadened internet access, remote patient monitoring has become a feasible option. By utilizing a dashboard, healthcare professionals can visualize information collected remotely to monitor patients/ older adults living in senior residences, which will save a considerable time of healthcare professionals and support them to serve more patients. However, it is important to consider that senior residences usually host a large number of older adults and healthcare professionals that serve them. Each healthcare professional is driven with certain goals, and they have different tasks and priorities. This difference, change how each healthcare professional will utilize the dashboard, as they need information that helps them in their main tasks. The information that a group of healthcare professionals find beneficial might not be useful for another group, and the visualization method used for an individual might not be meaningful for another. Therefore, information should be presented in a personalized way to the targeted user. It is important to emphasize that appropriate visualization of interesting information, in dashboards is a key factor to deliver real value to dashboard users. Due to the variety of roles and interests that exists in senior residences, and considering high development cost of a dashboard, developing separate dashboards for each party is not only difficult but also time consuming. Still, existing solutions in the literature are developed using static methods and they focused on satisfying the needs of a particular group in their domain. These approaches limited the capabilities of existing dashboards to adapt to the needs of different people. We argue that dashboard has to be tailored in order to address the diversity in needs, preferences and priorities of healthcare professionals. In this study we introduce AMI-Dash as an attempt to achieve a dashboard solution that allows dynamic design and information visualization. Our solution focused on providing the right information to the right people while minimizing the time required to deliver a dashboard to health professionals, so that supporting them in performing their duties by accessing timely and actionable information. We also evaluated our proposed solution from two aspects: Human-Computer Interaction Evaluation and Technical Evaluation. The result of our evaluation shows that proposed solution can satisfy both end-user and technical requirements while maintaining a high-level of satisfaction among users

Automatic generation of software interfaces for supporting decisionmaking processes. An application of domain engineering & machine learning

[EN] Data analysis is a key process to foster knowledge generation in particular domains or fields of study. With a strong informative foundation derived from the analysis of collected data, decision-makers can make strategic choices with the aim of obtaining valuable benefits in their specific areas of action. However, given the steady growth of data volumes, data analysis needs to rely on powerful tools to enable knowledge extraction. Information dashboards offer a software solution to analyze large volumes of data visually to identify patterns and relations and make decisions according to the presented information. But decision-makers may have different goals and, consequently, different necessities regarding their dashboards. Moreover, the variety of data sources, structures, and domains can hamper the design and implementation of these tools. This Ph.D. Thesis tackles the challenge of improving the development process of information dashboards and data visualizations while enhancing their quality and features in terms of personalization, usability, and flexibility, among others. Several research activities have been carried out to support this thesis. First, a systematic literature mapping and review was performed to analyze different methodologies and solutions related to the automatic generation of tailored information dashboards. The outcomes of the review led to the selection of a modeldriven approach in combination with the software product line paradigm to deal with the automatic generation of information dashboards. In this context, a meta-model was developed following a domain engineering approach. This meta-model represents the skeleton of information dashboards and data visualizations through the abstraction of their components and features and has been the backbone of the subsequent generative pipeline of these tools. The meta-model and generative pipeline have been tested through their integration in different scenarios, both theoretical and practical. Regarding the theoretical dimension of the research, the meta-model has been successfully integrated with other meta-model to support knowledge generation in learning ecosystems, and as a framework to conceptualize and instantiate information dashboards in different domains. In terms of the practical applications, the focus has been put on how to transform the meta-model into an instance adapted to a specific context, and how to finally transform this later model into code, i.e., the final, functional product. These practical scenarios involved the automatic generation of dashboards in the context of a Ph.D. Programme, the application of Artificial Intelligence algorithms in the process, and the development of a graphical instantiation platform that combines the meta-model and the generative pipeline into a visual generation system. Finally, different case studies have been conducted in the employment and employability, health, and education domains. The number of applications of the meta-model in theoretical and practical dimensions and domains is also a result itself. Every outcome associated to this thesis is driven by the dashboard meta-model, which also proves its versatility and flexibility when it comes to conceptualize, generate, and capture knowledge related to dashboards and data visualizations

Dividing Complexity to Conquer New Dimensions – Towards a Framework for Designing Augmented Reality Solutions

Augmented reality (AR) can foster service innovation and thus cope with some of the most urgent challenges in the service science domain, namely supporting frontline workers while ensuring high safety standards. Therefore, the utilization of AR can help to achieve these goals. On the contrary, AR remains a complex technology with specific requirements and preconditions that demand expertise to overcome them. Based on a case study, we derive a framework for designing AR solutions, which helps divide the complexity of designing and developing AR-based services to support the adoption and diffusion of AR applications. Such an encompassing perspective on initial AR explorations helps to transform the acquired information into a thorough proof of concept, pilot implementations and ultimately productive software

Arigatō : effects of adaptive guidance on engagement and performances in augmented reality learning environments

Funding information: This research was supported by European Commission through the InnoRenew CoE project (Grant Agreement 739574) under the Horizon2020 Widespread-Teaming program and the Republic of Slovenia (investment funding of the Republic of Slovenia and the European Union of the European Regional Development Fund). We also acknowledge support from the Slovenian research agency ARRS (program no. BI-DE/20-21-002, P1-0383, J1-9186, J1-1715, J5-1796, and J1-1692).Experiential learning (ExL) is the process of learning through experience or more specifically “learning through reflection on doing”. In this paper, we propose a simulation of these experiences, in Augmented Reality (AR), addressing the problem of language learning. Such systems provide an excellent setting to support “adaptive guidance”, in a digital form, within a real environment. Adaptive guidance allows the instructions and learning content to be customised for the individual learner, thus creating a unique learning experience. We developed an adaptive guidance AR system for language learning, we call Arigato (Augmented Reality Instructional ¯ Guidance & Tailored Omniverse), which offers immediate assistance, resources specific to the learner's needs, manipulation of these resources, and relevant feedback. Considering guidance, we employ this prototype to investigate the effect of the amount of guidance (fixed vs. adaptive-amount) and the type of guidance (fixed vs. adaptive-associations) on the engagement and consequently the learning outcomes of language learning in an AR environment. The results for the amount of guidance show that compared to the adaptive-amount, the fixed-amount of guidance group scored better in the immediate and delayed (after 7 days) recall tests. However, this group also invested a significantly higher mental effort to complete the task. The results for the type of guidance show that the adaptive-associations group outperforms the fixed-associations group in the immediate, delayed (after 7 days) recall tests, and learning efficiency. The adaptive-associations group also showed significantly lower mental effort and spent less time to complete the task.PostprintPeer reviewe