Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable

Requirements of API Documentation: A Case Study into Computer Vision Services

Using cloud-based computer vision services is gaining traction, where developers access AI-powered components through familiar RESTful APIs, not needing to orchestrate large training and inference infrastructures or curate/label training datasets. However, while these APIs seem familiar to use, their non-deterministic run-time behaviour and evolution is not adequately communicated to developers. Therefore, improving these services' API documentation is paramount-more extensive documentation facilitates the development process of intelligent software. In a prior study, we extracted 34 API documentation artefacts from 21 seminal works, devising a taxonomy of five key requirements to produce quality API documentation. We extend this study in two ways. Firstly, by surveying 104 developers of varying experience to understand what API documentation artefacts are of most value to practitioners. Secondly, identifying which of these highly-valued artefacts are or are not well-documented through a case study in the emerging computer vision service domain. We identify: (i) several gaps in the software engineering literature, where aspects of API documentation understanding is/is not extensively investigated; and (ii) where industry vendors (in contrast) document artefacts to better serve their end-developers. We provide a set of recommendations to enhance intelligent software documentation for both vendors and the wider research community.Comment: Early Access preprint for an upcoming issue of the IEEE Transactions on Software Engineerin

Fighting Against XSS Attacks. A Usability Evaluation of OWASP ESAPI Output Encoding

Cross Site Scripting (XSS) is one of the most critical vulnerabilities exist in web applications. XSS can be prevented by encoding untrusted data that are loaded into browser content of web applications. Security Application Programming Interfaces (APIs) such as OWASP ESAPI provide output encoding functionalities for programmers to use to protect their applications from XSS attacks. However, XSS still being ranked as one of the most critical vulnerabilities in web applications suggests that programmers are not effectively using those APIs to encode untrusted data. Therefore, we conducted an experimental study with 10 programmers where they attempted to fix XSS vulnerabilities of a web application using the output encoding functionality of OWASP ESAPI. Results revealed 3 types of mistakes that programmers made which resulted in them failing to fix the application by removing XSS vulnerabilities. We also identified 16 usability issues of OWASP ESAPI. We identified that some of these usability issues as the reason for mistakes that programmers made. Based on these results, we provided suggestions on how the usability of output encoding APIs should be improved to give a better experience to programmers

Interactive Machine Learning for User-Innovation Toolkits – An Action Design Research approach

Machine learning offers great potential to developers and end users in the creative industries. However, to better support creative software developers' needs and empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. This thesis asks the following research questions: How can we apply a user-centred approach to the design of developer tools for rapid prototyping with Interactive Machine Learning? In what ways can we design better developer tools to accelerate and broaden innovation with machine learning? This thesis presents a three-year longitudinal action research study that I undertook within a multi-institutional consortium leading the EU H2020 -funded Innovation Action RAPID-MIX. The scope of the research presented here was the application of a user-centred approach to the design and evaluation of developer tools for rapid prototyping and product development with machine learning. This thesis presents my work in collaboration with other members of RAPID-MIX, including design and deployment of a user-centred methodology for the project, interventions for gathering requirements with RAPID-MIX consortium stakeholders and end users, and prototyping, development and evaluation of a software development toolkit for interactive machine learning. This thesis contributes with new understanding about the consequences and implications of a user-centred approach to the design and evaluation of developer tools for rapid prototyping of interactive machine learning systems. This includes 1) new understanding about the goals, needs, expectations, and challenges facing creative machine-learning non-expert developers and 2) an evaluation of the usability and design trade-offs of a toolkit for rapid prototyping with interactive machine learning. This thesis also contributes with 3) a methods framework of User-Centred Design Actions for harmonising User-Centred Design with Action Research and supporting the collaboration between action researchers and practitioners working in rapid innovation actions, and 4) recommendations for applying Action Research and User-Centred Design in similar contexts and scale

Processo de Seleção de Participantes numa Plataforma de Testes de Usabilidade

With the exponential growth of web and mobile applications, the practice of usability tests began to be more prevalent in different organizations and revealed a positive impact on the empathy of their products with the intended users. Skipping this step can cause major usability issues on the final product, as it was not tested with its users before it was launched. Despite companies realizing the importance of user testing and starting to make use of it, this task requires participants who are willing to participate in the tests, and this aspect is often a major obstacle for UX (User eXperience) researchers, not only in terms of their recruitment but also the suitability of the participants’ attributes to the product to be created. The aim of this project is to develop a web application that unifies the steps present in carrying out usability tests, from their creation to their execution, with a special focus on the selection of participants for them, which is the main problem to be solved. The hypothesis of research here is to confirm that using other platforms' APIs (Application Programming Interface) to recruit testers for usability tests is an efficient way of finding testers from a wide variety of market niches, which we will confirm or deny by the end of the project. In the project's initial phase, it is intended to study the state of the art in order to bring to light deeper understandings of the UX field and usability testing, as well as current applications on the market for usability testing management and potential technologies for this project’s development. Then, possible solutions for approaching the problem will be described, and a value analysis will be carried out focusing on strategic and business concepts for the project, namely its value proposition. In this stage, it is intended to choose a solution for the problem, taking into account factors such as time, adequation, and complexity of each one. As a result of the previous steps, an initial concept for the UI (User Interface) of the chosen solution will be sketched, and its usability will be tested in order to find any issues and fix them before going on to the final design. After this phase, the solution will be implemented, and the technique for doing so, including the technology used, the code architecture and documentation, will be described. Accordingly, the project will be experimented and evaluated again after its implementation phase, in order to assess the effectiveness of its requirements’ execution and potential problems the testers may have run into, which we will have to analyze and consider fixing at a later stage. At last, thorough conclusions about the project will be held out, including those regarding the challenges and limitations faced, achieved objectives, and, lastly, the work to be developed in the future.Com o crescimento das aplicações web e mobile, a prática de testes de usabilidade começou a estar mais presente nas diversas organizações e revelou um impacto positivo na empatia dos seus produtos com os utilizadores a que se destinam. Saltar esta etapa pode causar grandes problemas de usabilidade no produto final, já que não foi testado com o público-alvo antes de ser lançado. Apesar das empresas perceberem a importância de testes de usabilidade e começarem a fazer uso deste método, estes testes requerem participantes que estejam dispostos a participarem nos mesmos (conhecidos como avaliadores ou, em inglês, testers), sendo este aspeto, muitas vezes, um grande obstáculo para os investigadores na área de UX (User eXperience), não só no que toca à sua procura, como também à adequação dos atributos destes ao produto a criar. O objetivo deste projeto é desenvolver uma aplicação web que unifique as etapas presentes na realização de testes de usabilidade, desde a sua criação à sua execução, com um foco especial na seleção de participantes para os mesmos, sendo este o problema a resolver. A aplicação terá as suas funcionalidades principais para criação e partilha dos testes, assim como seleção de participantes para os mesmos, em funcionamento, podendo, futuramente, ser desenvolvida na sua totalidade, isto é, com a presença de mais funcionalidades extra, úteis para a realização destes testes. A hipótese de investigação aqui é confirmar que usar APIs (Application Programming Interface) de outras plataformas para recrutar avaliadores para testes de usabilidade é uma forma eficiente de encontrar testers dos mais diversos nichos de mercado, o que iremos confirmar ou negar no final do projeto. Na fase inicial do projeto, pretende-se estudar o estado da arte de forma a obter conhecimentos mais aprofundados sobre a área de UX e testes de usabilidade (respondendose, para isso, a questões como: o que é UX Design, o que é usability testing e qual a sua importância; que tipos de testes de usabilidade existem; quando se conduzem; quantos participantes são necessários; entre outras questões), bem como perceber as aplicações atualmente existentes no mercado para gestão destes testes (percebendo quem é a nossa concorrência e que funcionalidades é que oferecem nas suas aplicações, identificando-se também a nossa oportunidade), e, por fim, potenciais tecnologias para o desenvolvimento deste projeto (incluindo possíveis APIs de aplicações externas que poderão ser usadas para procura de avaliadores). Em seguida, serão pleaneadas possíveis soluções para a abordagem do problema, incluindo-se uma solução completa (que não será implementada neste projeto devido à sua complexidade e tempo que exige), e três outras soluções simplificadas possíveis de serem implementadas, permitindo, caso haja algum imprevisto na fase de desenvolvimento que impossibilite a implementação da solução escolhida a tempo, a implementação de uma outra solução aqui especificada, visto que estas variam no número de requisitos que exigem. Será ainda realizada uma análise de valor com foco nos conceitos estratégicos e de negócio do projeto, nomeadamente a oportunidade, uma análise SWOT (Strengths, Weaknesses, Oppurtunities, Threats), proposta de valor, modelo de negócio, e os requisitos a ter em conta para satisfazer os desejos dos clientes, através de uma técnica denominada Quality Function Deployment (QFD). Nesta etapa, pretende-se também escolher uma solução para o problema, levando em consideração fatores como tempo, adequação e complexidade de cada uma. Para isso, será utilizado o método de Analytic Hierarchy Process (AHP). Como resultado das etapas anteriores, será desenhada a solução para o problema. Em primeiro lugar, serão definidos os tipos de utilizador que poderão registar-se na aplicação, esboçando também um mapa de navegação distinto para cada um destes. Em segundo lugar, será esboçado um conceito inicial para o UI (User Interface) da solução escolhida, onde será depois conduzido um teste de usabilidade – System Usability Scale (SUS) – para que se possam identificar e corrigir problemas antes da sua versão final. Na versão final do design será também construída uma identidade visual, escolhendo-se um nome para a aplicação e desenhando-se o seu logótipo, definindo-se também a paleta de cores e tipografia a utilizar, mantendo coerência ao longo das páginas da aplicação. Após esta fase, a solução será implementada: será escolhida e fundamentada a tecnologia a utilizar; será descrito o processo de implementação, nomeadamente a definição de requisitos e a utilização de um sistema de controlo de versões; será explicada a arquitetura do código e a sua respetiva documentação, com foco em casos especiais do projeto; e, por último, será disponibilizada a aplicação final. Assim, o projeto será novamente experimentado e avaliado após a sua fase de implementação, de forma a classificar-se o sucesso da implementação dos seus requisitos e potenciais problemas que os testers possam ter encontrado, os quais terão de ser analisados e ponderados para corrigir numa fase posterior. Para isto, será utilizado um QEF (Quantitative Evaluation Framework) e um formulário de feedback, por forma a obtermos uma avaliação quantitativa e qualitativa da aplicação desenvolvida. Por último, serão tiradas conclusões aprofundadas sobre o projeto, incluindo as relativas aos desafios e limitações enfrentados, objetivos alcançados (focando-nos também em justificar a hipótese apresentada) e, por último, o trabalho a desenvolver-se no futuro

Automated Driver Management for Selenium WebDriver

Selenium WebDriver is a framework used to control web browsers automatically. It provides a cross-browser Application Programming Interface (API) for different languages (e.g., Java, Python, or JavaScript) that allows automatic navigation, user impersonation, and verification of web applications. Internally, Selenium WebDriver makes use of the native automation support of each browser. Hence, a platform-dependent binary file (the so-called driver) must be placed between the Selenium WebDriver script and the browser to support this native communication. The management (i.e., download, setup, and maintenance) of these drivers is cumbersome for practitioners. This paper provides a complete methodology to automate this management process. Particularly, we present WebDriverManager, the reference tool implementing this methodology. WebDriverManager provides different execution methods: as a Java dependency, as a Command-Line Interface (CLI) tool, as a server, as a Docker container, and as a Java agent. To provide empirical validation of the proposed approach, we surveyed the WebDriverManager users. The aim of this study is twofold. First, we assessed the extent to which WebDriverManager is adopted and used. Second, we evaluated the WebDriverManager API following Clarke’s usability dimensions. A total of 148 participants worldwide completed this survey in 2020. The results show a remarkable assessment of the automation capabilities and API usability of WebDriverManager by Java users, but a scarce adoption for other languages.This work has been been supported in part by the "Análisis en tiempo Real de sensores sociALes y EStimación de recursos para transporte multimodal basada en aprendizaje profundo" project (MaGIST-RALES), funded by the Spanish Agencia Estatal de Investigación (AEI, doi 10.13039/501100011033) under grant PID2019-105221RB-C44. This work also received partial support from FEDER/Ministerio de Ciencia, Innovación y Universidades - Agencia Estatal de Investigación through project Smartlet (TIN2017-85179-C3-1-R), and from the eMadrid Network, which is funded by the Madrid Regional Government (Comunidad de Madrid) with grant No. S2018/TCS-4307

Cognitive dimensions usability assessment of textual and visual VHDL environments

Visual programming languages promise to make programming easier with simpler graphical methods, broadening access to computing by lessening the need for would-be users to become proficient with textual programming languages, with their somewhat arcane grammars and methods removed from the problem space of the user. However, after more than forty years of research in the field, visual methods remain in the margins of use and programming remains the bailiwick of people devoted to the endeavor. VPL designers need to understand the mechanisms of usability that pertain to complex systems like programming language environments. Effective research tools for studying usability, and sufficiently constrained, mature subjects for investigation are scarce. This study applies a usability research tool, with its origins in applied psychology, to a programming language surrogate from the hardware description language class of notations. The substitution is reasonable because of the great similarity between hardware description languages and programming languages. Considering VHDL (the VHSIC Hardware Description Language) is especially worthwhile for several reasons, but primarily because significant numbers of digital designers regularly employ both textual and visual VHDL environments to meet the same real-world design challenges. A comparative analysis of Cognitive Dimensions assessments of textual and visual VHDL environments should further understanding of the usability issues specifically related to visual methods – in many cases, the same visual methods used in visual programming languages. Furthermore, with this real-world ‘field lab’ better understood, it should be possible to design experiments to pursue the formalization of the CDs framework as a theory

SDK development for bridging heterogeneous data sources through connect bridge platform

Nesta dissertação apresentou-se um SDK para a criação de conectores a integrar com o CB Server, que pretende: acelerar o desenvolvimento, garantir melhores práticas e simplificar as diversas atividades e tarefas no processo de desenvolvimento. O SDK fornece uma API pública e simples, suportada por um conjunto de ferramentas, que facilitam o processo de desenvolvimento, explorando as facilidades disponibilizadas através da API. Para analisar a exatidão, viabilidade, integridade e acessibilidade da solução apresentam-se dois exemplos e casos de estudo. Através dos casos de estudo foi possível identificar uma lista de problemas, de pontos sensíveis e melhorias na solução proposta. Para avaliar a usabilidade da API, uma metodologia baseada em vários métodos de avaliação de usabilidade foi estabelecida. O múltiplo caso de estudo funciona como o principal método de avaliação, combinando vários métodos de pesquisa. O caso de estudo consiste em três fases de avaliação: um workshop, uma avaliação heurística e uma análise subjetiva. O caso de estudo envolveu três engenheiros de software (incluindo programadores e avaliadores). A metodologia aplicada gerou resultados com base num método de inspeção, testes de utilizador e entrevistas. Identificou-se não só pontos sensíveis e falhas no código-fonte, mas também problemas estruturais, de documentação e em tempo de execução, bem como problemas relacionados com a experiência do utilizador. O contexto do estudo é apresentado de modo a tirar conclusões acerca dos resultados obtidos. O trabalho futuro incluirá o desenvolvimento de novas funcionalidades. Adicionalmente, pretende-se resolver problemas encontrados na metodologia aplicada para avaliar a usabilidade da API, nomeadamente problemas e falhas no código fonte (por exemplo, validações) e problemas estruturais.In this dissertation, we present an SDK for the creation of connectors to integrate with CB Server which accelerates deployment, ensures best practices and simplifies the various activities and tasks in the development process. The SDK provides a public and simple API leveraged by a set of tools around the API developed which facilitate the development process by exploiting the API facilities. To analyse the correctness, feasibility, completeness, and accessibility of our solution, we presented two examples and case studies. From the case studies, we derived a list of issues found in our solution and a set of proposals for improvement. To evaluate the usability of the API, a methodology based on several usability evaluation methods has been established. Multiple case study works as the main evaluation method, combining several research methods. The case study consists of three evaluation phases – a hands-on workshop, a heuristic evaluation and subjective analysis. The case study involved three computer science engineers (including novice and expert developers and evaluators). The applied methodology generated insights based on an inspection method, a user test, and interviews. We identify not only problems and flaws in the source code, but also runtime, structural and documentation problems, as well as problems related to user experience. To help us draw conclusion from the results, we point out the context of the study. Future work will include the development of new functionalities. Additionally, we aim to solve problems found in the applied methodology to evaluate the usability of the API, namely problems and flaws in the source code (e.g. validations) and structural problems

Measuring Developer Experience of a Digital Platform

Smart city and smart transportation are concepts that have emerged as an enabling solution which facilitates the grassroots social innovations to mitigate the problems generated by rapid urbanization and population growth. The digital service platform has fostered a new paradigm of transportation by involving all key players to create a novel environment. It is concerned developer are also the user of the platform as they are using the system development tools and methods for further development, that is why developer experience over the platform plays a vital role. Delightful developer experience not only improving the platform performance but also invokes to introduce new innovations. In this research we off to measure developer experience and answering the research questions “how to measure developer experience on top of the digital service platform” and “how to analyse the developer experience”. In the state of measuring developer experience, an application has been developed over the digital service platform and a measurement procedure has been introduced by modifying System Usability Scale (SUS) to more suit the context of the developer. The SUS has been borrowed from UX measurement tools as developers are the user of system, system development tools and methods as well as SUS is a widely accepted tool by the usability researchers for measuring usability. The result of the proposed method showed superior experience from the developer’s perspective to develop the application over the living lab bus platform. The result is almost same when it is compared with another method, but it is arguable as it showed small discrepancy. Furthermore, it can be said that, this research provides a straight forward way to measure developer experience on a digital service platform. The answer of the research questions provides a detail guideline of the measurement process and analysing criteria of developer experience. Moreover, it comes out with few recommendations that can be helpful for the developers of the platform to improve the platform in future, so that it could ensure the delightful experience for the developers