Convo: What does conversational programming need? An exploration of machine learning interface design

Vast improvements in natural language understanding and speech recognition have paved the way for conversational interaction with computers. While conversational agents have often been used for short goal-oriented dialog, we know little about agents for developing computer programs. To explore the utility of natural language for programming, we conducted a study ($n$=45) comparing different input methods to a conversational programming system we developed. Participants completed novice and advanced tasks using voice-based, text-based, and voice-or-text-based systems. We found that users appreciated aspects of each system (e.g., voice-input efficiency, text-input precision) and that novice users were more optimistic about programming using voice-input than advanced users. Our results show that future conversational programming tools should be tailored to users' programming experience and allow users to choose their preferred input mode. To reduce cognitive load, future interfaces can incorporate visualizations and possess custom natural language understanding and speech recognition models for programming.Comment: 9 pages, 7 figures, submitted to VL/HCC 2020, for associated user study video: https://youtu.be/TC5P3OO5ex

Evaluating usability evaluation methods for location-­aware interactive systems in contextually rich environments.

PhDIn this research we investigate the evaluation of usability evaluations methods (UEMs). In particular we are concerned with evaluating their suitability for the evaluation of location-­‐ aware systems. Not all approaches for the evaluation of UEMs have been extensively validated for such types of dynamic interaction, while their application is not clearly documented. We overview the strengths of the current approach and suggest how to improve them. We examine navigation systems as examples for issues with location-­‐aware systems in a contextually rich environment. The setting is very different to a traditional desktop-­‐based application. Take the use of the navigation device for example. It is a secondary task; the primary task is to safely drive the car. The interface is continuously changing to adapt to the current location of the user. The user navigates in a complex dynamic environment encompassing various stimuli and unpredictable external factors. We present in the thesis a methodological and systematic way to approach the evaluation of UEMs. A comparative study of analytical and empirical techniques was carried out, to assess them in identifying usability problems within both static and dynamic contexts of use. Four analytical methods (CW, UAN, EMU, and Design Guidelines) and one empirical were compared. In this thesis, we validate the existing classification scheme of Blandford et al. (2008) and highlight relevant issues. We present an alternative systematic approach building on this scheme (CoHUM), and its shortcomings with dynamic systems. We show how a rigorous and systematic error analysis identifies phenotypes as the outcome of empirical techniques, whilst genotypes are the outcome of analytical techniques. Finally, we present new dimensions that previous literature had not identified for the evaluation of UEMs. This research will help future researchers by providing them with a stronger methodological approach for comparing UEMs and, in particular, categories of UEMs

Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

A Software Framework to Create 3D Browser-Based Speech Enabled Applications

The advances in automatic speech recognition have pushed the humancomputer interface researchers to adopt speech as one mean of input data. It is natural to humans, and complements very well other input interfaces. However, integrating an automatic speech recognizer into a complex system (such as a 3D visualization system or a Virtual Reality system) can be a difficult and time consuming task. In this paper we present our approach to the problem, a software framework requiringminimum additional coding from the application developer. The framework combines voice commands with existing interaction code, automating the task of creating a new speech grammar (to be used by the recognizer). A new listener component for theXj3D was created, which makes transparent to the user the integration between the 3D browser and the recognizer. We believe this is a desirable feature for virtual reality system developers, and also to be used as a rapid prototyping tool when experimenting with speech technology

Timbre hybridization processes and strategies. A Portfolio of Compositions

This document describes the processes and development of my compositional work, particularly concerning the introduction of modifications of timbral qualities, including combinations, and hybridization procedures. It describes compositional ethodologies, developed within a technological environment, and the interrelation between theoretical thought and computational approach. The following chapters present time, frequency, and timbre as materials of investigation, analysis, and re-composition, through real time electroacoustic strategies and treatments. The preparation and design of specific software, through the utilization of programming language Max/MSP Jitter, will illustrate the computational approach to composing, its inner correspondence with the theoretical approach, and interconnections with preparation and performing activity. Procedures progressively applied to the portfolio of compositions are presented in the final chapters of the document. The portfolio consists of six works completed during the last six years, for instruments and real time electronic treatment, presented as a CD with the complete recordings of three compositions, four scores, and a DVD, containing video recording of two works. The last three compositions presented are also part of a cycle of works –still in progress- dedicated to the whole instrumental spectrum, in which the voice represents the physical-musical material of each work