Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

Five-Factor Model as a Predictor for Spoken Dialog Systems

Human behavior varies widely as does the design of spoken dialog systems (SDS). The search for predictors to match a user’s preference and efficiency for a specific dialog interface type in an SDS was the focus of this research. By using personality as described by the Five-Factor Method (FFM) and the Wizard of Oz technique for delivering three system initiatives of the SDS, participants interacted with each of the SDS initiatives in scheduling an airline flight. The three system initiatives were constructed as strict system, which did not allow the user control of the interaction; mixed system, which allowed the user some control of the interaction but with a system override; and user system, which allowed the user control of the interaction. In order to eliminate gender bias in using the FFM as the instrument, participants were matched in gender and age. Participants were 18 years old to 70 years old, passed a hearing test, had no disability that prohibited the use of the SDS, and were native English speakers. Participants completed an adult consent form, a 50-question personality assessment as described by the FFM, and the interaction with the SDS. Participants also completed a system preference indication form at the end of the interaction. Observations for efficiency were recorded on paper by the researcher. Although the findings did not show a definitive predictor for a SDS due to the small population sample, by using a multinomial regression approach to the statistical analysis, odds ratios of the data helped draw conclusions that support certain personality factors as important roles in a user’s preference and efficiency in choosing and using a SDS. This gives an area for future research. Also, the presumption that preference and efficiency always match was not supported by the results from two of the three systems. An additional area for future research was discovered in the gender data. Although not an initial part of the research, the data shows promise in predicting preference and efficiency for certain SDS. Future research is indicated

Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards

Wizard of Oz (WoZ) as a prototyping method has been used to simulate intelligent user interfaces, particularly for speech-based systems. However, as our societies' expectations on artificial intelligence (AI) grows, the question remains whether a single Wizard is sufficient for it to simulate smarter systems and more complex interactions. Optimistic visions of 'what artificial intelligence (AI) can do' places demands on WoZ platforms to simulate smarter systems and more complex interactions. This raises the question of whether the typical approach of employing a single Wizard is sufficient. Moreover, while existing work has employed multiple Wizards in WoZ studies, a multi-Wizard approach has not been systematically studied in terms of feasibility, effectiveness, and challenges. We offer Wizundry, a real-time, web-based WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. We outline the design and technical specifications of our open-source platform, which we iterated over two design phases. We report on two studies in which participant-Wizards were tasked with negotiating how to cooperatively simulate an interface that can handle natural speech for dictation and text editing as well as other intelligent text processing tasks. We offer qualitative findings on the Multi-Wizard experience for Dyads and Triads of Wizards. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.Comment: 34 page

Implementing a Wizard of Oz Tool for Augmented Reality

This thesis aims to explore Wizard of Oz testing in conjunction with Augmented Reality (AR) and focus has been put on testing AR with Head Mounted Displays. The recent increase of interest in HMDs with products such as MOD Live from Recon Instruments and Google's Project Glass puts new demands and possibilities on human-computer interaction. Since the commercial market for HMDs is still in its infancy the need to explore different design approaches is very much present. One way to conduct experiments on human-machine interaction is with the help of a Wizard of Oz tool. During the thesis we have developed such a tool to support designers in researching usability and interaction. The tool provides a user friendly framework to carry out user case studies focused on AR with HMDs. After input and feedback from stakeholders and experts we believe that, even though the tool is mainly meant to be used in conjunction with AR in HMDs, the tool can be applied to other areas as well

Semi-aural Interfaces: Investigating Voice-controlled Aural Flows

To support mobile, eyes-free web browsing, users can listen to ‘playlists’ of web content— aural flows . Interacting with aural flows, however, requires users to select interface buttons, tethering visual attention to the mobile device even when it is unsafe (e.g. while walking). This research extends the interaction with aural flows through simulated voice commands as a way to reduce visual interaction. This paper presents the findings of a study with 20 participants who browsed aural flows either through a visual interface only or by augmenting it with voice commands. Results suggest that using voice commands reduced the time spent looking at the device by half but yielded similar system usability and cognitive effort ratings as using buttons. Overall, the low-cognitive effort engendered by aural flows, regardless of the interaction modality, allowed participants to do more non-instructed (e.g. looking at the surrounding environment) than instructed activities (e.g. focusing on the user interface)

Applying a User-centred Approach to Interactive Visualization Design

Analysing users in their context of work and finding out how and why they use different information resources is essential to provide interactive visualisation systems that match their goals and needs. Designers should actively involve the intended users throughout the whole process. This chapter presents a user-centered approach for the design of interactive visualisation systems. We describe three phases of the iterative visualisation design process: the early envisioning phase, the global specification hase, and the detailed specification phase. The whole design cycle is repeated until some criterion of success is reached. We discuss different techniques for the analysis of users, their tasks and domain. Subsequently, the design of prototypes and evaluation methods in visualisation practice are presented. Finally, we discuss the practical challenges in design and evaluation of collaborative visualisation environments. Our own case studies and those of others are used throughout the whole chapter to illustrate various approaches

Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities

Application of Machine Learning within Visual Content Production

We are living in an era where digital content is being produced at a dazzling pace. The heterogeneity of contents and contexts is so varied that a numerous amount of applications have been created to respond to people and market demands. The visual content production pipeline is the generalisation of the process that allows a content editor to create and evaluate their product, such as a video, an image, a 3D model, etc. Such data is then displayed on one or more devices such as TVs, PC monitors, virtual reality head-mounted displays, tablets, mobiles, or even smartwatches. Content creation can be simple as clicking a button to film a video and then share it into a social network, or complex as managing a dense user interface full of parameters by using keyboard and mouse to generate a realistic 3D model for a VR game. In this second example, such sophistication results in a steep learning curve for beginner-level users. In contrast, expert users regularly need to refine their skills via expensive lessons, time-consuming tutorials, or experience. Thus, user interaction plays an essential role in the diffusion of content creation software, primarily when it is targeted to untrained people. In particular, with the fast spread of virtual reality devices into the consumer market, new opportunities for designing reliable and intuitive interfaces have been created. Such new interactions need to take a step beyond the point and click interaction typical of the 2D desktop environment. The interactions need to be smart, intuitive and reliable, to interpret 3D gestures and therefore, more accurate algorithms are needed to recognise patterns. In recent years, machine learning and in particular deep learning have achieved outstanding results in many branches of computer science, such as computer graphics and human-computer interface, outperforming algorithms that were considered state of the art, however, there are only fleeting efforts to translate this into virtual reality. In this thesis, we seek to apply and take advantage of deep learning models to two different content production pipeline areas embracing the following subjects of interest: advanced methods for user interaction and visual quality assessment. First, we focus on 3D sketching to retrieve models from an extensive database of complex geometries and textures, while the user is immersed in a virtual environment. We explore both 2D and 3D strokes as tools for model retrieval in VR. Therefore, we implement a novel system for improving accuracy in searching for a 3D model. We contribute an efficient method to describe models through 3D sketch via an iterative descriptor generation, focusing both on accuracy and user experience. To evaluate it, we design a user study to compare different interactions for sketch generation. Second, we explore the combination of sketch input and vocal description to correct and fine-tune the search for 3D models in a database containing fine-grained variation. We analyse sketch and speech queries, identifying a way to incorporate both of them into our system's interaction loop. Third, in the context of the visual content production pipeline, we present a detailed study of visual metrics. We propose a novel method for detecting rendering-based artefacts in images. It exploits analogous deep learning algorithms used when extracting features from sketches