Entertaining and Opinionated but Too Controlling: A Large-Scale User Study of an Open Domain Alexa Prize System

Conversational systems typically focus on functional tasks such as scheduling appointments or creating todo lists. Instead we design and evaluate SlugBot (SB), one of 8 semifinalists in the 2018 AlexaPrize, whose goal is to support casual open-domain social inter-action. This novel application requires both broad topic coverage and engaging interactive skills. We developed a new technical approach to meet this demanding situation by crowd-sourcing novel content and introducing playful conversational strategies based on storytelling and games. We collected over 10,000 conversations during August 2018 as part of the Alexa Prize competition. We also conducted an in-lab follow-up qualitative evaluation. Over-all users found SB moderately engaging; conversations averaged 3.6 minutes and involved 26 user turns. However, users reacted very differently to different conversation subtypes. Storytelling and games were evaluated positively; these were seen as entertaining with predictable interactive structure. They also led users to impute personality and intelligence to SB. In contrast, search and general Chit-Chat induced coverage problems; here users found it hard to infer what topics SB could understand, with these conversations seen as being too system-driven. Theoretical and design implications suggest a move away from conversational systems that simply provide factual information. Future systems should be designed to have their own opinions with personal stories to share, and SB provides an example of how we might achieve this.Comment: To appear in 1st International Conference on Conversational User Interfaces (CUI 2019

Intelligent techniques for context-aware systems

Nowadays, with advances in communication technologies, researches are focused in the fields of designing new devices with increasing capabilities, implanting software frameworks or middleware to make these devices interoperable. Building better human interfaces is a challenging task and the adoption of Artificial Intelligence (AI) techniques to the process help associating semantic meaning to devices which makes possible the gesture recognition and voice recognition. This thesis is mainly concerned with the open problem in context-aware systems: the evaluation of these systems in Ambient Intelligence (AmI) environments. With regard to this issue, we argue that due to highly dynamic properties of the AmI environments, it should exist a methodology for evaluating these systems taking into account the type of scenarios. However in order to support with a solid ground for that discussion, some elements are to be discussed as well. In particular, we: • use a commercial platform that allows us to design and manage the contextual information of context- aware systems by means of a context manager included in the architecture; • analyze the formal representation of this contextual information by means of a knowledge based system (KBS); • discuss the possible methodologies to be used for modelling knowledge in KBS and our approach; • give reasons why intelligent agents is a valid technique to be applied to systems in AmI environments; • propose a generic multi-agent system (MAS) architecture that can be applied to a large class of envisaged AmI applications; • propose a multimodal user interface and its integration with our MAS; • propose an evaluation methodology for context-aware systems in AmI scenarios. The formulation of the above mentioned elements became necessary as this thesis was developed. The lack of an evaluation methodology for context-aware systems in AmI environments, where so many issues to be covered, took us to the main objective of this thesis. In this regard: • we provide an updated and exhaustive state-of-the-art of this matter; • examine the properties and characteristics of AmI scenarios; • put forward an evaluation methodology and experimentally test our methodology in AmI scenarios. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------La Inteligencia Ambiental y los entornos inteligentes hacen hincapié en una mayor facilidad de uso, soporte de servicios más eficientes, el apoderamiento de los usuarios, y el apoyo a las interacciones humanas. En esta visión, las personas estarán rodeadas de interfaces inteligentes e intuitivas incrustados en objetos cotidianos que nos rodean y los sistemas desarrollados para este ambiente deberán reconocer y responder a la presencia de individuos de una manera invisible y transparente a ellos. Esta tesis se centra principalmente en el problema abierto en los sistemas sensibles al contexto: la evaluación de estos sistemas en los entornos de Inteligencia Ambiental. Con respecto a este tema, se argumenta que debido a las propiedades altamente dinámica de los entornos de inteligencia ambiental, debería existir una metodología para la evaluación de estos sistemas, teniendo en cuenta el tipo de escenarios. Sin embargo, con el fin de apoyar con una base sólida para la discusión, algunos elementos deben ser discutidos también. En particular, nosotros: • Usamos una plataforma comercial que nos permite diseñar y gestionar la información contextual de los sistemas sensibles al contexto a través de un gestor de contexto incluido en la arquitectura; • Analizamos la representación formal de esta información contextual a través de un sistema basado en el conocimiento (SBC); • Discutimos las posibles metodologías que se utilizarán para el modelado del conocimiento en SBC y nuestra aproximación y propuesta; • Discutimos las razones del por qué los agentes inteligentes son una técnica válida para ser aplicada a los sistemas en entornos inteligencia ambiental; • Proponemos un sistema multi-agente (SMA), con una arquitectura genérica que se puede aplicar a una gran clase de aplicaciones de inteligencia ambiental; • Proponemos una interfaz de usuario multimodales y su integración con nuestro SMA; • Proponemos una metodología de evaluación de los sistemas sensibles al contexto en los escenarios de inteligencia ambiental. La formulación de los elementos antes mencionados se hizo necesaria en la medida que esta tesis se ha desarrollado. La falta de una metodología de evaluación de los sistemas sensibles al contexto en entornos de inteligencia ambiental, donde existen tantos temas a tratar, nos llevó al objetivo principal de esta tesis. En este sentido, en esta tesis: • Proporcionamos un estado del arte actualizado y exhaustivo de este asunto; • Examinamos las propiedades y características de los escenarios de inteligencia ambiental; • Proponemos una metodología de evaluación para este tipo de sistemas y experimentalmente probamos nuestra metodología en diversos escenarios de inteligencia ambiental

Evaluating Spoken Language Interaction

To study the spoken language interface in the context of a complex problem-solving task, a group of users were asked to perform a spreadsheet task, alternating voice and keyboard input. A total of 40 tasks were performed by each participant, the first thirty in a group (over several days), the remaining ones a month later. The voice spreadsheet program used in this study was extensively instrumented to provide detailed information about the components of the interaction. These data, as well as analysis of the participants's utterances and recognizer output, provide a fairly detailed picture of spoken language interaction. Although task completion by voice took longer than by keyboard, analysis shows that users would be able to perform the spreadsheet task faster by voice, if two key criteria could be met: recognition occurs in real-time, and the error rate is sufficiently low. This initial experience with a spoken language system also allows us to identify several metrics, beyond those traditionally associated with speech recognition, that can be used to characterize system performance