Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time

Crowd-powered conversational assistants have been shown to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. A promising direction is to combine the two approaches for high quality, low latency, and low cost solutions. In this paper, we introduce Evorus, a crowd-powered conversational assistant built to automate itself over time by (i) allowing new chatbots to be easily integrated to automate more scenarios, (ii) reusing prior crowd answers, and (iii) learning to automatically approve response candidates. Our 5-month-long deployment with 80 participants and 281 conversations shows that Evorus can automate itself without compromising conversation quality. Crowd-AI architectures have long been proposed as a way to reduce cost and latency for crowd-powered systems; Evorus demonstrates how automation can be introduced successfully in a deployed system. Its architecture allows future researchers to make further innovation on the underlying automated components in the context of a deployed open domain dialog system.Comment: 10 pages. To appear in the Proceedings of the Conference on Human Factors in Computing Systems 2018 (CHI'18

Democratizing Chatbot Debugging: A Computational Framework for Evaluating and Explaining Inappropriate Chatbot Responses

Evaluating and understanding the inappropriateness of chatbot behaviors can be challenging, particularly for chatbot designers without technical backgrounds. To democratize the debugging process of chatbot misbehaviors for non-technical designers, we propose a framework that leverages dialogue act (DA) modeling to automate the evaluation and explanation of chatbot response inappropriateness. The framework first produces characterizations of context-aware DAs based on discourse analysis theory and real-world human-chatbot transcripts. It then automatically extracts features to identify the appropriateness level of a response and can explain the causes of the inappropriate response by examining the DA mismatch between the response and its conversational context. Using interview chatbots as a testbed, our framework achieves comparable classification accuracy with higher explainability and fewer computational resources than the deep learning baseline, making it the first step in utilizing DAs for chatbot response appropriateness evaluation and explanation.Comment: 7 pages, 4 figures, accepted to CUI 2023 poster trac

ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles

Large language model (LLM) prompting is a promising new approach for users to create and customize their own chatbots. However, current methods for steering a chatbot's outputs, such as prompt engineering and fine-tuning, do not support users in converting their natural feedback on the model's outputs to changes in the prompt or model. In this work, we explore how to enable users to interactively refine model outputs through their feedback, by helping them convert their feedback into a set of principles (i.e. a constitution) that dictate the model's behavior. From a formative study, we (1) found that users needed support converting their feedback into principles for the chatbot and (2) classified the different principle types desired by users. Inspired by these findings, we developed ConstitutionMaker, an interactive tool for converting user feedback into principles, to steer LLM-based chatbots. With ConstitutionMaker, users can provide either positive or negative feedback in natural language, select auto-generated feedback, or rewrite the chatbot's response; each mode of feedback automatically generates a principle that is inserted into the chatbot's prompt. In a user study with 14 participants, we compare ConstitutionMaker to an ablated version, where users write their own principles. With ConstitutionMaker, participants felt that their principles could better guide the chatbot, that they could more easily convert their feedback into principles, and that they could write principles more efficiently, with less mental demand. ConstitutionMaker helped users identify ways to improve the chatbot, formulate their intuitive responses to the model into feedback, and convert this feedback into specific and clear principles. Together, these findings inform future tools that support the interactive critiquing of LLM outputs

Rule-based lip-syncing algorithm for virtual character in voice chatbot

Virtual characters changed the way we interact with computers. The underlying key for a believable virtual character is accurate synchronization between the visual (lip movements) and the audio (speech) in real-time. This work develops a 3D model for the virtual character and implements the rule-based lip-syncing algorithm for the virtual character's lip movements. We use the Jacob voice chatbot as the platform for the design and implementation of the virtual character. Thus, audio-driven articulation and manual mapping methods are considered suitable for real-time applications such as Jacob. We evaluate the proposed virtual character using hedonic motivation system adoption model (HMSAM) with 70 users. The HMSAM results for the behavioral intention to use is 91.74%, and the immersion is 72.95%. The average score for all aspects of the HMSAM is 85.50%. The rule-based lip-syncing algorithm accurately synchronizes the lip movements with the Jacob voice chatbot's speech in real-time

Artificial Intelligence Agents and Knowledge Acquisition in Health Information System

This research work highlights the need for AI-powered applications and their usages for theoptimization of information flow processes in the medical sector, from the perspective of howAI-agents can impact human-machine interaction (HCI) for acquiring relevant and necessaryinformation in emergency department (ED). This study investigates how AI-agents can be applied to manage situations of patient related unexpected experiences, such as long waiting times,overcrowding issues, and high number of patients leaving without being diagnosed. For knowledge acquisition, we incorporated modelling workshop techniques for gathering domain information from the domain experts in the context of emergency department in Karolinska Hospi-tal, Solna, Stockholm, Sweden, and for designing the AI-agent utilizing NLP techniques. We dis-cuss how the proposed solution can be used as an assistant to healthcare practitioners and workers to improve medical assistance in various medical procedures to increase flow and to reduce workloads and anxiety levels. The implementation part of this work is based on the natural language processing (NLP) techniques that help to develop the intelligent behavior for information acquisition and itsretriev-al in a natural way to support patients/relatives’ communication with the healthcare organization efficiently and in a natural way

Chatbot for training and assisting operators in inspecting containers in seaports

The paper presents the chatbot applicability for the health and safety of workers in the container transportation context. Starting from a literature review of risks and hazardous activities in sea container terminals, the paper underlines the need of innovative systems to ensure the lowest level of risks for labours. An analysis of the 4.0 technologies solutions in sea container terminals shows the lack of empirical application of chatbots in such a context. Focus is given to the current chatbot applications, and on the conceptual methodology for the chatbot design, defining five models and presenting a taxonomy for the chatbot feature definition. A case study shows the possible application of the conceptual methodology and the taxonomy, introducing the Popeye chatbot, consisting of a voice service, spoken language understanding component and an image processing app, to cope with the hazards in the process of examining freight and containers in dock areas. The main application of Popeye is the training of new employees involved in container safety-critical quality inspection and controls operations