I Probe, Therefore I Am: Designing a Virtual Journalist with Human Emotions

By utilizing different communication channels, such as verbal language, gestures or facial expressions, virtually embodied interactive humans hold a unique potential to bridge the gap between human-computer interaction and actual interhuman communication. The use of virtual humans is consequently becoming increasingly popular in a wide range of areas where such a natural communication might be beneficial, including entertainment, education, mental health research and beyond. Behind this development lies a series of technological advances in a multitude of disciplines, most notably natural language processing, computer vision, and speech synthesis. In this paper we discuss a Virtual Human Journalist, a project employing a number of novel solutions from these disciplines with the goal to demonstrate their viability by producing a humanoid conversational agent capable of naturally eliciting and reacting to information from a human user. A set of qualitative and quantitative evaluation sessions demonstrated the technical feasibility of the system whilst uncovering a number of deficits in its capacity to engage users in a way that would be perceived as natural and emotionally engaging. We argue that naturalness should not always be seen as a desirable goal and suggest that deliberately suppressing the naturalness of virtual human interactions, such as by altering its personality cues, might in some cases yield more desirable results.Comment: eNTERFACE16 proceeding

Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time

Crowd-powered conversational assistants have been shown to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. A promising direction is to combine the two approaches for high quality, low latency, and low cost solutions. In this paper, we introduce Evorus, a crowd-powered conversational assistant built to automate itself over time by (i) allowing new chatbots to be easily integrated to automate more scenarios, (ii) reusing prior crowd answers, and (iii) learning to automatically approve response candidates. Our 5-month-long deployment with 80 participants and 281 conversations shows that Evorus can automate itself without compromising conversation quality. Crowd-AI architectures have long been proposed as a way to reduce cost and latency for crowd-powered systems; Evorus demonstrates how automation can be introduced successfully in a deployed system. Its architecture allows future researchers to make further innovation on the underlying automated components in the context of a deployed open domain dialog system.Comment: 10 pages. To appear in the Proceedings of the Conference on Human Factors in Computing Systems 2018 (CHI'18

ZOE: A cloud-less dialog-enabled continuous sensing wearable exploiting heterogeneous computation

The wearable revolution, as a mass-market phenomenon, has finally arrived. As a result, the question of how wearables should evolve over the next 5 to 10 years is assuming an increasing level of societal and commercial importance. A range of open design and system questions are emerging, for instance: How can wearables shift from being largely health and fitness focused to tracking a wider range of life events? What will become the dominant methods through which users interact with wearables and consume the data collected? Are wearables destined to be cloud and/or smartphone dependent for their operation? Towards building the critical mass of understanding and experience necessary to tackle such questions, we have designed and implemented ZOE – a match-box sized (49g) collar- or lapel-worn sensor that pushes the boundary of wearables in an important set of new directions. First, ZOE aims to perform multiple deep sensor inferences that span key aspects of everyday life (viz. personal, social and place information) on continuously sensed data; while also offering this data not only within conventional analytics but also through a speech dialog system that is able to answer impromptu casual questions from users. (Am I more stressed this week than normal?) Crucially, and unlike other rich-sensing or dialog supporting wearables, ZOE achieves this without cloud or smartphone support – this has important side-effects for privacy since all user information can remain on the device. Second, ZOE incorporates the latest innovations in system-on-a-chip technology together with a custom daughter-board to realize a three-tier low-power processor hierarchy. We pair this hardware design with software techniques that manage system latency while still allowing ZOE to remain energy efficient (with a typical lifespan of 30 hours), despite its high sensing workload, small form-factor, and need to remain responsive to user dialog requests.This work was supported by Microsoft Research through its PhD Scholarship Program. We would also like to thank the anonymous reviewers and our shepherd, Jeremy Gummeson, for helping us improve the paper.This is the author accepted manuscript. The final version is available from ACM at http://dl.acm.org/citation.cfm?doid=2742647.2742672

Towards responsive Sensitive Artificial Listeners

This paper describes work in the recently started project SEMAINE, which aims to build a set of Sensitive Artificial Listeners – conversational agents designed to sustain an interaction with a human user despite limited verbal skills, through robust recognition and generation of non-verbal behaviour in real-time, both when the agent is speaking and listening. We report on data collection and on the design of a system architecture in view of real-time responsiveness

Examining the relationship between language divergence and word-of-mouth intentions

More than half the countries in the world are multilingual, and more than half the world’s consumers speak more than one language. Thus, bilingual consumers often receive services provided in a second or nonnative language. This article examines these consumers’ word-of-mouth intentions after a service provision in a second language. Two studies show that consumers served in a second language are less likely to spread positive word of mouth. The results also reveal that consumers served in a second language perceive the service provider as less responsive in general. Furthermore, the service provider’s perceived responsiveness appears far more important for determining positive word-of-mouth intentions than other factors, such as service reliability. This study therefore contributes to the fields of service and sociolinguistics, with important implications for managers as well.publisher: Elsevier articletitle: Examining the relationship between language divergence and word-of-mouth intentions journaltitle: Journal of Business Research articlelink: http://dx.doi.org/10.1016/j.jbusres.2013.09.008 content_type: article copyright: Copyright © 2013 Elsevier Inc. All rights reserved.status: publishe

Contours of Inclusion: Inclusive Arts Teaching and Learning

The purpose of this publication is to share models and case examples of the process of inclusive arts curriculum design and evaluation. The first section explains the conceptual and curriculum frameworks that were used in the analysis and generation of the featured case studies (i.e. Understanding by Design, Differentiated Instruction, and Universal Design for Learning). Data for the cases studies was collected from three urban sites (i.e. Los Angeles, San Francisco, and Boston) and included participant observations, student and teacher interviews, curriculum documentation, digital documentation of student learning, and transcripts from discussion forum and teleconference discussions from a professional learning community.The initial case studies by Glass and Barnum use the curricular frameworks to analyze and understand what inclusive practices look like in two case studies of arts-in-education programs that included students with disabilities. The second set of precedent case studies by Kronenberg and Blair, and Jenkins and Agois Hurel uses the frameworks to explain their process of including students by providing flexible arts learning options to support student learning of content standards. Both sets of case studies illuminate curricular design decisions and instructional strategies that supported the active engagement and learning of students with disabilities in educational settings shared with their peers. The second set of cases also illustrate the reflective process of using frameworks like Universal Design for Learning (UDL) to guide curricular design, responsive instructional differentiation, and the use of the arts as a rich, meaningful, and engaging option to support learning. Appended are curriculum design and evaluation tools. (Individual chapters contain references.

An Analysis of Mixed Initiative and Collaboration in Information-Seeking Dialogues

The ability to engage in mixed-initiative interaction is one of the core requirements for a conversational search system. How to achieve this is poorly understood. We propose a set of unsupervised metrics, termed ConversationShape, that highlights the role each of the conversation participants plays by comparing the distribution of vocabulary and utterance types. Using ConversationShape as a lens, we take a closer look at several conversational search datasets and compare them with other dialogue datasets to better understand the types of dialogue interaction they represent, either driven by the information seeker or the assistant. We discover that deviations from the ConversationShape of a human-human dialogue of the same type is predictive of the quality of a human-machine dialogue.Comment: SIGIR 2020 short conference pape

Incremental Unit Networks for Distributed, Symbolic Multimodal Processing and Representation

Incremental dialogue processing has been an important topic in spoken dialogue systems research, but the broader research community that makes use of language interaction (e.g., chatbots, conversational AI, spoken interaction with robots) have not adopted incremental processing despite research showing that humans perceive incremental dialogue as more natural. In this paper, we extend prior work that identifies the requirements for making spoken interaction with a system natural with the goal that our framework will be generalizable to many domains where speech is the primary method of communication. The Incremental Unit framework offers a model of incremental processing that has been extended to be multimodal, temporally aligned, enables real-time information updates, and creates complex network of information as a fine-grained information state. One challenge is that multimodal dialogue systems often have computationally expensive modules, requiring computation to be distributive. Most importantly, when speech is the means of communication, it brings the added expectation that systems understand what they (humans) say, but also that systems understand and respond without delay. In this paper, we build on top of the Incremental Unit framework and make it amenable to a distributive architecture made up of a robot and spoken dialogue system modules. To enable fast communication between the modules and to maintain module state histories, we compared two different implementations of a distributed Incremental Unit architecture. We compare both implementations systematically then with real human users and show that the implementation that uses an external attribute-value database is preferred, but there is some flexibility in which variant to use depending on the circumstances. This work offers the Incremental Unit framework as an architecture for building powerful, complete, and natural dialogue systems, specifically applicable to robots and multimodal systems researchers

A Virtual Conversational Agent for Teens with Autism: Experimental Results and Design Lessons

We present the design of an online social skills development interface for teenagers with autism spectrum disorder (ASD). The interface is intended to enable private conversation practice anywhere, anytime using a web-browser. Users converse informally with a virtual agent, receiving feedback on nonverbal cues in real-time, and summary feedback. The prototype was developed in consultation with an expert UX designer, two psychologists, and a pediatrician. Using the data from 47 individuals, feedback and dialogue generation were automated using a hidden Markov model and a schema-driven dialogue manager capable of handling multi-topic conversations. We conducted a study with nine high-functioning ASD teenagers. Through a thematic analysis of post-experiment interviews, identified several key design considerations, notably: 1) Users should be fully briefed at the outset about the purpose and limitations of the system, to avoid unrealistic expectations. 2) An interface should incorporate positive acknowledgment of behavior change. 3) Realistic appearance of a virtual agent and responsiveness are important in engaging users. 4) Conversation personalization, for instance in prompting laconic users for more input and reciprocal questions, would help the teenagers engage for longer terms and increase the system's utility