An End-to-End Conversational Style Matching Agent

We present an end-to-end voice-based conversational agent that is able to engage in naturalistic multi-turn dialogue and align with the interlocutor's conversational style. The system uses a series of deep neural network components for speech recognition, dialogue generation, prosodic analysis and speech synthesis to generate language and prosodic expression with qualities that match those of the user. We conducted a user study (N=30) in which participants talked with the agent for 15 to 20 minutes, resulting in over 8 hours of natural interaction data. Users with high consideration conversational styles reported the agent to be more trustworthy when it matched their conversational style. Whereas, users with high involvement conversational styles were indifferent. Finally, we provide design guidelines for multi-turn dialogue interactions using conversational style adaptation

The evolution of rhythmic cognition: New perspectives and technologies in comparative research

Music is a pervasive phenomenon in human culture, and musical rhythm is virtually present in all musical traditions. Research on the evolution and cognitive underpinnings of rhythm can benefit from a number of approaches. We outline key concepts and definitions, allowing fine-grained analysis of rhythmic cognition in experimental studies. We advocate comparative animal research as a useful approach to answer questions about human music cognition and review experimental evidence from different species. Finally, we suggest future directions for research on the cognitive basis of rhythm. Apart from research in semi-natural setups, possibly allowed by “drum set for chimpanzees” prototypes presented here for the first time, mathematical modeling and systematic use of circular statistics may allow promising advances

Chorusing, synchrony, and the evolutionary functions of rhythm

A central goal of biomusicology is to understand the biological basis of human musicality. One approach to this problem has been to compare core components of human musicality (relative pitch perception, entrainment, etc.) with similar capacities in other animal species. Here we extend and clarify this comparative approach with respect to rhythm. First, whereas most comparisons between human music and animal acoustic behavior have focused on spectral properties (melody and harmony), we argue for the central importance of temporal properties, and propose that this domain is ripe for further comparative research. Second, whereas most rhythm research in non-human animals has examined animal timing in isolation, we consider how chorusing dynamics can shape individual timing, as in human music and dance, arguing that group behavior is key to understanding the adaptive functions of rhythm. To illustrate the interdependence between individual and chorusing dynamics, we present a computational model of chorusing agents relating individual call timing with synchronous group behavior. Third, we distinguish and clarify mechanistic and functional explanations of rhythmic phenomena, often conflated in the literature, arguing that this distinction is key for understanding the evolution of musicality. Fourth, we expand biomusicological discussions beyond the species typically considered, providing an overview of chorusing and rhythmic behavior across a broad range of taxa (orthopterans, fireflies, frogs, birds, and primates). Finally, we propose an “Evolving Signal Timing” hypothesis, suggesting that similarities between timing abilities in biological species will be based on comparable chorusing behaviors. We conclude that the comparative study of chorusing species can provide important insights into the adaptive function(s) of rhythmic behavior in our “proto-musical” primate ancestors, and thus inform our understanding of the biology and evolution of rhythm in human music and language

What does not happen: quantifying embodied engagement using NIMI and self-adaptors

Previous research into the quantification of embodied intellectual and emotional engagement using non-verbal movement parameters has not yielded consistent results across different studies. Our research introduces NIMI (Non-Instrumental Movement Inhibition) as an alternative parameter. We propose that the absence of certain types of possible movements can be a more holistic proxy for cognitive engagement with media (in seated persons) than searching for the presence of other movements. Rather than analyzing total movement as an indicator of engagement, our research team distinguishes between instrumental movements (i.e. physical movement serving a direct purpose in the given situation) and non-instrumental movements, and investigates them in the context of the narrative rhythm of the stimulus. We demonstrate that NIMI occurs by showing viewers’ movement levels entrained (i.e. synchronised) to the repeating narrative rhythm of a timed computer-presented quiz. Finally, we discuss the role of objective metrics of engagement in future context-aware analysis of human behaviour in audience research, interactive media and responsive system and interface design

The Extended Importance of the Social Creation of Value in Evolutionary Processes

This is a single-authored paper delivered at the biennial European Conference on Artificial Intelligence (ECAI) in Riva del Garda, Italy 28 – 29 August 2006 and published in proceedings. The paper proposed that computational modelling be employed in order to test two processes that might hypothetically distinguish particular dimensions of human creativity. The first process is identified by the researcher as one in which the pursuit of novelty in artistic invention – especially music – tends to words the production of increasingly perceptually complex artefacts. The second process moves from a perspective orientated towards the individual artist and the individual art work’s reception to a more collective one: namely, whether cultural behaviour that tends towards novelty might find itself being reinforced by clustering of similar activities. This latter process would be one that explains why the process of “making special” – that may distinguish art in an anthropological sense – is one that forms particularly strong community bonds. These bonds between novelty seekers – which in the case of the researchers paper can be understood as musicians or artists – may reciprocally reinforce to support yet more novelty seeking. The relation of art and the new is not itself innovative. Boris Groys’ “On The New” provides a scoping of that territory. What is innovative is the proposal to use of computer simulation of individual and collective behaviour as a kind of artificial laboratory to determine the complex tendencies that animate these processes of novelty seeking and, by extension, artistic production. Computationally simulating behaviour that parallels both novelty-seeking as an individual practice (the artist) and the emergence of clusters of novelty seekers (the artistic styles) may, according to the researcher, provide us with new insights the historical evolution of creativity

Empathy, engagement, entrainment: the interaction dynamics of aesthetic experience

A recent version of the view that aesthetic experience is based in empathy as inner imitation explains aesthetic experience as the automatic simulation of actions, emotions, and bodily sensations depicted in an artwork by motor neurons in the brain. Criticizing the simulation theory for committing to an erroneous concept of empathy and failing to distinguish regular from aesthetic experiences of art, I advance an alternative, dynamic approach and claim that aesthetic experience is enacted and skillful, based in the recognition of others’ experiences as distinct from one’s own. In combining insights from mainly psychology, phenomenology, and cognitive science, the dynamic approach aims to explain the emergence of aesthetic experience in terms of the reciprocal interaction between viewer and artwork. I argue that aesthetic experience emerges by participatory sense-making and revolves around movement as a means for creating meaning. While entrainment merely plays a preparatory part in this, aesthetic engagement constitutes the phenomenological side of coupling to an artwork and provides the context for exploration, and eventually for moving, seeing, and feeling with art. I submit that aesthetic experience emerges from bodily and emotional engagement with works of art via the complementary processes of the perception–action and motion–emotion loops. The former involves the embodied visual exploration of an artwork in physical space, and progressively structures and organizes visual experience by way of perceptual feedback from body movements made in response to the artwork. The latter concerns the movement qualities and shapes of implicit and explicit bodily responses to an artwork that cue emotion and thereby modulate over-all affect and attitude. The two processes cause the viewer to bodily and emotionally move with and be moved by individual works of art, and consequently to recognize another psychological orientation than her own, which explains how art can cause feelings of insight or awe and disclose aspects of life that are unfamiliar or novel to the viewer

Producing Acoustic-Prosodic Entrainment in a Robotic Learning Companion to Build Learner Rapport

abstract: With advances in automatic speech recognition, spoken dialogue systems are assuming increasingly social roles. There is a growing need for these systems to be socially responsive, capable of building rapport with users. In human-human interactions, rapport is critical to patient-doctor communication, conflict resolution, educational interactions, and social engagement. Rapport between people promotes successful collaboration, motivation, and task success. Dialogue systems which can build rapport with their user may produce similar effects, personalizing interactions to create better outcomes. This dissertation focuses on how dialogue systems can build rapport utilizing acoustic-prosodic entrainment. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic features of speech, such as tone of voice or loudness, to one another over the course of a conversation. Correlated with liking and task success, a dialogue system which entrains may enhance rapport. Entrainment, however, is very challenging to model. People entrain on different features in many ways and how to design entrainment to build rapport is unclear. The first goal of this dissertation is to explore how acoustic-prosodic entrainment can be modeled to build rapport. Towards this goal, this work presents a series of studies comparing, evaluating, and iterating on the design of entrainment, motivated and informed by human-human dialogue. These models of entrainment are implemented in the dialogue system of a robotic learning companion. Learning companions are educational agents that engage students socially to increase motivation and facilitate learning. As a learning companion’s ability to be socially responsive increases, so do vital learning outcomes. A second goal of this dissertation is to explore the effects of entrainment on concrete outcomes such as learning in interactions with robotic learning companions. This dissertation results in contributions both technical and theoretical. Technical contributions include a robust and modular dialogue system capable of producing prosodic entrainment and other socially-responsive behavior. One of the first systems of its kind, the results demonstrate that an entraining, social learning companion can positively build rapport and increase learning. This dissertation provides support for exploring phenomena like entrainment to enhance factors such as rapport and learning and provides a platform with which to explore these phenomena in future work.Dissertation/ThesisDoctoral Dissertation Computer Science 201

What Code-Switching Strategies are Effective in Dialogue Systems?

Since most people in the world today are multilingual, code-switching is ubiquitous in spoken and written interactions. Paving the way for future adaptive, multilingual conversational agents, we incorporate linguistically-motivated strategies of code-switching into a rule-based goal-oriented dialogue system. We collect and release CommonAmigos, a corpus of 587 human-computer text conversations between our dialogue system and human users in mixed Spanish and English. From this new corpus, we analyze the amount of elicited code-switching, preferred patterns of user code-switching, and the impact of user demographics on code-switching. Based on these exploratory findings, we give recommendations for future effective code-switching dialogue systems, highlighting user\u27s language proficiency and gender as critical considerations