Visual world studies of conversational perspective taking: similar findings, diverging interpretations

Visual-world eyetracking greatly expanded the potential for insight into how listeners access and use common ground during situated language comprehension. Past reviews of visual world studies on perspective taking have largely taken the diverging findings of the various studies at face value, and attributed these apparently different findings to differences in the extent to which the paradigms used by different labs afford collaborative interaction. Researchers are asking questions about perspective taking of an increasingly nuanced and sophisticated nature, a clear indicator of progress. But this research has the potential not only to improve our understanding of conversational perspective taking. Grappling with problems of data interpretation in such a complex domain has the unique potential to drive visual world researchers to a deeper understanding of how to best map visual world data onto psycholinguistic theory. I will argue against this interactional affordances explanation, on two counts. First, it implies that interactivity affects the overall ability to form common ground, and thus provides no straightforward explanation of why, within a single noninteractive study, common ground can have very large effects on some aspects of processing (referential anticipation) while having negligible effects on others (lexical processing). Second, and more importantly, the explanation accepts the divergence in published findings at face value. However, a closer look at several key studies shows that the divergences are more likely to reflect inconsistent practices of analysis and interpretation that have been applied to an underlying body of data that is, in fact, surprisingly consistent. The diverging interpretations, I will argue, are the result of differences in the handling of anticipatory baseline effects (ABEs) in the analysis of visual world data. ABEs arise in perspective-taking studies because listeners have earlier access to constraining information about who knows what than they have to referential speech, and thus can already show biases in visual attention even before the processing of any referential speech has begun. To be sure, these ABEs clearly indicate early access to common ground; however, access does not imply integration, since it is possible that this information is not used later to modulate the processing of incoming speech. Failing to account for these biases using statistical or experimental controls leads to over-optimistic assessments of listeners’ ability to integrate this information with incoming speech. I will show that several key studies with varying degrees of interactional affordances all show similar temporal profiles of common ground use during the interpretive process: early anticipatory effects, followed by bottom-up effects of lexical processing that are not modulated by common ground, followed (optionally) by further late effects that are likely to be post-lexical. Furthermore, this temporal profile for common ground radically differs from the profile of contextual effects related to verb semantics. Together, these findings are consistent with the proposal that lexical processes are encapsulated from common ground, but cannot be straightforwardly accounted for by probabilistic constraint-based approaches

Timescales of Massive Human Entrainment

The past two decades have seen an upsurge of interest in the collective behaviors of complex systems composed of many agents entrained to each other and to external events. In this paper, we extend concepts of entrainment to the dynamics of human collective attention. We conducted a detailed investigation of the unfolding of human entrainment - as expressed by the content and patterns of hundreds of thousands of messages on Twitter - during the 2012 US presidential debates. By time locking these data sources, we quantify the impact of the unfolding debate on human attention. We show that collective social behavior covaries second-by-second to the interactional dynamics of the debates: A candidate speaking induces rapid increases in mentions of his name on social media and decreases in mentions of the other candidate. Moreover, interruptions by an interlocutor increase the attention received. We also highlight a distinct time scale for the impact of salient moments in the debate: Mentions in social media start within 5-10 seconds after the moment; peak at approximately one minute; and slowly decay in a consistent fashion across well-known events during the debates. Finally, we show that public attention after an initial burst slowly decays through the course of the debates. Thus we demonstrate that large-scale human entrainment may hold across a number of distinct scales, in an exquisitely time-locked fashion. The methods and results pave the way for careful study of the dynamics and mechanisms of large-scale human entrainment.Comment: 20 pages, 7 figures, 6 tables, 4 supplementary figures. 2nd version revised according to peer reviewers' comments: more detailed explanation of the methods, and grounding of the hypothese

Conversational Sensing

Recent developments in sensing technologies, mobile devices and context-aware user interfaces have made it possible to represent information fusion and situational awareness as a conversational process among actors - human and machine agents - at or near the tactical edges of a network. Motivated by use cases in the domain of security, policing and emergency response, this paper presents an approach to information collection, fusion and sense-making based on the use of natural language (NL) and controlled natural language (CNL) to support richer forms of human-machine interaction. The approach uses a conversational protocol to facilitate a flow of collaborative messages from NL to CNL and back again in support of interactions such as: turning eyewitness reports from human observers into actionable information (from both trained and untrained sources); fusing information from humans and physical sensors (with associated quality metadata); and assisting human analysts to make the best use of available sensing assets in an area of interest (governed by management and security policies). CNL is used as a common formal knowledge representation for both machine and human agents to support reasoning, semantic information fusion and generation of rationale for inferences, in ways that remain transparent to human users. Examples are provided of various alternative styles for user feedback, including NL, CNL and graphical feedback. A pilot experiment with human subjects shows that a prototype conversational agent is able to gather usable CNL information from untrained human subjects

"How May I Help You?": Modeling Twitter Customer Service Conversations Using Fine-Grained Dialogue Acts

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained "dialogue acts" frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between customer and agent behavior in Twitter customer service conversations, and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes, and present actionable rules based on our findings. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.Comment: 13 pages, 6 figures, IUI 201

The listening talker: A review of human and algorithmic context-induced modifications of speech

International audienceSpeech output technology is finding widespread application, including in scenarios where intelligibility might be compromised - at least for some listeners - by adverse conditions. Unlike most current algorithms, talkers continually adapt their speech patterns as a response to the immediate context of spoken communication, where the type of interlocutor and the environment are the dominant situational factors influencing speech production. Observations of talker behaviour can motivate the design of more robust speech output algorithms. Starting with a listener-oriented categorisation of possible goals for speech modification, this review article summarises the extensive set of behavioural findings related to human speech modification, identifies which factors appear to be beneficial, and goes on to examine previous computational attempts to improve intelligibility in noise. The review concludes by tabulating 46 speech modifications, many of which have yet to be perceptually or algorithmically evaluated. Consequently, the review provides a roadmap for future work in improving the robustness of speech output

Wh-pronoun and complementizer relative clauses : unintegration features in conversational Polish

Celem niniejszego artykułu jest analiza dwóch typów zdań względnych w mówionym języku polskim - tj. wprowadzanych przez zaimek względny który oraz przez nieodmienny relator co. Głównym obszarem zainteresowania są niekanoniczne konstrukcje, w których obserwuje się rozluźnioną integrację akomodacyjną pomiędzy grupą rzeczownikową a zdaniem względnym. Dla obu wskaźników zespolenia (który i co), tekst omawia poszczególne typy cech formalnych, które powodują taką niezintegrowaną strukturę. Analiza danych korpusowych pozwala również na ilościowe określenie stopnia dezintegracji w obu typach dań. Mimo że spontaniczny język mówiony wymusza pewną dozę dezintegracji w obu przypadkach, zdania względne z co (zwłaszcza te w funkcji innej niż podmiot) znacznie częściej charakteryzują się taką właśnie budową. Zdania z co odbiegają od kanonicznej relatywizacji jeszcze w innym sensie: oprócz funkcji relativum generale, co może pełnić inne funkcje semantyczne, takie jak spójniki podrzędne miejsca i czasu (porównywalne z gdzie i kiedy) lub spójnik ogólnego zastosowania. Tego rodzaju użycia wskazują na ekspansję statusu kategorialnego co. Zaobserwowane zjawiska pokrywają się z doniesieniami innych autorów badających składnię spontanicznego języka mówionego (Miller and Weinert 1998).The paper examines syntactic features of non-canonical relativization in spoken Polish that loosen the structural integration of two types of relative clauses - one introduced by the complementizer co, the other by the wh-pronoun który. The resulting unintegration holds between the head NP and the co/który clause and contrasts with the integrated structure of canonical relatives. I discuss the range of unintegration features observed for both types in corpus data and indicate the distinct quantitative extents to which the two types are unintegrated. Although the nature of spontaneous conversation is such that it imposes some loosening of structural cohesion in both types, co clauses (especially non-subject relative clauses) are far more frequently unintegrated than który clauses. Also, co clauses depart functionally from the canonical relative structure in that the complementizer co serves functions other than that of a straightforward relativizer, namely it has conjunction-like uses (temporal, spatial, and general conjunction), indicating an expansion of the categorial status of co. The observed unintegration of Polish conversational relatives is in line with previous analyses of the syntax of unplanned speech (e.g. Miller and Weinert 1998)

Prefrontal Cortex: Role in Language Communication during Social Interaction

One important question that remains open for the relationship between the brain and social behavior is whether and how prefrontal mechanisms responsible for social cognitive processes take place in language communication. Conventional studies have highlighted the role of inferior frontal gyrus (IFG) in processing context-independent linguistic information in speech and discourse. However, it is unclear how the medial prefrontal cortex (mPFC), the lateral prefrontal cortex (lPFC), and other structures (such as medial superior frontal gyrus, premotor cortex, anterior cingulate cortex, etc.) are involved when socially relevant language is encountered in real-life scenarios. Emerging neuroimaging and patient studies have suggested the association of prefrontal regions with individual differences and impairments in the comprehension of speech act, nonliteral language, or construction-based pragmatic information. By summarizing and synthesizing the most recent functional magnetic resonance imaging (fMRI) studies, this chapter aims to show how neurocognitive components underlying the social function of prefrontal cortex support pragmatic language processing, such as weighing relevant social signals, resolving ambiguities, and identifying hidden speaker meanings. The conclusion lends impact on an emerging interest in neuropragmatics and points out a promising line of research to address the mediating role of prefrontal cortex in the relation of language and social cognition