Reference and the facilitation of search in spatial domains

This is a pre-final version of the article, whose official publication is expected in the winter of 2013-14.Peer reviewedPreprin

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past decade or so, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of relatively recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artificial intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of Natural Language Processing, with an emphasis on different evaluation methods and the relationships between them.Comment: Published in Journal of AI Research (JAIR), volume 61, pp 75-170. 118 pages, 8 figures, 1 tabl

Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries

Recognising objects according to a pre-defined fixed set of class labels has been well studied in the Computer Vision. There are a great many practical applications where the subjects that may be of interest are not known beforehand, or so easily delineated, however. In many of these cases natural language dialog is a natural way to specify the subject of interest, and the task achieving this capability (a.k.a, Referring Expression Comprehension) has recently attracted attention. To this end we propose a unified framework, the ParalleL AttentioN (PLAN) network, to discover the object in an image that is being referred to in variable length natural expression descriptions, from short phrases query to long multi-round dialogs. The PLAN network has two attention mechanisms that relate parts of the expressions to both the global visual content and also directly to object candidates. Furthermore, the attention mechanisms are recurrent, making the referring process visualizable and explainable. The attended information from these dual sources are combined to reason about the referred object. These two attention mechanisms can be trained in parallel and we find the combined system outperforms the state-of-art on several benchmarked datasets with different length language input, such as RefCOCO, RefCOCO+ and GuessWhat?!.Comment: 11 page

Speaker emotion can affect ambiguity production

Does speaker emotion affect degree of ambiguity in referring expressions? We used referential communication tasks preceded by mood induction to examine whether positive emotional valence may be linked to ambiguity of referring expressions. In Experiment 1, participants had to identify sequences of objects with homophonic labels (e.g., the animal bat, a baseball bat) for hypothetical addressees. This required modification of the homophones. Happy speakers were less likely to modify the second homophone to repair a temporary ambiguity (i.e., they were less likely to say … first cover the bat, then cover the baseball bat …). In Experiment 2, participants had to identify one of two identical objects in an object array, which required a modifying relative clause (the shark that's underneath the shoe). Happy speakers omitted the modifying relative clause twice as often as neutral speakers (e.g., by saying Put the shark underneath the sheep), thereby rendering the entire utterance ambiguous in the context of two sharks. The findings suggest that one consequence of positive mood appears to be more ambiguity in speech. This effect is hypothesised to be due to a less effortful processing style favouring an egocentric bias impacting perspective taking or monitoring of alignment of utterances with an addressee's perspective

How Do I Address You? Modelling addressing behavior based on an analysis of a multi-modal corpora of conversational discourse

Addressing is a special kind of referring and thus principles of multi-modal referring expression generation will also be basic for generation of address terms and addressing gestures for conversational agents. Addressing is a special kind of referring because of the different (second person instead of object) role that the referent has in the interaction. Based on an analysis of addressing behaviour in multi-party face-to-face conversations (meetings, TV discussions as well as theater plays), we present outlines of a model for generating multi-modal verbal and non-verbal addressing behaviour for agents in multi-party interactions

Exploiting visual salience for the generation of referring expressions

In this paper we present a novel approach to generating referring expressions (GRE) that is tailored to a model of the visual context the user is attending to. The approach integrates a new computational model of visual salience in simulated 3-D environments with Dale and Reiter’s (1995) Incremental Algorithm. The advantage of our GRE framework are: (1) the context set used by the GRE algorithm is dynamically computed by the visual saliency algorithm as a user navigates through a simulation; (2) the integration of visual salience into the generation process means that in some instances underspecified but sufficiently detailed descriptions of the target object are generated that are shorter than those generated by GRE algorithms which focus purely on adjectival and type attributes; (3) the integration of visual saliency into the generation process means that our GRE algorithm will in some instances succeed in generating a description of the target object in situations where GRE algorithms which focus purely on adjectival and type attributes fail

The effect of perceptual availability and prior discourse on young children's use of referring expressions.

Choosing appropriate referring expressions requires assessing whether a referent is “available” to the addressee either perceptually or through discourse. In Study 1, we found that 3- and 4-year-olds, but not 2-year-olds, chose different referring expressions (noun vs. pronoun) depending on whether their addressee could see the intended referent or not. In Study 2, in more neutral discourse contexts than previous studies, we found that 3- and 4-year-olds clearly differed in their use of referring expressions according to whether their addressee had already mentioned a referent. Moreover, 2-yearolds responded with more naming constructions when the referent had not been mentioned previously. This suggests that, despite early social–cognitive developments, (a) it takes time tomaster the given/new contrast linguistically, and (b) children understand the contrast earlier based on discourse, rather than perceptual context