Do You See What I Mean? Visual Resolution of Linguistic Ambiguities

Understanding language goes hand in hand with the ability to integrate complex contextual information obtained via perception. In this work, we present a novel task for grounded language understanding: disambiguating a sentence given a visual scene which depicts one of the possible interpretations of that sentence. To this end, we introduce a new multimodal corpus containing ambiguous sentences, representing a wide range of syntactic, semantic and discourse ambiguities, coupled with videos that visualize the different interpretations for each sentence. We address this task by extending a vision model which determines if a sentence is depicted by a video. We demonstrate how such a model can be adjusted to recognize different interpretations of the same underlying sentence, allowing to disambiguate sentences in a unified fashion across the different ambiguity types.Comment: EMNLP 201

Mapping therapy for sentence production impairments in nonfluent aphasia

This study investigated a new treatment in which sentence production abilities were trained in a small group of individuals and nonfluent aphasia. It was based upon a mapping therapy approach which holds that sentence production and comprehension impairments are due to difficulties in mapping between the meaning form (thematic roles) and the syntactic form of sentences. We trained production of both canonical and noncanonical reversible sentences.Three patients received treatment and two served as control participants. Patients who received treatment demonstrated acquisition of all trained sentence structures. They also demonstrated across-task generalisation of treated and some untreated sentence structures on two tasks of constrained sentence production, and showed some improvements on a narrative task. One control participant improved on some of these measures and the other did not. There was no noted improvement in sentence comprehension abilities following treatment. Results are discussed with reference to the heterogeneity of underlying impairments in sentence production impairments in nonfluent patients, and the possible mechanisms by which improvement in sentence production might have been achieved in treatment

Selective Encoding for Abstractive Sentence Summarization

We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization. It consists of a sentence encoder, a selective gate network, and an attention equipped decoder. The sentence encoder and decoder are built with recurrent neural networks. The selective gate network constructs a second level sentence representation by controlling the information flow from encoder to decoder. The second level representation is tailored for sentence summarization task, which leads to better performance. We evaluate our model on the English Gigaword, DUC 2004 and MSR abstractive sentence summarization datasets. The experimental results show that the proposed selective encoding model outperforms the state-of-the-art baseline models.Comment: 10 pages; To appear in ACL 201