Learning Topic-Sensitive Word Representations

Distributed word representations are widely used for modeling words in NLP tasks. Most of the existing models generate one representation per word and do not consider different meanings of a word. We present two approaches to learn multiple topic-sensitive representations per word by using Hierarchical Dirichlet Process. We observe that by modeling topics and integrating topic distributions for each document we obtain representations that are able to distinguish between different meanings of a given word. Our models yield statistically significant improvements for the lexical substitution task indicating that commonly used single word representations, even when combined with contextual information, are insufficient for this task.Comment: 5 pages, 1 figure, Accepted at ACL 201

On the Utility of Conjoint and Compositional Frames and Utterance

This paper reports the results of a series of connectionist simulations aimed at establishing the value of different types of contexts as predictors of the grammatical categories of words. A comparison is made between ‘compositional’ frames (Monaghan & Christiansen, 2004), and non-compositional or ‘conjoint’ frames (Mintz, 2003). Attention is given to the role of utterance boundaries both as a category to be predicted and as a predictor. The role of developmental constraints is investigated by examining the effect of restricting the analysis to utterance-final frames. In line with results reported by Monaghan and Christiansen compositional frames are better predictors than conjoint frames, though the latter provide a small performance improvement when combined with compositional frames. Utterance boundaries are shown to be detrimental to performance when included as an item to be predicted while improving performance when included as a predictor. The utility of utterance boundaries is further supported by the finding that when the analysis is restricted to utterance-final frames (which are likely to be a particularly important source of information early in development) frames including utterance boundaries are far better predictors than lexical frames

Simulating the Noun-Verb Asymmetry in the Productivity of Children’s Speech

Several authors propose that children may acquire syntactic categories on the basis of co-occurrence statistics of words in the input. This paper assesses the relative merits of two such accounts by assessing the type and amount of productive language that results from computing co-occurrence statistics over conjoint and independent preceding and following contexts. This is achieved through the implementation of these methods in MOSAIC, a computational model of syntax acquisition that produces utterances that can be directly compared to child speech, and has a developmental component (i.e. produces increasingly long utterances). It is shown that the computation of co-occurrence statistics over conjoint contexts or frames results in a pattern of productive speech that more closely resembles that displayed by language learning children. The simulation of the developmental patterning of children’s productive speech furthermore suggests two refinements to this basic mechanism: inclusion of utterance boundaries, and the weighting of frames for their lexical content