10,893 research outputs found
Probabilistic Models of Short and Long Distance Word Dependencies in Running Text
This article describes two complementary models that represent dependencies between words in loca/ and non-local contexts. The type of local dependencies considered are sequences of part of speech categories for words. The non-local context of word dependency considered here is that of word recurrence, which is typical in a text. Both are models of phenomena that are to a reasonable extent domain independent, and thus are useful for doing prediction in systems using large vocabularies. Modeling Part of Speech Sequences A common method for modeling local word dependencies is by means of second order Markov models (also known as trigram models). In such a model the context for predicting word wi at position i in a text consists of the two words wi_l, wi-2 that precede it. The model is built from conditional probabilities: P(wi I wi_l, wi-2). The parameters of a part of speech (POS) model are of the form: P(wi [ Ci) x P(Ci [ Ci-1, Ci-2)
Character-Aware Neural Language Models
We describe a simple neural language model that relies only on
character-level inputs. Predictions are still made at the word-level. Our model
employs a convolutional neural network (CNN) and a highway network over
characters, whose output is given to a long short-term memory (LSTM) recurrent
neural network language model (RNN-LM). On the English Penn Treebank the model
is on par with the existing state-of-the-art despite having 60% fewer
parameters. On languages with rich morphology (Arabic, Czech, French, German,
Spanish, Russian), the model outperforms word-level/morpheme-level LSTM
baselines, again with fewer parameters. The results suggest that on many
languages, character inputs are sufficient for language modeling. Analysis of
word representations obtained from the character composition part of the model
reveals that the model is able to encode, from characters only, both semantic
and orthographic information.Comment: AAAI 201
- …