102 research outputs found
Colorless green recurrent networks dream hierarchically
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues ( The colorless green ideas I ate with the chair sleep furiously ), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence
Recommended from our members
Can language models learn constraints on gap-filler dependency? Case of Japanese relative clause islands
Does Syntactic Knowledge in Multilingual Language Models Transfer Across Languages?
Recent work has shown that neural models can be successfully trained on multiple languages simultaneously. We investigate whether such models learn to share and exploit common syntactic knowledge among the languages on which they are trained. This extended abstract presents our preliminary result
What Syntactic Structures block Dependencies in RNN Language Models?
Recurrent Neural Networks (RNNs) trained on a language modeling task have
been shown to acquire a number of non-local grammatical dependencies with some
success. Here, we provide new evidence that RNN language models are sensitive
to hierarchical syntactic structure by investigating the filler--gap dependency
and constraints on it, known as syntactic islands. Previous work is
inconclusive about whether RNNs learn to attenuate their expectations for gaps
in island constructions in particular or in any sufficiently complex syntactic
environment. This paper gives new evidence for the former by providing control
studies that have been lacking so far. We demonstrate that two state-of-the-art
RNN models are are able to maintain the filler--gap dependency through
unbounded sentential embeddings and are also sensitive to the hierarchical
relationship between the filler and the gap. Next, we demonstrate that the
models are able to maintain possessive pronoun gender expectations through
island constructions---this control case rules out the possibility that island
constructions block all information flow in these networks. We also evaluate
three untested islands constraints: coordination islands, left branch islands,
and sentential subject islands. Models are able to learn left branch islands
and learn coordination islands gradiently, but fail to learn sentential subject
islands. Through these controls and new tests, we provide evidence that model
behavior is due to finer-grained expectations than gross syntactic complexity,
but also that the models are conspicuously un-humanlike in some of their
performance characteristics.Comment: To Appear at the 41st Annual Meeting of the Cognitive Science
Society, Montreal, Canada, July 201
- …