10 research outputs found
Distinct patterns of syntactic agreement errors in recurrent networks and humans
Determining the correct form of a verb in context requires an understanding
of the syntactic structure of the sentence. Recurrent neural networks have been
shown to perform this task with an error rate comparable to humans, despite the
fact that they are not designed with explicit syntactic representations. To
examine the extent to which the syntactic representations of these networks are
similar to those used by humans when processing sentences, we compare the
detailed pattern of errors that RNNs and humans make on this task. Despite
significant similarities (attraction errors, asymmetry between singular and
plural subjects), the error patterns differed in important ways. In particular,
in complex sentences with relative clauses error rates increased in RNNs but
decreased in humans. Furthermore, RNNs showed a cumulative effect of attractors
but humans did not. We conclude that at least in some respects the syntactic
representations acquired by RNNs are fundamentally different from those used by
humans.Comment: Proceedings of the 40th Annual Conference of the Cognitive Science
Societ
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing
Two of the central factors believed to underpin human sentence processing
difficulty are expectations and retrieval from working memory. A recent attempt
to create a unified cognitive model integrating these two factors relied on the
parallels between the self-attention mechanism of transformer language models
and cue-based retrieval theories of working memory in human sentence processing
(Ryu and Lewis 2021). While Ryu and Lewis show that attention patterns in
specialized attention heads of GPT-2 are consistent with similarity-based
interference, a key prediction of cue-based retrieval models, their method
requires identifying syntactically specialized attention heads, and makes the
cognitively implausible assumption that hundreds of memory retrieval operations
take place in parallel. In the present work, we develop a recurrent neural
language model with a single self-attention head, which more closely parallels
the memory system assumed by cognitive theories. We show that our model's
single attention head captures semantic and syntactic interference effects
observed in human experiments.Comment: To appear in Findings of the Association for Computational
Linguistics: EMNLP 202