33,910 research outputs found
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Large-scale pretrained language models are the major driving force behind
recent improvements in performance on the Winograd Schema Challenge, a widely
employed test of common sense reasoning ability. We show, however, with a new
diagnostic dataset, that these models are sensitive to linguistic perturbations
of the Winograd examples that minimally affect human understanding. Our results
highlight interesting differences between humans and language models: language
models are more sensitive to number or gender alternations and synonym
replacements than humans, and humans are more stable and consistent in their
predictions, maintain a much higher absolute performance, and perform better on
non-associative instances than associative ones. Overall, humans are correct
more often than out-of-the-box models, and the models are sometimes right for
the wrong reasons. Finally, we show that fine-tuning on a large, task-specific
dataset can offer a solution to these issues.Comment: ACL 202
Neuronal modulation in the prefrontal cortex in a transitive inference task: evidence of neuronal correlates of mental schema management
When informed that A > B and B > C, humans and other animals can easily conclude that A > C. This remarkable trait of advanced animals, which allows them to manipulate knowledge flexibly to infer logical relations, has only recently garnered interest in mainstream neuroscience. How the brain controls these logical processes remains an unanswered question that has been merely superficially addressed in neuroimaging and lesion studies, which are unable to identify the underlying neuronal computations. We observed that the activation pattern of neurons in the prefrontal cortex (PFC) during pair comparisons in a highly demanding transitive inference task fully supports the behavioral performance of the two monkeys that we tested. Our results indicate that the PFC contributes to the construction and use of a mental schema to represent premises. This evidence provides a novel framework for understanding the function of various areas of brain in logic processes and impairments to them in degenerative, traumatic, and psychiatric pathologies.
SIGNIFICANCE STATEMENT:
In cognitive neuroscience, it is unknown how information that leads to inferential deductions are encoded and manipulated at the neuronal level. We addressed this question by recording single-unit activity from the dorsolateral prefrontal cortex of monkeys that were performing a transitive inference (TI) task. The TI required one to choose the higher ranked of two items, based on previous, indirect experience. Our results demonstrated that single-neuron activity supports the construction of an abstract, mental schema of ordered items in solving the task and that this representation is independent of the reward value that is experienced for the single items. These findings identify the neural substrates of abstract mental representations that support inferential thinking
- …