201 research outputs found
Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity
We evaluate 8 different word embedding models on their usefulness for
predicting the neural activation patterns associated with concrete nouns. The
models we consider include an experiential model, based on crowd-sourced
association data, several popular neural and distributional models, and a model
that reflects the syntactic context of words (based on dependency parses). Our
goal is to assess the cognitive plausibility of these various embedding models,
and understand how we can further improve our methods for interpreting brain
imaging data.
We show that neural word embedding models exhibit superior performance on the
tasks we consider, beating experiential word representation model. The
syntactically informed model gives the overall best performance when predicting
brain activation patterns from word embeddings; whereas the GloVe
distributional method gives the overall best performance when predicting in the
reverse direction (words vectors from brain images). Interestingly, however,
the error patterns of these different models are markedly different. This may
support the idea that the brain uses different systems for processing different
kinds of words. Moreover, we suggest that taking the relative strengths of
different embedding models into account will lead to better models of the brain
activity associated with words.Comment: accepted at Cognitive Modeling and Computational Linguistics 201
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
How does the brain represent different modes of information? Can we design a
system that automatically understands what the user is thinking? Such questions
can be answered by studying brain recordings like functional magnetic resonance
imaging (fMRI). As a first step, the neuroscience community has contributed
several large cognitive neuroscience datasets related to passive
reading/listening/viewing of concept words, narratives, pictures and movies.
Encoding and decoding models using these datasets have also been proposed in
the past two decades. These models serve as additional tools for basic research
in cognitive science and neuroscience. Encoding models aim at generating fMRI
brain representations given a stimulus automatically. They have several
practical applications in evaluating and diagnosing neurological conditions and
thus also help design therapies for brain damage. Decoding models solve the
inverse problem of reconstructing the stimuli given the fMRI. They are useful
for designing brain-machine or brain-computer interfaces. Inspired by the
effectiveness of deep learning models for natural language processing, computer
vision, and speech, recently several neural encoding and decoding models have
been proposed. In this survey, we will first discuss popular representations of
language, vision and speech stimuli, and present a summary of neuroscience
datasets. Further, we will review popular deep learning based encoding and
decoding architectures and note their benefits and limitations. Finally, we
will conclude with a brief summary and discussion about future trends. Given
the large amount of recently published work in the `computational cognitive
neuroscience' community, we believe that this survey nicely organizes the
plethora of work and presents it as a coherent story.Comment: 16 pages, 10 figure
Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models
Recent years have seen a growing interest within the natural language processing (NLP)community in evaluating the ability of semantic models to capture human meaning representation in the brain. Existing research has mainly focused on applying semantic models to de-code brain activity patterns associated with the meaning of individual words, and, more recently, this approach has been extended to sentences and larger text fragments. Our work is the first to investigate metaphor process-ing in the brain in this context. We evaluate a range of semantic models (word embeddings, compositional, and visual models) in their ability to decode brain activity associated with reading of both literal and metaphoric sentences. Our results suggest that compositional models and word embeddings are able to capture differences in the processing of literal and metaphoric sentences, providing sup-port for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension
Mapping Brains with Language Models: A Survey
Over the years, many researchers have seemingly made the same observation:
Brain and language model activations exhibit some structural similarities,
enabling linear partial mappings between features extracted from neural
recordings and computational language models. In an attempt to evaluate how
much evidence has been accumulated for this observation, we survey over 30
studies spanning 10 datasets and 8 metrics. How much evidence has been
accumulated, and what, if anything, is missing before we can draw conclusions?
Our analysis of the evaluation methods used in the literature reveals that some
of the metrics are less conservative. We also find that the accumulated
evidence, for now, remains ambiguous, but correlations with model size and
quality provide grounds for cautious optimism
Modeling affirmative and negated action processing in the brain with lexical and compositional semantic models
Recent work shows that distributional semantic models can be used to decode patterns of brain activity associated with individual words and sentence meanings. However, it is yet unclear to what extent such models can be used to study and ecode fMRI patterns associated with specific aspects of semantic composition such as the negation function. In this paper, we apply lexical and compositional semantic models to decode fMRI patterns associated with negated and affirmative sentences containing hand-action verbs. Our results show reduced decoding (correlation) of sentences where the verb is in the negated context, as compared to the affirmative one, within brain regions implicated in action-semantic processing. This supports behavioral and brain imaging studies, suggesting that negation involves reduced access to aspects of the affirmative mental representation. The results pave the way for testing alternate semantic models of negation against human semantic processing in the brain
- …