177 research outputs found
Probing with Noise: Unpicking the Warp and Weft of Taxonomic and Thematic Meaning Representations in Static and Contextual Embeddings
The semantic relatedness of words has two key dimensions: it can be based on taxonomic information or thematic, co-occurrence-based information. These are captured by different language resources—taxonomies and natural corpora—from which we can build different computational meaning representations that are able to reflect these relationships. Vector representations are arguably the most popular meaning representations in NLP, encoding information in a shared multidimensional semantic space and allowing for distances between points to reflect relatedness between items that populate the space. Improving our understanding of how different types of linguistic information are encoded in vector space can provide valuable insights to the field of model interpretability and can further our understanding of different encoder architectures.
Alongside vector dimensions, we argue that information can be encoded in more implicit ways and hypothesise that it is possible for the vector magnitude—the norm—to also carry linguistic information. We develop a method to test this hypothesis and provide a systematic exploration of the role of the vector norm in encoding the different axes of semantic relatedness across a variety of vector representations, including taxonomic, thematic, static and contextual embeddings.
The method is an extension of the standard probing framework and allows for relative intrinsic interpretations of probing results. It relies on introducing targeted noise that ablates information encoded in embeddings and is grounded by solid baselines and confidence intervals. We call the method probing with noise and test the method at both the word and sentence level, on a host of established linguistic probing tasks, as well as two new semantic probing tasks: hypernymy and idiomatic usage detection.
Our experiments show that the method is able to provide geometric insights into embeddings and can demonstrate whether the norm encodes the linguistic information being probed for. This confirms the existence of separate information containers in English word2vec, GloVe and BERT embeddings. The experiments and complementary analyses show that different encoders encode different kinds of linguistic information in the norm: taxonomic vectors store hypernym-hyponym information in the norm, while non-taxonomic vectors do not. Meanwhile, non-taxonomic GloVe embeddings encode syntactic and sentence length information in the vector norm, while the contextual BERT encodes contextual incongruity.
Our method can thus reveal where in the embeddings certain information is contained. Furthermore, it can be supplemented by an array of post-hoc analyses that reveal how information is encoded as well, thus offering valuable structural and geometric insights into the different types of embeddings
Understanding Word Embedding Stability Across Languages and Applications
Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this thesis, we consider several aspects of embedding spaces, including their stability. First, we propose a definition of stability, and show that common English word embeddings are surprisingly unstable. We explore how properties of data, words, and algorithms relate to instability. We extend this work to approximately 100 world languages, considering how linguistic typology relates to stability. Additionally, we consider contextualized output embedding spaces. Using paraphrases, we explore properties and assumptions of BERT, a popular embedding algorithm.
Second, we consider how stability and other word embedding properties affect tasks where embeddings are commonly used. We consider both word embeddings used as features in downstream applications and corpus-centered applications, where embeddings are used to study characteristics of language and individual writers. In addition to stability, we also consider other word embedding properties, specifically batching and curriculum learning, and how methodological choices made for these properties affect downstream tasks.
Finally, we consider how knowledge of stability affects how we use word embeddings. Throughout this thesis, we discuss strategies to mitigate instability and provide analyses highlighting the strengths and weaknesses of word embeddings in different scenarios and languages. We show areas where more work is needed to improve embeddings, and we show where embeddings are already a strong tool.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/162917/1/lburdick_1.pd
Not wacky vs. definitely wacky: a study of scalar adverbs in pretrained language models
Vector-space models of word meaning all assume that words occurring in similar contexts have similar meanings. Words that are similar in their topical associations but differ in their logical force tend to emerge as semantically close – creating well-known challenges for NLP applications that involve logical reasoning. Pretrained language models such as BERT, RoBERTa, GPT-2, and GPT-3 hold the promise of performing better on logical tasks than classic static word embeddings. However, reports are mixed about their success. Here, we advance this discussion through a systematic study of scalar adverbs, an under-explored class of words with strong logical force. Using three different tasks involving both naturalistic social media data and constructed examples, we investigate the extent to which BERT, RoBERTa, GPT-2 and GPT-3 exhibit knowledge of these common words. We ask: 1) Do the models distinguish amongst the three semantic categories of MODALITY, FREQUENCY and DEGREE? 2) Do they have implicit representations of full scales from maximally negative to maximally positive? 3) How do word frequency and contextual factors impact model performance? We find that despite capturing some aspects of logical meaning, the models still have obvious shortfalls
Not wacky vs. definitely wacky: A study of scalar adverbs in pretrained language models
Vector space models of word meaning all share the assumption that words
occurring in similar contexts have similar meanings. In such models, words that
are similar in their topical associations but differ in their logical force
tend to emerge as semantically close, creating well-known challenges for NLP
applications that involve logical reasoning. Modern pretrained language models,
such as BERT, RoBERTa and GPT-3 hold the promise of performing better on
logical tasks than classic static word embeddings. However, reports are mixed
about their success. In the current paper, we advance this discussion through a
systematic study of scalar adverbs, an under-explored class of words with
strong logical force. Using three different tasks, involving both naturalistic
social media data and constructed examples, we investigate the extent to which
BERT, RoBERTa, GPT-2 and GPT-3 exhibit general, human-like, knowledge of these
common words. We ask: 1) Do the models distinguish amongst the three semantic
categories of MODALITY, FREQUENCY and DEGREE? 2) Do they have implicit
representations of full scales from maximally negative to maximally positive?
3) How do word frequency and contextual factors impact model performance? We
find that despite capturing some aspects of logical meaning, the models fall
far short of human performance.Comment: Published in BlackBoxNLP workshop, EMNLP 202
LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond
Distributional semantics based on neural approaches is a cornerstone of
Natural Language Processing, with surprising connections to human meaning
representation as well. Recent Transformer-based Language Models have proven
capable of producing contextual word representations that reliably convey
sense-specific information, simply as a product of self-supervision. Prior work
has shown that these contextual representations can be used to accurately
represent large sense inventories as sense embeddings, to the extent that a
distance-based solution to Word Sense Disambiguation (WSD) tasks outperforms
models trained specifically for the task. Still, there remains much to
understand on how to use these Neural Language Models (NLMs) to produce sense
embeddings that can better harness each NLM's meaning representation abilities.
In this work we introduce a more principled approach to leverage information
from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We
also emphasize the versatility of these sense embeddings in contrast to
task-specific models, applying them on several sense-related tasks, besides
WSD, while demonstrating improved performance using our proposed approach over
prior work focused on sense embeddings. Finally, we discuss unexpected findings
regarding layer and model performance variations, and potential applications
for downstream tasks.Comment: Accepted to Artificial Intelligence Journal (AIJ
Neural models of language use:Studies of language comprehension and production in context
Artificial neural network models of language are mostly known and appreciated today for providing a backbone for formidable AI technologies. This thesis takes a different perspective. Through a series of studies on language comprehension and production, it investigates whether artificial neural networks—beyond being useful in countless AI applications—can serve as accurate computational simulations of human language use, and thus as a new core methodology for the language sciences
- …