120 research outputs found
Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution
Lexical substitution, i.e. generation of plausible words that can replace a
particular target word in a given context, is an extremely powerful technology
that can be used as a backbone of various NLP applications, including word
sense induction and disambiguation, lexical relation extraction, data
augmentation, etc. In this paper, we present a large-scale comparative study of
lexical substitution methods employing both rather old and most recent language
and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT,
RoBERTa, XLNet. We show that already competitive results achieved by SOTA
LMs/MLMs can be further substantially improved if information about the target
word is injected properly. Several existing and new target word injection
methods are compared for each LM/MLM using both intrinsic evaluation on lexical
substitution datasets and extrinsic evaluation on word sense induction (WSI)
datasets. On two WSI datasets we obtain new SOTA results. Besides, we analyze
the types of semantic relations between target words and their substitutes
generated by different models or given by annotators.Comment: arXiv admin note: text overlap with arXiv:2006.0003
A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models
Word representation has always been an important research area in the history
of natural language processing (NLP). Understanding such complex text data is
imperative, given that it is rich in information and can be used widely across
various applications. In this survey, we explore different word representation
models and its power of expression, from the classical to modern-day
state-of-the-art word representation language models (LMS). We describe a
variety of text representation methods, and model designs have blossomed in the
context of NLP, including SOTA LMs. These models can transform large volumes of
text into effective vector representations capturing the same semantic
information. Further, such representations can be utilized by various machine
learning (ML) algorithms for a variety of NLP related tasks. In the end, this
survey briefly discusses the commonly used ML and DL based classifiers,
evaluation metrics and the applications of these word embeddings in different
NLP tasks
- …