From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

Over the past years, distributed semantic representations have proved to be effective and flexible keepers of prior knowledge to be integrated into downstream applications. This survey focuses on the representation of meaning. We start from the theoretical background behind word vector space models and highlight one of their major limitations: the meaning conflation deficiency, which arises from representing a word with all its possible meanings as a single vector. Then, we explain how this deficiency can be addressed through a transition from the word level to the more fine-grained level of word senses (in its broader acceptation) as a method for modelling unambiguous lexical meaning. We present a comprehensive overview of the wide range of techniques in the two main branches of sense representation, i.e., unsupervised and knowledge-based. Finally, this survey covers the main evaluation procedures and applications for this type of representation, and provides an analysis of four of its important aspects: interpretability, sense granularity, adaptability to different domains and compositionality.Comment: 46 pages, 8 figures. Published in Journal of Artificial Intelligence Researc

Word vs. Class-Based Word Sense Disambiguation

As empirically demonstrated by the Word Sense Disambiguation (WSD) tasks of the last SensEval/SemEval exercises, assigning the appropriate meaning to words in context has resisted all attempts to be successfully addressed. Many authors argue that one possible reason could be the use of inappropriate sets of word meanings. In particular, WordNet has been used as a de-facto standard repository of word meanings in most of these tasks. Thus, instead of using the word senses defined in WordNet, some approaches have derived semantic classes representing groups of word senses. However, the meanings represented by WordNet have been only used for WSD at a very fine-grained sense level or at a very coarse-grained semantic class level (also called SuperSenses). We suspect that an appropriate level of abstraction could be on between both levels. The contributions of this paper are manifold. First, we propose a simple method to automatically derive semantic classes at intermediate levels of abstraction covering all nominal and verbal WordNet meanings. Second, we empirically demonstrate that our automatically derived semantic classes outperform classical approaches based on word senses and more coarse-grained sense groupings. Third, we also demonstrate that our supervised WSD system benefits from using these new semantic classes as additional semantic features while reducing the amount of training examples. Finally, we also demonstrate the robustness of our supervised semantic class-based WSD system when tested on out of domain corpus.This work has been partially supported by the NewsReader project (ICT-2011-316404), the Spanish project SKaTer (TIN2012-38584-C06-02)

Global Organization of the Lexicon

The lexicon consists of a set of word meanings and their semantic relationships. A systematic representation of the English lexicon based in psycholinguistic considerations has been put together in the database Wordnet in a long-term collaborative effort1. We present here a quantitative study of the graph structure of Wordnet in order to understand the global organization of the lexicon. We find that semantic links follow power-law, scale-invariant behaviors typical of self-organizing networks. Polysemy, the ambiguity of an individual word, can act as a link in the semantic network, relating the different meanings of a common word. Inclusion of polysemous links has a profound impact in the organization of the semantic graph, converting it into a small world, with clusters of high traffic (hubs) representing abstract concepts. Our results show that polysemy organizes the semantic graph in a compact and categorical representation, and thus may explain the ubiquity of polysemy across languages

Multi Sense Embeddings from Topic Models

Distributed word embeddings have yielded state-of-the-art performance in many NLP tasks, mainly due to their success in capturing useful semantic information. These representations assign only a single vector to each word whereas a large number of words are polysemous (i.e., have multiple meanings). In this work, we approach this critical problem in lexical semantics, namely that of representing various senses of polysemous words in vector spaces. We propose a topic modeling based skip-gram approach for learning multi-prototype word embeddings. We also introduce a method to prune the embeddings determined by the probabilistic representation of the word in each topic. We use our embeddings to show that they can capture the context and word similarity strongly and outperform various state-of-the-art implementations

Реалізація форм словотвірного значення у дієслівних субстантивах з категоріальним значенням активного руху (на матеріалі сучасної німецької мови)

The article explores the correlation between word-forming meaning and forms representing it. Word-forming meaning takes an intermediate position between lexical and grammatical meanings and belongs simultaneously to the two planes - of expression and of contents. For this reason it is realized through interaction between these two planes. Word-forming meaning is actualized in the following ways: word-forming morphemes impart wordforming meaning to derivatives; word-forming meaning is inherent in word-forming models and it can result from the interaction between the stem and a word-forming morpheme.У статті досліджується проблема співвідношення між словотвірним значенням та формами його реалізації. Словотвірне значення займає проміжну позицію між лексичним та граматичним значеннями і належить одночасно до двох планів - вираження і змісту. Тому воно реалізується через взаємодію між цими двома планами. Словотвірне значення актуалізується таким чином: словотворчі морфеми надають словотвірного значення дериватам; словотвірне значення є значенням словотвірної моделі, а також може виникати в результаті взаємодії твірної основи та афікса. При цитуванні документа, використовуйте посилання http://essuir.sumdu.edu.ua/handle/123456789/105

Codifcadores para modelos latentes con múltiples realizaciones no homomórfcas

Natural Language Processing (NLP) is a commonly used feld of AI for tasks ranging from translation, speech recognition, handwriting recognition, or even part of speech (POS) tagging. Word embedding is the practice of representing words in a mathematical manner to perform NLP tasks. Typically this embedding is done through vectors. The idea behind this thesis is related: to create probabilistic representations for a word’s part of speech. For this, a special Conditioned Variational AutoEncoder (CVAE) will be used. The CVAE is trained to copy its input by creating a latent space from which the model will sample to generate its output, in this case a word (and its context). The variable it will be conditioned to is the POS tag. These can be easily obtained with libraries like NLTK, but the word that will be input to the model must have some sort of representation. A popular approach is word2vec, which assigns a vectors representation to each word given a large text corpus. However, word2vec does not include contextual representation in its vectors. Another solution is Google’s Transformer BERT, a bidirectional language model trained to predict a masked word in a sequence, and also determine whether two sequences are a continuation of each other. BERT can also be used to create word embeddings, by performing feature extraction of its hidden layers for any given word in a sequence. Because of its bidirectional nature, there embeddings contain ordered contextual information from the left and right. Although not consistently, the CVAE is able to represent three different POS of a word with three different Gaussians. A series of weights that select one Gaussian when sampling from the latent space, prove to be essential to accomplish this task. This work could be expanded so that the representations are for word meanings, or even word meanings across multiple languages

Tone of voice guides word learning in informative referential contexts

Listeners infer which object in a visual scene a speaker refers to from the systematic variation of the speaker's tone of voice (ToV). We examined whether ToV also guides word learning. During exposure, participants heard novel adjectives (e.g., “daxen”) spoken with a ToV representing hot, cold, strong, weak, big, or small while viewing picture pairs representing the meaning of the adjective and its antonym (e.g., elephant-ant for big-small). Eye fixations were recorded to monitor referent detection and learning. During test, participants heard the adjectives spoken with a neutral ToV, while selecting referents from familiar and unfamiliar picture pairs. Participants were able to learn the adjectives' meanings, and, even in the absence of informative ToV, generalise them to new referents. A second experiment addressed whether ToV provides sufficient information to infer the adjectival meaning or needs to operate within a referential context providing information about the relevant semantic dimension. Participants who saw printed versions of the novel words during exposure performed at chance during test. ToV, in conjunction with the referential context, thus serves as a cue to word meaning. ToV establishes relations between labels and referents for listeners to exploit in word learning