From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

Over the past years, distributed semantic representations have proved to be effective and flexible keepers of prior knowledge to be integrated into downstream applications. This survey focuses on the representation of meaning. We start from the theoretical background behind word vector space models and highlight one of their major limitations: the meaning conflation deficiency, which arises from representing a word with all its possible meanings as a single vector. Then, we explain how this deficiency can be addressed through a transition from the word level to the more fine-grained level of word senses (in its broader acceptation) as a method for modelling unambiguous lexical meaning. We present a comprehensive overview of the wide range of techniques in the two main branches of sense representation, i.e., unsupervised and knowledge-based. Finally, this survey covers the main evaluation procedures and applications for this type of representation, and provides an analysis of four of its important aspects: interpretability, sense granularity, adaptability to different domains and compositionality.Comment: 46 pages, 8 figures. Published in Journal of Artificial Intelligence Researc

Embeddings for word sense disambiguation: an evaluation study

Recent years have seen a dramatic growth in the popularity of word embeddings mainly owing to their ability to capture semantic information from massive amounts of textual content. As a result, many tasks in Natural Language Processing have tried to take advantage of the potential of these distributional models. In this work, we study how word embeddings can be used in Word Sense Disambiguation, one of the oldest tasks in Natural Language Processing and Artificial Intelligence. We propose different methods through which word embeddings can be leveraged in a state-of-the-art supervised WSD system architecture, and perform a deep analysis of how different parameters affect performance. We show how a WSD system that makes use of word embeddings alone, if designed properly, can provide significant performance improvement over a state-of-the-art WSD system that incorporates several standard WSD features

Joint Approaches for Learning Word Representations from Text Corpora and Knowledge Bases

The work presented in this thesis is directed at investigating the possibility of combining text corpora and Knowledge Bases (KBs) for learning word representations. More specifically, the aim was to propose joint approaches that leverage the two types of resources for the purpose of enhancing the word meaning representations. The main research question to be answered was “Is it possible to enhance the word representations by jointly incorporating text corpora and KBs into the word representations learning process? If so, what are the aspects of word meaning that can be enhanced by combining those two types of resources? ”. The primary contribution of the thesis is three main joint approaches for learning word representations: (i) Joint Representation Learning for Additional Evidence (JointReps), (ii) Joint Hierarchical Word Representation (HWR) and (iii) Sense-Aware Word Representations (SAWR). The JointReps was founded to improve the overall semantic representation of words. To this end, it sought additional evidence from a KB to the co-occurrence statistics in the corpus. In particular, JointReps enforced two words that are in a particular semantic relationship in the KB to have similar word representations. The HWR approach was then proposed to learn word representations in a specific order to encode the hierarchical information in a KB in the learnt representations. The HWR considered not only the hypernym relations that exist between words in a KB, but also contextual information in a text corpus. Specifically, given a training corpus and a KB, HWR learnt word representations that simultaneously encoded the hierarchical structure in the KB as well as the co-occurrence statistics between pairs of words in the corpus. A particularly novel aspect of the HWR approach was that it exploits the full hierarchical path of words existing in the KB. The SAWR approach was then introduced to consider not only word representations but also the different senses (different meanings) associated with each word. The SAWR required the learnt representations to predict the word and the senses accurately. It learnt the sense-aware word representations jointly using both unlabelled and sense-labelled text corpora. The approaches were comprehensively analysed and evaluated in various standard and newly-proposed tasks using a wide range of benchmark datasets. The evaluation was conducted to compare the quality of the learnt word representations by the proposed approaches with word representations learnt by sole-resource baselines and previously proposed joint approaches in the literature. All the proposed joint approaches have proven to be effective for enhancing the learnt word representations. More specifically, the proposed joint approaches were found to report significant improvements over the approaches that use only one type of resources and the previously proposed joint approaches

Driving Context into Text-to-Text Privatization

\textit{Metric Differential Privacy} enables text-to-text privatization by adding calibrated noise to the vector of a word derived from an embedding space and projecting this noisy vector back to a discrete vocabulary using a nearest neighbor search. Since words are substituted without context, this mechanism is expected to fall short at finding substitutes for words with ambiguous meanings, such as \textit{'bank'}. To account for these ambiguous words, we leverage a sense embedding and incorporate a sense disambiguation step prior to noise injection. We encompass our modification to the privatization mechanism with an estimation of privacy and utility. For word sense disambiguation on the \textit{Words in Context} dataset, we demonstrate a substantial increase in classification accuracy by $6.05\%$

Embedding-based real-time change point detection with application to activity segmentation in smart home time series data

[EN]Human activity recognition systems are essential to enable many assistive applications. Those systems can be sensor-based or vision-based. When sensor-based systems are deployed in real environments, they must segment sensor data streams on the fly in order to extract features and recognize the ongoing activities. This segmentation can be done with different approaches. One effective approach is to employ change point detection (CPD) algorithms to detect activity transitions (i.e. determine when activities start and end). In this paper, we present a novel real-time CPD method to perform activity segmentation, where neural embeddings (vectors of continuous numbers) are used to represent sensor events. Through empirical evaluation with 3 publicly available benchmark datasets, we conclude that our method is useful for segmenting sensor data, offering significant better performance than state of the art algorithms in two of them. Besides, we propose the use of retrofitting, a graph-based technique, to adjust the embeddings and introduce expert knowledge in the activity segmentation task, showing empirically that it can improve the performance of our method using three graphs generated from two sources of information. Finally, we discuss the advantages of our approach regarding computational cost, manual effort reduction (no need of hand-crafted features) and cross-environment possibilities (transfer learning) in comparison to others.This work was carried out with the financial support of FuturAALEgo (RTI2018-101045-A-C22) granted by Spanish Ministry of Science, Innovation and Universities

Measuring associational thinking through word embeddings

[EN] The development of a model to quantify semantic similarity and relatedness between words has been the major focus of many studies in various fields, e.g. psychology, linguistics, and natural language processing. Unlike the measures proposed by most previous research, this article is aimed at estimating automatically the strength of associative words that can be semantically related or not. We demonstrate that the performance of the model depends not only on the combination of independently constructed word embeddings (namely, corpus- and network-based embeddings) but also on the way these word vectors interact. The research concludes that the weighted average of the cosine-similarity coefficients derived from independent word embeddings in a double vector space tends to yield high correlations with human judgements. Moreover, we demonstrate that evaluating word associations through a measure that relies on not only the rank ordering of word pairs but also the strength of associations can reveal some findings that go unnoticed by traditional measures such as Spearman's and Pearson's correlation coefficients.s Financial support for this research has been provided by the Spanish Ministry of Science, Innovation and Universities [grant number RTC 2017-6389-5], the Spanish ¿Agencia Estatal de Investigación¿ [grant number PID2020-112827GB-I00 / AEI / 10.13039/501100011033], and the European Union¿s Horizon 2020 research and innovation program [grant number 101017861: project SMARTLAGOON]. Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.Periñán-Pascual, C. (2022). Measuring associational thinking through word embeddings. Artificial Intelligence Review. 55(3):2065-2102. https://doi.org/10.1007/s10462-021-10056-62065210255