28 research outputs found
MorphTE: Injecting Morphology in Tensorized Embeddings
In the era of deep learning, word embeddings are essential when dealing with
text tasks. However, storing and accessing these embeddings requires a large
amount of space. This is not conducive to the deployment of these models on
resource-limited devices. Combining the powerful compression capability of
tensor products, we propose a word embedding compression method with
morphological augmentation, Morphologically-enhanced Tensorized Embeddings
(MorphTE). A word consists of one or more morphemes, the smallest units that
bear meaning or have a grammatical function. MorphTE represents a word
embedding as an entangled form of its morpheme vectors via the tensor product,
which injects prior semantic and grammatical knowledge into the learning of
embeddings. Furthermore, the dimensionality of the morpheme vector and the
number of morphemes are much smaller than those of words, which greatly reduces
the parameters of the word embeddings. We conduct experiments on tasks such as
machine translation and question answering. Experimental results on four
translation datasets of different languages show that MorphTE can compress word
embedding parameters by about 20 times without performance loss and
significantly outperforms related embedding compression methods.Comment: 20 pages, 6 figures, 18 tables. Published at NeurIPS 202
Towards learning word representation
Continuous vector representations, as a distributed representations for words have gained a lot of attention in Natural Language Processing (NLP) field. Although they are considered as valuable methods to model both semantic and syntactic features, they still may be improved. For instance, the open issue seems to be to develop different strategies to introduce the knowledge about the morphology of words. It is a core point in case of either dense languages where many rare words appear and texts which have numerous metaphors or similies. In this paper, we extend a recent approach to represent word information. The underlying idea of our technique is to present a word in form of a bag of syllable and letter n-grams. More specifically, we provide a vector representation for each extracted syllable-based and letter-based n-gram, and perform concatenation. Moreover, in contrast to the previous method, we accept n-grams of varied length n. Further various experiments, like tasks-word similarity ranking or sentiment analysis report our method is competitive with respect to other state-of-theart techniques and takes a step toward more informative word representation construction
Using deep learning models for learning semantic text similarity of Arabic questions
Question-answering platforms serve millions of users seeking knowledge and solutions for their daily life problems. However, many knowledge seekers are facing the challenge to find the right answer among similar answered questions and writer’s responding to asked questions feel like they need to repeat answers many times for similar questions. This research aims at tackling the problem of learning the semantic text similarity among different asked questions by using deep learning. Three models are implemented to address the aforementioned problem: i) a supervised-machine learning model using XGBoost trained with pre-defined features, ii) an adapted Siamese-based deep learning recurrent architecture trained with pre-defined features, and iii) a Pre-trained deep bidirectional transformer based on BERT model. Proposed models were evaluated using a reference Arabic dataset from the mawdoo3.com company. Evaluation results show that the BERT-based model outperforms the other two models with an F1=92.99%, whereas the Siamese-based model comes in the second place with F1=89.048%, and finally, the XGBoost as a baseline model achieved the lowest result of F1=86.086%