22 research outputs found
Homograph Disambiguation Through Selective Diacritic Restoration
Lexical ambiguity, a challenging phenomenon in all natural languages, is
particularly prevalent for languages with diacritics that tend to be omitted in
writing, such as Arabic. Omitting diacritics leads to an increase in the number
of homographs: different words with the same spelling. Diacritic restoration
could theoretically help disambiguate these words, but in practice, the
increase in overall sparsity leads to performance degradation in NLP
applications. In this paper, we propose approaches for automatically marking a
subset of words for diacritic restoration, which leads to selective homograph
disambiguation. Compared to full or no diacritic restoration, these approaches
yield selectively-diacritized datasets that balance sparsity and lexical
disambiguation. We evaluate the various selection strategies extrinsically on
several downstream applications: neural machine translation, part-of-speech
tagging, and semantic textual similarity. Our experiments on Arabic show
promising results, where our devised strategies on selective diacritization
lead to a more balanced and consistent performance in downstream applications.Comment: accepted in WANLP 201
Recommended from our members
Using automatic speech recognition to evaluate Arabic to English transliteration
Increased travel and international communication has led to an increased need for transliteration of Arabic proper names for people, places, technical terms and organisations. There are a variety of available Arabic to English transliteration systems such as Unicode, the Buckwalter Arabic transliteration, and ArabTeX. The transliteration tables have been developed and used by researchers for many years, but there are only limited attempts to evaluate and compare different transliteration systems. This thesis investigates whether or not speech recognition technology could be used to evaluate different Arabic-English transliteration systems. In order to do so there were 5 main objectives: firstly, to investigate the possibility of using English speech recognition engines to recognize Arabic words; secondly, to establish the possibility of automatic transliteration of diacritised Arabic words for the purpose of creating a vocabulary for the speech recognition engine; thirdly, to explore the possibility of automatically generating transliterations of non diacritised Arabic words; fourthly to construct a general method to compare and evaluate different transliteration; and finally, to test the system and use it to experiment with new transliterations ideas