183 research outputs found
A Sentiment Analysis Dataset for Code-Mixed Malayalam-English
There is an increasing demand for sentiment analysis of text from social
media which are mostly code-mixed. Systems trained on monolingual data fail for
code-mixed data due to the complexity of mixing at different levels of the
text. However, very few resources are available for code-mixed data to create
models specific for this data. Although much research in multilingual and
cross-lingual sentiment analysis has used semi-supervised or unsupervised
methods, supervised methods still performs better. Only a few datasets for
popular languages such as English-Spanish, English-Hindi, and English-Chinese
are available. There are no resources available for Malayalam-English
code-mixed data. This paper presents a new gold standard corpus for sentiment
analysis of code-mixed text in Malayalam-English annotated by voluntary
annotators. This gold standard corpus obtained a Krippendorff's alpha above 0.8
for the dataset. We use this new corpus to provide the benchmark for sentiment
analysis in Malayalam-English code-mixed texts
Neural Sequence Labeling on Social Media Text
As social media (SM) brings opportunities to study societies across the world, it also brings a variety of challenges to automate the processing of SM language. In particular, most of the textual content in SM is considered noisy; it does not always stick to the rules of the written language, and it tends to have misspellings, arbitrary abbreviations, orthographic inconsistencies, and flexible grammar. Additionally, SM platforms provide a unique space for multilingual content. This polyglot environment requires modern systems to adapt to a diverse range of languages, imposing another linguistic barrier to processing and understanding of text from SM domains. This thesis aims at providing novel sequence labeling approaches to handle noise and linguistic code-switching (i.e., the alternation of languages in the same utterance) in SM text. In particular, the first part of this thesis focuses on named entity recognition for English SM text, where I propose linguistically-inspired methods to address phonological writing and flexible syntax. Besides, I investigate whether the performance of current state-of-the-art models relies on memorization or contextual generalization of entities. In the second part of this thesis, I focus on three sequence labeling tasks for code-switched SM text: language identification, part-of-speech tagging, and named entity recognition. Specifically, I propose transfer learning methods from state-of-the-art monolingual and multilingual models, such as ELMo and BERT, to the code-switching setting for sequence labeling. These methods reduce the demand for code-switching annotations and resources while exploiting multilingual knowledge from large pre-trained unsupervised models. The methods presented in this thesis are meant to benefit higher-level NLP applications oriented to social media domains, including but not limited to question-answering, conversational systems, and information extraction
- …