SMTPOST: Using statistical machine translation approach in Filipino part-of-speech tagging

The field of Natural Language Processing (NLP) in the country has been continually developing. However, the transition between Tagalog to the progressing Filipino language left tools and resources behind. This paper introduces a Statistical Machine Translation Part-of-Speech (POS) Tagger for Filipino (SMTPOST), with the purpose of reviving, updating and widening the scope of technologies in the POS tagging domain, catering to the changes made by the Filipino language. Resources built are comprised mainly of a tagset (218 tags), parallel corpus (2,668 sentences), affix rules (59 rules) and word-tag dictionary (309 entries). SMTPOST was tested to different tagsets and domains, producing 84.75% as its highest accuracy score, at least 3.75% increase from the available Tagalog POS taggers. Despite SMTPOST\u27s utilization of Filipino resources and good performance, there are room for improvements and opportunities. Recommendations include a better feature extractor (preferably a morphological analyzer), an increase in scope for all of the resources, implementation of pre- And/or postprocessing, and the utilization of SMTPOST research to other NLP applications

Using Stanford part-of-speech tagger for the morphologically-rich Filipino Language

This research focuses on the implementation of a Maximum Entropy-based Part-of-Speech (POS) tagger for Filipino. It uses the Stanford POS tagger - a trainable POS tagger that has been trained on English, Chinese, Arabic, and other languages and producing one of the highest results in each language. The tagger was trained for Filipino using a 406k token corpus and considering unique Filipino linguistic phenomena such as high morphology and intra-sentential code-switches. The Filipino POS tagger resulted to 96.15% tagging accuracy which currently presents the highest accuracy and with a large lead among existing POS taggers for Filipino. Copyright © 2017 Matthew Phillip Go and Nicco Noco

Building a Filipino colloquialism translator using sequence-to-sequence model

Colloquialism in the Philippines has been prominently used in day-to-day conversations. Its vast emergence is evident especially on social media platforms but poses issues in terms of understandability to certain groups. For this research, machine translators have been implemented to fill in that gap. The translators cover Filipino Textspeak or Shortcuts, Swardspeak or Gay-lingo, Conyo, and Datkilab-implemented on Tensorflow library and Moses tool. Implementing in Tensorflow achieved 85.88 BLEU score when evaluated to the training data and 14.67 to the test data, while Moses garnered 95.27 BLEU score on training data and 79.91 on test data. Analyses on both implementations include advantages and disadvantages in using each one. Through the analyses and development of this research, it is recommended to implement the following in the future: addition of colloquialism samples, experimentation on sequence-to-sequence configurations, applying Graphical User Interface (GUI) to the translators, implementing the translators to Natural Language Processing (NLP) tools, and to deploy the translators as a web application

Gramatika: A grammar checker for the low-resourced Filipino language

This research focuses on the implementation of Gramatika, a grammar checker designed for the Filipino language given its available resources and linguistic tools. The checker uses hybrid n-grams generated from n-grams of words, part-of-speech tags, and lemmas of grammatically-correct texts. It covers a variety of error types including those unique in Filipino: wrong word form, and incorrectly merged/unmerged words. The grammar checker performed 64% accuracy on producing the correct suggestions on erroneous phrases and 85% on error-free texts when using Part-of-Speech (POS) tags from a Hybrid POS tagger (HPOST) for Filipino. Recommendations to improve Gramatika is to implement linguistic tools such as constituency parser, incorrect affix detection system, and a spell checker for the Filipino language

NormAPI: An API for normalizing Filipino shortcut texts

© 2014 IEEE. As the number of Internet and mobile phone users grow, texting and chatting have become popular means of communication. Reaching new heights, the extensive use of cellphones and Internet led into the creation of a new language, where words are transformed and made shorter using various styles. Shortcut texting is used in informal venues such as SMS, online, chat rooms, forums and posts in social networks. Huge amounts of data originating from these informal sources can be utilized for various tasks in machine learning and data analytics. As these data may be written in shortcut forms, text normalization is necessary before NLP actions such as information extraction, data mining, text summarization, opinion classification, and even bilingual translations can be fully achieved, by acting as a preprocessing stage that transforms all informal texts back to their original and more understandable forms. This paper is about NormAPI, an API for normalizing Filipino shortcut texts. NormAPI primarily intends to be used as a preprocessing system that corrects informalities in shortcut texts before they are handed for complete data processing

Philippine component of the network-based ASEAN language translation public service

Communication between different nations is essential. Languages which are foreign to another impose difficulty in understanding. For this problem to be resolved, options are limited to learning the language, having a dictionary as a guide, or making use of a translator. This paper discusses the development of ASEANMT-Phil, a phrase-based statistical machine translator, to be utilized as a tool beneficial for assisting ASEAN countries. The data used for training and testing came from Wikipedia articles comprising of 124,979 and 1,000 sentence pairs, respectively. ASEANMT-Phil was experimented on different settings producing the BLEU score of 32.71 for Filipino-English and 31.15 for English-Filipino. Future Directions for the translator includes the following: improvement of data through changing or adding the domain or size; implementing an additional approach; and utilizing a larger dictionary to the approach. © 2014 IEEE

NormAPI: An API for normalizing Filipino shortcut texts

As the number of Internet and mobile phone users grows, texting and chatting have become popular means of communication. Reaching new heights, the extensive use of cellphones and Internet led into the creation of a new language, where words are transformed and made shorter using various styles. Shortcut texting is used all over the world and in recent years, numerous researchers have created normalization systems in different languages that would transform shortcut texts back into their original forms. This research designed techniques and developed NormAPI, a system that will normalize Filipino shortcut texts. Focused on modern Filipino language which includes code-switching, the system primarily contributes to Natural Language Processing (NLP) research as a preprocessing system that corrects informalities in shortcut texts before they are handed for complete data processing. Functionalities include using four normalization variations namely, Dictionary Substitution Approach (DSA), Statistical Machine Translation (SMT), SMT after DSA and SMT before DSA, with 0.68384, 0.79650, 0.75634 and 0.80750 BLEU scores, respectively. Additionally, options such as setting the dictionary, generating language models, getting BLEU scores and more can be utilized by users based on their preferences