This study presents a new language identification model for pluricentric languages that uses n-gram language models at the character and word level. The model is evaluated in two steps. The first step consists of the identification of two varieties of Spanish (Argentina and Spain) and two varieties of French (Quebec and France) evaluated independently in binary classification schemes. The second step integrates these language models in a six-class classification with two Portuguese varieties

Diwersy, S.

Gebre, B.

Zampieri, M.

MPG.PuRe

Swedish Language Technology Conference 201279Classifying Pluricentric Languages: Extending the Monolingual ModelMarcos Zampieri1, Binyam Gebrekidan Gebre2, Sascha Diwersy3University of Cologne1,3, Max Planck Institute for Psycholinguistics1Albertus-Magnus-Platz 1, 50931, Cologne, Germany1,3Wundtlaan 1, 6525 XD, Nijmegen, Holland2mzampier@uni-koeln.de, bingeb@mpi.nl, sascha.diwersy@uni-koeln.deAbstractThis study presents a new language identification model for pluricentric languages that uses n-gram language models at thecharacter and word level. The model is evaluated in two steps. The first step consists of the identification of two varieties ofSpanish (Argentina and Spain) and two varieties of French (Quebec and France) evaluated independently in binary classificationschemes. The second step integrates these language models in a six-class classification with two Portuguese varieties.1. IntroductionAutomatic Language Identification is a well-known re-search topic in NLP. State-of-the-art methods consist of theapplication of n-gram language models to distinguish a setof languages automatically. One of the first studies to ap-ply n-gram models to language identification was Dunning(1994) and more recent experiments include Martins andSilva (2005) and Lui and Baldwin (2012).Martins and Silva use internet data to identify a set of 12languages and they report results ranging from 80% to 99%accuracy depending on the language. Lui and Baldwin re-port 91.3% for a set of 67 languages using Wikipedia data.Their software, called langid.py, claims to have languagemodels for 97 languages by using various data sources.These two examples do not take language varieties intoaccount. Pluricentric languages are always modeled as aunique class and this is one reason for the good results thesen-gram methods report. The level of success is usually highwhen classification includes languages which are typolog-ically not closely related (e.g. Finnish and Spanish) andlanguages with unique character sets (e.g. Hebrew).1.1 Classifying VarietiesOnly recently the automatic identification of language vari-eties has received more attention. A few studies have beenpublished such as Ljubesic et al. (2007) for the formerSerbo-Croatian varieties, Huang and Lee (2008) for Main-land and Taiwanese Chinese and Trieschnigg et al. (2012)for Dutch dialects.These methods aim to distinguish varieties and to ourknowledge none of them has yet been integrated into abroader language identification setting. Here we try toreplicate the experiments carried out by Zampieri and Ge-bre (2012) for Brazilian and European Portuguese on twovarieties of French and two of Spanish. Subsequently, weintegrate these four new language models into a six-classclassification scheme.2. MethodsWe compiled four journalistic corpora containing textsfrom each of the four varieties. To create comparable sam-ples, we retrieved texts published in the same year (2001for French and 2008 for Spanish) and all metainformationand tags were removed. The length of texts in the corporavaries and as language identification benefits from longertexts, we only used texts consisting of up to 300 tokens.Location Newspaper YearArgentina La Nacio´n 2008Spain El Mundo 2008France Le Monde 2001Quebec Le Devoir 2001Table 1: CorporaThe identification method works on three different aspectsof language: orthography, lexicon and lexico/syntax. Forthe orthographical differences, we used character n-gramsranging from 2 to 6-grams. At the lexical level we usedword uni-grams and finally, to explore lexico-syntacticaldifferences, word bi-grams were used. The language mod-els were calculated with Laplace probability distribution:Plap(w1...wn) =C(w1...wn) + 1N +B(1)In equation number 1: C is the count of the frequency ofw1to w2 in the training data, N is the total number of n-gramsandB is the number of distinct n-grams in the training data.For probability estimation, we used the log-likelihood func-tion:P (L|text) = argmaxLN∑i=1logP (ni|L) + logP (L) (2)N is the number of n-grams in the test text, ni is the ithn-gram and L stands for the language models. Given a testtext, we calculate the probability for each of the languagemodels. The language model with higher probability deter-mines the identified language of the text.3. ResultsEvaluation was done using each of the feature groups ina set of 1,000 documents sampled randomly. The samplecontains 50% of the texts from each variety and it is dividedinto 500 documents for training and 500 for testing.Swedish Language Technology Conference 2012803.1 Binary ClassificationWe report results in terms of accuracy for binary classifica-tion as seen in table 2.Feature AR x ES FR x QUWord 1-grams 0.948 0.968Word 2-grams 0.894 0.956Character 2-grams 0.898 0.956Character 3-grams 0.948 0.990Character 4-grams 0.944 0.968Character 5-grams 0.962 0.960Character 6-grams 0.960 0.934Table 2: Binary ClassificationThe results suggest that, on average, the French corporahave a stronger variation than the two varieties of Spanish.French scores were higher in most groups of features exceptcharacter 5 and 6-grams. The results for French and Span-ish are, however, lower than those obtained by Portuguese,which reached 0.998 accuracy for character 4-grams.3.2 Identifying Varieties or Identifying Newspapers?Studies on identification of language varieties use standardcorpora sampled from newspapers and magazines (Huang,2008). They do not, however, address the question of tex-tual genres and stylistics that underlie these samples. News-papers and magazines could contain distinctive features thatinfluence the performance of the classifiers. To explore thisvariable we carried out a controlled experiment using twonewspapers from Spain. A corpus with texts from El Paı´s,published in 2008, was compiled and classified with the ElMundo corpus.Feature Mundo x Paı´s DifferenceW 1-grams 0.614 -33.4%W 2-grams 0.498 -39.6%C 2-grams 0.658 -24.0%C 3-grams 0.654 -29.4%C 4-grams 0.728 -21.6%C 5-grams 0.688 -27.4%C 6-grams 0.564 -39.6%Table 3: El Mundo x El PaisResults are 21.6% to 39.6% worse than the classifica-tion of Argentinian and Peninsular Spanish. In one case,word bi-grams, the classification result is lower than the50% baseline expected for binary classification. The poorresults obtained suggest that the language models appliedhere are actually distinguishing the varieties and that thetext types and genres do not substantialy influence the al-gorithm’s choice.3.3 Multilingual ClassificationTo evaluate the classification model we integrated the fourlanguage models described so far: Spain, Argentina, Franceand Quebec, with two Portuguese varieties: Brazil and Por-tugal (Zampieri and Gebre, 2012). The results for this six-class classification model are presented in terms of Accu-racy, Recall, Precision and F-Measure.Feature A R P FW 1-grams 0.917 0.917 0.905 0.911W 2-grams 0.878 0.878 0.866 0.872C 2-grams 0.898 0.898 0.880 0.889C 3-grams 0.947 0.947 0.933 0.940C 4-grams 0.910 0.910 0.890 0.899C 5-grams 0.924 0.924 0.905 0.915C 6-grams 0.935 0.935 0.932 0.933Table 4: 6-Class Classification4. Conclusion and Future WorkStudies on language indentification neglect pluricentric lan-guages. We therefore presented a language identificationmodel focusing on language varieties. The best results onthe binary classification were obtained by using character n-grams. For Argentinian and Peninsular Spanish, 0.962 us-ing 5-grams and for Quebec and Hexagonal French, 0.990using 3-grams. The six-class model reached 0.947 accuracyand 0.940 f-measure using character 3-grams.This work shed light on two areas. First, it shows thatlanguage identification models may include language vari-eties without substantial loss of performance. This shouldhelp to increase performance in NLP applications such asspell checking and MT systems. The second area is con-trastive linguistics. Pluricentric languages are often the ob-ject of study due to their variation in grammar and syntax.Classification experiments such as this can provide a quan-titative overview on how varieties diverge and converge.Further experiments include the integration of thesemodels into broader classification schemes (up to 20-fold)and the use of more knowledge-rich features such as POSbi-grams, to measure the extent to which these varieties dif-fer in terms of grammar.5. ReferencesT. Dunning. 1994. Statistical Identification of LanguageTechnical Report MCCS-94-273 New Mexico StateUniversityC. Huang; L. Lee. 2008 Contrastive Approach towardsText Source Classification based on Top-Bag-of-WordSimilarity Proceedings of PACLIC 2008 p. 404-410N. Ljubesic; N. Mikelic; D. Boras. 2007. Language Iden-tificaiton: How to Distinguish Similar Languages? Pro-ceedings of the 29th International Conference on Infor-mation Technology InterfacesM. Lui; T. Baldwin. 2012 langid.py: An Off-the-shelf Lan-guage Identification Tool Proceedings of the 50th Meet-ing of the ACL p. 25-30B. Martins; M. Silva. 2005. Language Identification inWeb Pages Proceedings of the 20th ACM Symposium onApplied Computing (SAC) Santa Fe, EUA. p. 763-768D. Trieschnigg; D. Hiemstra; M. Theune; F. de Jong; T.Meder. 2012. An exploration of language identificationtechniques for the Dutch Folktale Database. Proceedingsof LREC2012M. Zampieri; B. G. Gebre. 2012. Automatic Identificationof Language Varieties: The Case of Portuguese Proceed-ings of KONVENS2012. Vienna, Austria. p. 233-237

Classifying pluricentric languages: Extending the monolingual model

Abstract

Similar works

Full text

Available Versions

MPG.PuRe