Evaluation of language identification methods using 285 languages

Peer reviewe

Evaluating HeLI with non-linear mappings

Peer reviewe

Discriminating between Mandarin Chinese and Swiss-German varieties using adaptive language models

Peer reviewe

Language Set Identification in Noisy Synthetic Multilingual Documents

Proceeding volume: Part IIn this paper, we reconsider the problem of language identification of multilingual documents. Automated language identification algorithms have been improving steadily from the seventies until recent years. The current state-of-the-art language identifiers are quite efficient even with only a few characters and this gives us enough reason to again evaluate the possibility to use existing language identifiers for monolingual text to detect the language set of a multilingual document. We are using a previously developed language identifier for monolingual documents with the multilingual documents from the WikipediaMulti dataset published in a recent study. Our method outperforms previous methods tested with the same data, achieving an F 1-score of 97.6 when classifying between 44 languages.Peer reviewe

Language and Dialect Identification of Cuneiform Texts

Peer reviewe

HeLI, a Word-Based Backoff Method for Language Identification

In this paper we describe the Helsinki language identification method, HeLI, and the resources we created for and used in the 3rd edition of the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial 2016 workshop. The shared task comprised of a total of 8 tracks, of which we participated in 7. The shared task had a record number of participants, with 17 teams providing results for the closed track of the test set A. Our system reached the 2nd position in 4 tracks (A closed and open, B1 open and B2 open) and in this paper we are focusing on the methods and data used for those tracks. We describe our word-based back-off method in mathematical notation. We also describe how we selected the corpus we used in the open tracks.Peer reviewe

HeLI-based Experiments in Swiss German Dialect Identification

Peer reviewe

HeLI-based Experiments in Discriminating Between Dutch and Flemish Subtitles

Peer reviewe

Iterative Language Model Adaptation for Indo-Aryan Language Identification

Peer reviewe

Improving OCR of historical newspapers and journals published in Finland

Peer reviewe