FreeLing is an open-source multilingual language processing library providing a wide range of language analyzers for
several languages. It offers text processing and language annotation facilities to natural language processing application
developers, simplifying the task of building those applications. FreeLing is customizable and extensible. Developers can
use the default linguistic resources (dictionaries, lexicons, grammars, etc.) directly, or extend them, adapt them to specific
domains, or even develop new ones for specific languages.
This paper overviews the recent history of this tool, summarizes the improvements and extensions incorporated in the latest
version, and depicts the architecture of the library. Special focus is brought to the fact and consequences of the library
being open-source: After five years and over 35,000 downloads, a growing user community has extended the initial three
languages (English, Spanish and Catalan) to eight (adding Galician, Italian, Welsh, Portuguese, and Asturian), proving that
the collaborative open model is a productive approach for the development of NLP tools and resources.Postprint (published version

Castellón, Irene

Collado, Miquel

Lloberes, Marina

Padró, Lluís

Reese, Samuel

English

FreeLing is an open-source multilingual language processing library providing a wide range of language analyzers for

several languages. It offers text processing and language annotation facilities to natural language processing application

developers, simplifying the task of building those applications. FreeLing is customizable and extensible. Developers can

use the default linguistic resources (dictionaries, lexicons, grammars, etc.) directly, or extend them, adapt them to specific

domains, or even develop new ones for specific languages.

This paper overviews the recent history of this tool, summarizes the improvements and extensions incorporated in the latest

version, and depicts the architecture of the library. Special focus is brought to the fact and consequences of the library

being open-source: After five years and over 35,000 downloads, a growing user community has extended the initial three

languages (English, Spanish and Catalan) to eight (adding Galician, Italian, Welsh, Portuguese, and Asturian), proving that

the collaborative open model is a productive approach for the development of NLP tools and resources

UPCommons

FreeLing 2.1: Five Years of Open-Source Language Processing Tools

UPCommons. Portal del coneixement obert de la UPC

FreeLing 2.1: Five years of open-source language processing toolsLluı´s Padro´∗, Miquel Collado∗, Samuel Reese⋄, Marina Lloberes†, Irene Castello´n†∗Software Department – TALP Research CenterUniversitat Polite`cnica de Catalunyapadro@lsi.upc.edu, miquel.collado@gmail.com⋄ISAE – SupaeroUniversite´ Paul Sabatiersamuel.reese@supaero.org†GRIAL research groupUniversitat de Barcelona{marina.lloberes,icastellon}@ub.eduAbstractFreeLing is an open-source multilingual language processing library providing a wide range of language analyzers forseveral languages. It offers text processing and language annotation facilities to natural language processing applicationdevelopers, simplifying the task of building those applications. FreeLing is customizable and extensible. Developers canuse the default linguistic resources (dictionaries, lexicons, grammars, etc.) directly, or extend them, adapt them to specificdomains, or even develop new ones for specific languages.This paper overviews the recent history of this tool, summarizes the improvements and extensions incorporated in the latestversion, and depicts the architecture of the library. Special focus is brought to the fact and consequences of the librarybeing open-source: After five years and over 35,000 downloads, a growing user community has extended the initial threelanguages (English, Spanish and Catalan) to eight (adding Galician, Italian, Welsh, Portuguese, and Asturian), proving thatthe collaborative open model is a productive approach for the development of NLP tools and resources.1. IntroductionBasic language processing tasks such as tokenizing,morphological analysis, lemmatizing, part-of-speechtagging, word sense disambiguation (WSD), depen-dency parsing, etc. are a need for most natural lan-guage processing (NLP) applications such as MachineTranslation, Summarization, Dialogue systems, Textmining, etc.This makes language analyzers very valuable re-sources for researchers and developers in NLP. Also,the lack of out-of-the-box state-of-the-art systems is asevere bottleneck for faster progress in the area, bothin research and development.FreeLing was undertaken with the believe that stepsshould be taken towards general availability of basicNLP tools and resources, which may be used withoutrestrictions. Thus, to enable faster advances and moreportable systems in our area, an open–source modelwas chosen.After five years (first version was released on 2004),over 35,000 downloads, and a growing user commu-nity which has extended the initial three languages(English, Spanish and Catalan) to eight (adding Gali-cian, Italian, Welsh, Portuguese, and Asturian) provethat the collaborative open model is a productive ap-proach for the development of NLP tools and re-sources.In the next section we review FreeLing evolution inits five-year life. Section 3 describes the most recentenhancements to the library, and Section 4 presents ashallow technical overview of the library. Section 5outlines some conclusions and further work.2. FreeLing project evolutionThe first version of FreeLing (1.0) was released in2003, and presented to the community in LREC 2004(Carreras et al., 2004). That first version was a re-implementation of several tools (Atserias et al., 1998;Carreras and Padro´, 2002) developed at the TALP re-search center1 in UPC2.The new version, was rewritten from scratch with thefollowing goals:• Build a fast state-of-the-art analyzer that could beused to process large amounts of text and to developNLP applications.1http://www.talp.upc.edu2http://www.upc.edu• Disseminate the results of the NLP research atTALP center, and make it available to the commu-nity• Set a collaborative environment for the develop-ment of language analyzers, where contributionsfrom any NLP community member can fit and re-vert in a global benefit.To achieve the these goals, the tool was structuredas a highly modular C++ library consisting of sev-eral classes, each able to perform a different kind ofanalysis. Under this approach, new classes or servicescan be added to the library, and applications can selectwhich services will be used, how, and when.The goals related to dissemination and communitycollaboration were achieved releasing the software un-der an open-source GPL license, which has made itpossible to enrich the project with many contributions,and make the tool useful for a larger number of people,as the increasing number of downloads suggests.Currently, the number of downloads of the libraryis over 35,000 since it was first released, showing asteady growth at each release, and a large expansionat the latest 2.1 version (see Table 1).total monthlyVersions #downl average#downl1.0, 1.0.1, Oct.03–Sep.04 608 541.11.2, 1.3, Oct.04–Sep.06 2,250 961.4, 1.5b11.5 Oct.06–Jan.08 2,451 1612.0, 2.1a1, Feb.08–Sep.09 3,297 1632.1b1,2.1a1, 2.1 Oct.09–Mar.10 26,713 4,452Table 1: Evolution of FreeLing downloadsThe original three supported languages (Catalan,Spanish, and English) have been brought up to eightby this community, with the inclusion of Italian, Gali-cian, Portuguese, Welsh, and Asturian. Also, the orig-inal size-limited dictionaries for Catalan and Spanishhave been replaced by much larger state-of-the art re-sources.Finally, the increasing availability of other open-source language analysis tools has enabled us to ex-tend the library capabilities with long-awaited servicessuch as Word Sense Disambiguation (WSD), obtainedthanks to the inclusion of the UKB disambiguator de-veloped by (Agirre and Soroa, 2009). See (Padro´ etal., 2010) for details on its integration in FreeLing.We regard this evolution as very satisfactory, since it isfulfilling our goals of creating a useful tool which canbe enlarged by the community, used by application de-velopers, and become a flexible platform enabling theintegration of new resources and services.3. Recent extensionsThe most recent extensions included in FreeLing li-brary can be classified in three types: extensions forlinguistic resources for existing languages, support fornew languages, and processing modules offering newservices.3.1. Improving existing languagesAs in most open-source projects, a large part of FreeL-ing contributions are related to the enhancement of al-ready existing features. In our case, noticeable contri-butions include:• Development of a number recognition module forItalian, which was not present in previous versions,thanks to Vitalie Scurtu3.• Enlargement of Spanish dictionary from 6,000 to75,000 lemmas, thanks to the Spanish ResourceGrammar (SRG) project (Marimon et al., 2007).• Enlargement of the Catalan dictionary from 7,000to 71,000 lemmas, thanks to the Catalan spellerproject El Corrector4.• Enlargement of the Galician dictionary from 7,000to 50,000 lemmas, thanks to the PLN researchgroup at Universidade de Santiago de Compostela5.• Development and improvement of Spanish, Cata-lan, and English chunking and dependency gram-mars by the GRIAL research group6 (Carrera et al.,2008), and thanks to EuroOpenTrad and KNOWprojects, funded by Spanish Government3.2. Support for new languagesAnother important bulk of contributions are those de-voted to include support for new languages in FreeL-ing. This task requires on the one hand, the compi-lation of a morphological dictionary which providespossible lemmas and PoS tags for each form, and onthe other hand, the development of a hand tagged cor-pus (consistent with the dictionary) to enable the train-ing of the PoS taggers. In addition, other minor adjust-ments have to be done, such as adapting the tokenizerrules to the particularities of the language, writing ap-propriate rules to deal with suffixation, etc.Languages that have been recently included are Welsh,Asturian, and Portuguese. All of them offer a chain ofprocessing from plain text up to PoS tagging.3http://scurtu.sitonline.it4http://www.elcorrector.cat5http://gramatica.usc.es6http://grial.uab.es3.3. Modules offering new servicesThere are several recently added functionalities, someare small (such as the possibility of prefix handling inmorphological analysis), some are entire new modulesoffering a new NLP analysis task, such as Word SenseDisambiguation or Coreference Resolution:• A language-independent WSD module has beenintegrated. The code is straightforwardly ex-tracted from the UKB disambiguator7 developed by(Agirre and Soroa, 2009). The only requirement touse that module on a certain language is the exis-tence of a WordNet-like ontology that relates thewords to senses, and the senses among themselves.• A coreference resolution module has been devel-oped from scratch, using Machine Learning tech-niques. The approach is based on that of (Soon etal., 2001), and currently, trained models are pro-vided only for Spanish (still with under state-of-the-art accuracy).4. Data structure and language analysisservicesFreeLing is conceived as a library on top of whichpowerful NLP applications can be developed, and ori-ented to ease the integration of language analysis ser-vices into higher level applications.Its architecture consists of a simple two-layer client-server approach: A basic linguistic service layerwhich provides analysis services (morphological anal-ysis, tagging, parsing, ...), and an application layerwhich, acting as a client, requests the desired servicesfrom the analyzers.The internal architecture of the system is based on twokinds of objects: linguistic data objects and processingobjects.4.1. Linguistic Data ClassesThe basic classes in the library are used to contain lin-guistic data (such as a word, a PoS tag, a sentence,a document...). Any client application must be awareof those classes in order to be able to provide to eachprocessing module the right data, and to correctly in-terpret the module results.The linguistic classes supported by the current versionare:• analysis: A tuple <lemma, PoS tag, probability,sense list>.• word: A word form with a list of possibleanalysis objects.7http://ixa2.si.ehu.es/ukb/• sentence: A list of word known to be a com-plete sentence, it may include also a parse treeand/or a dependency tree.• paragraph: A list of sentence known to be anindependent paragraph.• document: A list of paragraph that form acomplete document. It may contain also corefer-ence information about the entity mentions in thedocument.Figure 1 presents a UML diagram with the linguisticdata classes.Figure 1: FreeLing-2.1 Linguistic Data Classes.4.2. Processing ClassesApart from classes containing linguistic data, the li-brary provides classes able to transform them. A UMLdiagram can be found in Figure 2.• tokenizer: Receives plain text and returns a listof word objects.• splitter: Receives a list of word objects andreturns a list of sentence objects.Figure 2: FreeLing-2.1 Main Processing Classes.• morfo: Receives a list of sentence and mor-phologically annotates each word of each sentencein the list. In fact, this class applies a cascade ofspecialized processors (number detection, date/timedetection, multi-word detection, dictionary search,etc.) each of which is in turn a processing class:– locutions: Multi-word recognizer.– dictionary: Dictionary lookup and suffixhandling.– numbers: Numerical expressions recognizer.– dates: Date/time expressions recognizer.– quantities: Ratio and percentage expres-sions and monetary amount recognizer.– punts: Punctuation symbol annotator.– probabilities: Lexical probabilities anno-tator and unknown word handler.– np: Proper noun recognizer. Two modules areprovided for this task. A fast and simple pattern–matching module based on capitalization (whichyields an accuracy near 90%), and a NE recog-nizer based on the CoNLL-2002 shared task win-ning system (Carreras et al., 2002), rather slower,but with an accuracy over 92%.• tagger: Receives a list of sentence and dis-ambiguates the PoS of each word in the given sen-tences. If the selected analysis carries retokeniza-tion information, the word may be split in two ormore new words. FreeLing offers two PoS taggerswith state-of-the-art accuracy (about 97%): OneHMM–based following (Brants, 2000) and anotherbased on relaxation labelling (Padro´, 1998).• NE classifier: Receives a list of sentenceand classifies all word tagged as proper nouns inthe given sentences. This module is based on theCoNLL-2002 shared task winning system (Carreraset al., 2002).• Sense annotator: Receives a list ofsentence and adds synset information tothe selected analysis for each word.• Word sense disambiguator: Receives alist of sentence and ranks the possible senses forfor each word selected analysis. This moduleis a direct inclusion of the UKB system (Agirre andSoroa, 2009).• chunk parser: Receives a list of sentenceand enriches each of them with a parse tree.This module consists of a chart parser, and is a reim-plementation of (Atserias and Rodrı´guez, 1998).• dependency parser: Receives a list of parsedsentence and enriches each of them with adependency tree. This module uses a set ofhand–written rules to build a dependency tree. Itsoriginal version is described in (Atserias et al.,2005), though several extensions to the expressivepower of the rules have been added.• coreference solver: Receives a documentformed by parsed sentence and enriches it withcoreference information. This module is based onthe system proposed by (Soon et al., 2001).5. ConclusionsWe reviewed the first five years of FreeLing project,an open-source library of language analyzers, whichis accomplishing its original goals of community par-ticipation, international dissemination, and utility bothto academy and industry.We also described the main enhancements included inversion 2.1, and shallowly described the internal ar-chitecture of the libraryFurther work will involve consolidating the commu-nity, as well as keep including new functionalities andlanguages to the library.AcknowledgementsThis work has been partially funded by the SpanishGovernment via the KNOW2 (TIN2009-14715-C04-03/04) project.6. ReferencesEneko Agirre and Aitor Soroa. 2009. Personalizingpagerank for word sense disambiguation. In Pro-ceedings of the 12th conference of the Europeanchapter of the Association for Computational Lin-guistics (EACL-2009), Athens, Greece.Jordi Atserias and Horacio Rodrı´guez. 1998. Tacat:Tagged corpus analizer tool. Technical report lsi-98-2-t, Departament de LSI. Universitat Polite`cnicade Catalunya.Jordi Atserias, Josep Carmona, Irene Castello´n,Sergi Cervell, Montserrat Civit, Lluı´s Ma`rquez,Ma Anto`nia Martı´, Lluı´s Padro´, Roberto Placer, Ho-racio Rodrı´guez, Mariona Taule´, and Jordi Turmo.1998. Morphosyntactic analysis and parsing of un-restricted spanish text. In Proceedings of the 1stInternational Conference on Language Resourcesand Evaluation, LREC, pages 1267–1274, Granada,Spain, May.Jordi Atserias, Elisabet Comelles, and Aingeru Mayor.2005. Txala un analizador libre de dependenciaspara el castellano. Procesamiento del LenguajeNatural, (35):455–456, September.Thorsten Brants. 2000. Tnt - a statistical part- of-speech tagger. In Proceedings of the 6th Conferenceon Applied Natural Language Processing, ANLP.ACL.Jordi Carrera, Irene Castello´n, Marina Lloberes, Lluı´sPadro´, and Nevena Tinkova. 2008. Dependencygrammars in freeling. Procesamiento del LenguajeNatural, (41):21–28, September.Xavier Carreras and Lluı´s Padro´. 2002. A flexible dis-tributed architecture for natural language analyzers.In Proceedings of the 3rd International Conferenceon Language Resources and Evaluation, LREC, LasPalmas de Gran Canaria, Spain.Xavier Carreras, Lluı´s Ma`rquez, and Lluı´s Padro´.2002. Named entity extraction using adaboost. InProceedings of CoNLL Shared Task, pages 167–170, Taipei, Taiwan.Xavier Carreras, Isaac Chao, Lluı´s Padro´, and MuntsaPadro´. 2004. Freeling: An open-source suite oflanguage analyzers. In Proceedings of the 4th In-ternational Conference on Language Resources andEvaluation (LREC’04).Montserrat Marimon, Nu´ria Bel, Sergio Espeja, andNatalia Seghezzi. 2007. The spanish resourcegrammar: pre-processing strategy and lexical ac-quisition. In Proceedings of the Workshop on DeepLinguistic Processing, Association for Computa-tional Linguistics (ACL-DLP).Lluı´s Padro´, Samuel Reese, Eneko Agirre, and AitorSoroa. 2010. Semantic services in freeling 2.1:Wordnet and ukb. In Proceedings of the 5th Inter-national Conference of the Global WordNet Associ-ation (GWC-2010), Bombay, India, February.Lluı´s Padro´. 1998. A Hybrid Environment for Syntax–Semantic Tagging. Ph.D. thesis, Dep. Llenguatgesi Sistemes Informa`tics. Universitat Polite`cnica deCatalunya, February. http://www.lsi.upc.es/˜padro.W.M. Soon, H. T. Ng, and D.C.Y. Lim. 2001. Amachine learning approach to coreference resolu-tion of noun phrases. Computational Linguistics,27(4):521–544.

https://upcommons.upc.edu/bitstream/2117/7616/6/padro10b.pdf

FreeLing 2.1: Five Years of Open-Source Language Processing Tools

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC