In this paper we describe how Dicionáio-Aberto, an online dictionary for the Portuguese language, is being used as the base to construct diverse resources that are relevant in the processing of the Portuguese language. We will briefly present its history, explaining how we got here. Then, we will describe the resources already available to download and use, followed by the discussion on the resources that are being currently developed

Almeida, J. J.

Iriarte Sanromán, Álvaro

Simões, Alberto

Universidade do Minho: RepositoriUM

Dicionário-Aberto:A Source of Resources for the PortugueseLanguage ProcessingAlberto Simões1, Álvaro Iriarte Sanromán1, and José João Almeida21 Center for Humanistic Studies, Minho’s University{ambs,alvaro}@ilch.uminho.pt2 Computer Science and Technology Center, Minho’s Universityjj@di.uminho.ptAbstract. In this paper we describe how Dicionário-Aberto, an onlinedictionary for the Portuguese language, is being used as the base toconstruct diverse resources that are relevant in the processing of thePortuguese language.We will brieﬂy present its history, explaining how we got here. Then,we will describe the resources already available to download and use,followed by the discussion on the resources that are being currently de-veloped.Keywords: dictionary, lexicography, open resource, thesauri.1 A Brief HistoryDicionário-Aberto1 started in June 2005, when a few people felt that the Por-tuguese language was missing an open dictionary for use (any use). The processof creating a dictionary from scratch is diﬃcult and expensive. When most ofthe interested persons are engineers and computer scientists, this task gets morediﬃcult. As one member of this group was responsible for the transcriptionof Portuguese books for the Project Gutenberg [1], the idea of transcribing afull dictionary appeared. An old dictionary with expired copyright was searchedand chosen2, digitalized and the transcription process started using the ProjectGutenberg Distributed Proofreaders web interface. A detailed description of thisprocess is described in [8].The transcription was performed by volunteers, using a simple textual syntax,very similar to a subset of common wiki syntaxes. In March, 2010, the full tran-scriptionwas concluded (diﬀerent validation roundswere performed for eachpage).This textual document was converted to a more formal syntax, based on XML.1 Available at http://www.dicionario-aberto.net/2 The chosen dictionary was “Novo Diccionário da Língua Portuguesa, Cândido deFigueiredo, 1913.” It was not chosen by its lexicographic quality, but only becauseof a set of circumstantial facts.H. Caseli et al. (Eds.): PROPOR 2012, LNAI 7243, pp. 121–127, 2012.c© Springer-Verlag Berlin Heidelberg 2012122 A. Simões, Á.I. Sanromán, and J.J. AlmeidaA subset of the TEI [3] (Text Encoding Initiative) format for dictionaries waschosen. Simões and Almeida [9] describe this conversion process.The chosen dictionary used an old orthographic form (prior to 1943/45 agree-ments). To be part of Project Gutenberg the books must be transcribed inoriginal form, so the transcribed documents needed orthographic modernizationto be useful. This task was automated and at the present moment, a set of volun-teers are approving the modernized entries3 Dicionário-Aberto has now 128 521entries, and about 8% of the entries were veriﬁed for modernization errors.The new orthographic agreement (1990) will require a new modernizationprocess. Fortunately, this will be easier to automate, as there are a couple ofgood conversion tools available [2].In the future, Dicionário-Aberto will be open as a dictionary Wiki, where thecommunity can edit corrections or add new words. To guarantee quality (and asomewhat controlled language) a two tier process will be implemented: a changeor addition (or even deletion) will be available right away, but in a “non oﬃcial”status, until a moderator approves the change.2 Currently Available ResourcesWith the current status of Dicionário-Aberto as described in the previous sec-tion, there are some resources that can be downloaded and used to learn aboutthe Portuguese language, its history and to process automatically using naturallanguage processing techniques.Original TXT and TEI TranscriptionsThe more basic resources are the plain text ﬁles with the original transcriptions,both in wiki or TEI formats. These resources are available in 28 separate ﬁles,one for each letter, plus a geographic and an onomastic appendixes. These doc-uments (specially the TEI version as it is annotated and is easier to processautomatically) are mostly useful to study the Portuguese language before the1943/45 orthographic agreement.Current Database SnapshotA view of the Dicionário-Aberto web-site database is available to download anduse. It is an SQL document that can be imported in any MySQL database server(and probably in other tools with minor changes). Figure 1 shows the structureof this view (further tables might be available in the future, accordingly withnew resources being developed).This is the better way to use the current database in oﬄine mode, as it en-ables the user to access the current versions for all dictionary entries, as wellas the previous versions (before orthographic modernization). Therefore thisformat can be used not just to perform text-mining in the dictionary but also3 Unfortunately the modernization process had some false positive substitutions.A Source of Resources for the Portuguese Language Processing 123					 !					"#$%$&$'			Fig. 1. Dicionário-Aberto database viewfor contrastive studies. The database is regularly checked for quality control,checking the database for consistency and validating all XML snippets. Anothertest, that validates the dictionary completeness (that all cross-references have avalid target) is currently disabled given the modernization process.Modernization RulesWith the modernization process we obtained two versions of the dictionary in twodiﬀerent versions of Portuguese. Although they are similar in great extend, theycan be considered two diﬀerent languages, and therefore all typical approachesto align and extract bilingual dictionaries automatically can be applied.Tables 1 present two types of orthographic modernization rules that can beextracted (these lists were constructed using lexdiff [2]). The ﬁrst one mapsfull words and the second maps letter sequences. The latter is easier to apply indocuments with words that are not in the dictionary (but that share a subse-quence of letters with some other word), but the ﬁrst is better for accuracy. Thethird column is the rule conﬁdence about its transformation.Table 1. Word and pattern rules for orthographic modernizationFull-word Rules Pattern Rulesgênero género 100% sôa soa 99%aquelle aquele 100% ﬀe fe 95%eﬀeito efeito 100% ph f 95%fórma forma 100% lle le 90%pessôa pessoa 100% llo lo 88%camillo camilo 100% aes ais 87%póde pode 99% gên gén 87%sôbre sobre 98% bôa boa 80%ás às 90% cci ci 40%Note, however, that these dictionaries were calculated with the current versionof Dicionário-Aberto, where some hundred words were not modernized correctlyby the automatic process4.4 Being a dictionary, it includes very odd words that are not easily modernized bygeneral rules.124 A. Simões, Á.I. Sanromán, and J.J. AlmeidaMorphologic DictionaryAll the dictionary entries include partial morphological information (at leastits main category, and in some situations the genre or type of verb), makingit possible to extract a list of words with associated morphologic information.Table 2 resumes the size of this dictionary and distribution by main morphologiccategories.Table 2. Morphologic distribution of Dicionário-Aberto entriesNouns Adjectives Adverbs Verbs Locutionsmasc. fem. transitive intransitive45 657 38 488 30 469 2 962 10 147 4 016 579REST APITo enable the use of the dictionary in the cloud a simple web service API basedon REST principles is available. It supports queries both in XML and JSON, andlets the user query for deﬁnitions, given a speciﬁc word, or search for near misses,preﬁxes and suﬃxes. This enables the development of mobile applications. Anexample of such application is the iPhone interface to Dicionário-Aberto53 Resources under DevelopmentFurther work is being done to make the Dicionário-Aberto experience more inter-esting, enabling new services in the web site, but also to develop new resourcesthat can be used by natural language processing researchers.Reverse-Order DictionaryReverse-Order Dictionaries are not very common (do not confuse with ReverseDictionaries, discussed below). They let the user browse the dictionary searchingby the end of the word, instead of its beginning (looking up for suﬃxes insteadof preﬁxes).One of their applications is the construction of a rhyme dictionary (noticehowever that this will be a partial rhyme dictionary, as some words with diﬀerentorthography have the same sound, like doce and fosse).Another use of these kind of dictionaries is the study of the morphology of alanguage [6], like the study of suﬃx productivity (productivity for some scientiﬁcterminology suﬃxes — -ato, -eto, -ito — the productivity of eﬀect/result words— -data, -ção, -são, -ança, -ância, etc.).5 Developed by log.oscon, available from the Apple AppStore. Further details can befound at http://log.pt/dicionarioaberto/A Source of Resources for the Portuguese Language Processing 125Reverse Search or Reverse DictionaryIn a standard (paper) dictionary, the query can only be performed browsing thelist of alphabetically sorted lemmas. In machine-readable dictionaries, the searchshould not be limited to the lemmas. The user should be able to search the fullentries, including its deﬁnition, examples, etymology or any other section.This reverse search capabilities transforms electronic dictionaries in ideologicalor conceptual databases, also known as analogical or onomasiological dictionar-ies. This feature is available for some languages like Spanish6 and English7.There is a long tradition of onomasiological dictionaries for the Europeanlanguages. Some ideological dictionaries were prepared during XIX and XX cen-turies8, that allowed the reader to search an idea or concept in a descriptorsstructure similar to a thesaurus [11], or a structured list of concepts sorted bysubjects (summary tables) together with a list of hypernyms or broader terms(categories, general ideas) that lead the reader to the searched word.The reverse search functionality can help surpassing some of the limitationspresent in paper versions of these dictionaries. This functionality can not beonly seen as a simple query tool. Imagine the potentiality of reverse search if thedictionary authors use a controlled language to write their deﬁnitions, just likea descriptors thesaurus.Its main usage is to search a word that we know adequate for a speciﬁcsituation, but that we can not remember at that moment, or to search for amore speciﬁc word, or even to check if there is some word to express someconcept [4].Dicionário-Aberto will not only oﬀer this functionality for end-users throughthe web interface, and for cloud applications through the REST API, but alsomake available the reverse index, for oﬄine processing.Ontology ViewHugo Oliveira [7] has been working in the creation of Onto.PT, a lexical ontologyfor the Portuguese Language. Oliveira performed some experiments with diﬀer-ent resources, and Dicionário-Aberto was also covered. A similar approach wasalso performed by Simões et al [10]. With these experiments in mind, and theinteresting results obtained, a new view for Dicionário-Aberto is being developed:a (lexical) ontology view.This view will enable the user to query the dictionary, using the standardsearch or one of the two new methods described earlier, and together with thedeﬁnition, the examples, and the etymology, consult a thesaurus-like structure.This structure includes a set of relations (like synonymy, antonymy, hyperonymy,6 Reverse dictionary search in the Dictionary of the Real Spanish Academy, by GabrielAlberich: http://dirae.es/7 OneLook Reverse Dictionary, by Doug Beeferman, that searches more than one thou-sand indexed dictionaries: http://www.onelook.com/reverse-dictionary.shtml8 Some examples are listed by Martínez de Sousa [5].126 A. Simões, Á.I. Sanromán, and J.J. Almeidainstances/species/genres, actor/action, etc) to other dictionary entries (in caseof multi-word expressions each component word will have its own link).The extraction uses a set of patterns, just like the methods described by thementioned authors. We decided not to reuse the extracted data because in thefuture, as stated in the introduction, Dicionário-Aberto will be a Wiki, making itcrucial to have an automatic method to recalculate the ontology. The extractionmethod will run everyday in an unsupervised way.The ontology completeness will be guaranteed by a set of completion rules.For example, the synonymy relation is symmetric, the hyperonymy relation isinverse of the hyponymy relation, the hyperonymy relation can be seen as aspecial case of a transitive relation, antisymmetric relations, anti-reﬂexive, etc.These properties can be described in a mathematical notation and used to ensurethat entries that do not have a reference to related words can still get the relationinformation.This kind of feature will make the dictionary much more interesting for theend user but also for the natural language processing researcher. For the ﬁrst,it will make entries browsable by concept. For the second, will be a complementof a Portuguese word-net, as concepts will also include deﬁnitions.4 ConclusionsIn this document we described the current status of Dicionário-Aberto, an openproject to the development of a knowledge and feature rich dictionary for thePortuguese language, both to be used as a standard dictionary but also as aresource for natural language processing tasks. We defend that resources con-struction should be automatic, especially in a case like Dicionário-Aberto that willbe open for the community to cooperate. This guarantees that the extracted re-sources can grow in size and quality at the same time as its main resource.References1. Project Gutenberg. Project Gutenberg Literary Archive Foundation (November2011), http://www.gutenberg.org/2. Almeida, J.J., Santos, A., Simões, A.: Bigorna – a toolkit for orthography migra-tion challenges. In: Calzolari, N., et al. (eds.) Seventh International Conference onLanguage Resources and Evaluation (LREC 2010), Valletta, Malta, pp. 227–232.European Language Resources Association (May 2010)3. TEI Consortium, editor. TEI P5: Guidelines for Electronic Text Encoding andInterchange, chapter 9. Dictionaries. TEI Consortium (January 2012),http://www.tei-c.org/release/doc/tei-p5-doc/en/html/DI.html,Version 2.0.1 edition, December 22, 20114. Porto da Pena, J.A.: Dicionário de uso del Español (2003),http://cvc.cervantes.es/actcult/mmoliner/diccionario/(retrieved November 3, 2011)5. Martínez de Sousa, J.: Diccionario de lexicografía práctica. Biblograf, Barcelona(1995)A Source of Resources for the Portuguese Language Processing 1276. Millán, J.A.: Zigzag, gong, ping-pong, iceberg. donde se descubre que hay dic-cionarios inversos, y su utilidad maniﬁesta para el progreso de la humanidad (1999),http://jamillan.com/inverso.htm (retrieved November 3, 2011)7. Oliveira, H.G., Gomes, P.: Onto.PT: automatic construction of a lexical ontology forPortuguese. In: 5th European Starting AI Researcher Symposium (STAIRS 2010)(August 2010)8. Simões, A., Farinha, R.: Dicionário Aberto: Um novo recurso para PLN.Vice-versa (16), 159–171 (2011)9. Simões, A., Almeida, J.J.: Processing XML: a rewriting system approach. In:Simões, A., da Cruz, D., Ramalho, J.C. (eds.) XATA 2010 — 8a ConferênciaNacional em XML, Aplicações e Tecnologias Aplicadas, Vila do Conde, Maio,pp. 27–38 (2010)10. Simões, A., Almeida, J.J., Farinha, R.: Processing and extracting data from Di-cionãrio Aberto. In: Calzolari, N., et al. (eds.) Seventh International Conference onLanguage Resources and Evaluation (LREC 2010), Valletta, Malta, pp. 2600–2605.European Language Resources Association (May 2010)11. van Slype, G.: Les langages indexation: conception, construction et utilisation dansles systmes documentaires. Les Editions d’Organisation, Paris (1987)

Dicionário-aberto: a source of resources for the portuguese language processing

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM