This paper reports the most significant issues related to the launching of a Monolingual Spanish Question Answering evaluation track at the Cross Language Evaluation Forum (CLEF 2003). It introduces some questions about multilingualism and describes the methodology for test suite production, task, judgment of answers as well as the results obtained by the participant systems

D.D. Palmer

I.H. Witten

J.R. Quinlan

T.M. Mitchell

B. Magnini

J.L. Vicedo

Crossref

Spanish Question Answering Evaluation

This paper reports the most significant issues related to the launching of a Monolingual Spanish Question Answering evaluation track at the Cross Language Evaluation Forum (CLEF 2003). It introduces some questions about multilingualism and describes the methodology for test suite production, task, judgment of answers as well as the results obtained by the participant systems.Depto. de Sistemas Informáticos y ComputaciónFac. de InformáticaTRUEpu

Peñas, Anselmo

Verdejo, Felisa

Herrera, Jesús

Docta Complutense

Spanish

Spanish question answering evaluation

In this paper we present a study comparing the performance of different systems found in the literature that perform the task of automatic text segmentation in sentences for English documents. We also show the difficulties found to adapt these systems to make them work with Portuguese documents and the results obtained after the adaptation. We analyzed two systems that use a machine learning approach: MxTerminator and Satz, and a customized system based on fixed rules expressed by Regular Expressions. The results achieved by the Satz system were surprisingly positive for Portuguese documents

Silla Jr, Carlos N.

Kaestner, Celso A.A.

Kent Academic Repository

An Analysis of Sentence Boundary DetectionSystems for English and Portuguese Documents?Carlos N. Silla Jr. and Celso A. A. KaestnerPontifical Catholic University of ParanaRua Imaculada Conceicao, 1155 - 80.215-901Curitiba - Parana - BRAZIL{silla, kaestner}@ppgia.pucpr.brAbstract. In this paper we present a study comparing the performanceof different systems found in the literature that perform the task of au-tomatic text segmentation in sentences for English documents. We alsoshow the difficulties found to adapt these systems to make them workwith Portuguese documents and the results obtained after the adapta-tion. We analyzed two systems that use a machine learning approach:MxTerminator and Satz, and a customized system based on fixed rulesexpressed by Regular Expressions. The results achieved by the Satz sys-tem were surprisingly positive for Portuguese documents.1 IntroductionWhen dealing with tasks related to the automatic processing of documents likesummarization, translation, etc. one of the procedures that frequently occur isthe segmentation of the text in sentences. This task is usually included in thepre-processing stage, and uses a simple criterion, tagged documents, or one ofthe approaches found in the literature.The systems found in the literature can be grouped in two classes: the onesthat use fixed rules to identify what is and what is not a sentence, and the onesthat use a machine learning approach. In this work we evaluate the performanceof one customized system that uses fixed rules, and two systems that use amachine learning approach: MxTerminator [1] and Satz [2]. The first systemuses templates based on Regular Expressions, considering the context where apunctuation mark appears, and will be refered to as RE (Regular Expressions)[3]. The MxTerminator uses a Maximum Entropy Model to detect the sentenceboundaries, while Satz considers the context where a possible punctuation markappears and can be used with any machine learning algorithm; in this work, itwas used with the C4.5 classifier [4].The remaining part of the article is divided as follows: section 2 presents ageneral view of the systems used for comparison and how they were adaptedto Brazilian Portuguese; section 3 describes the methodology used in the ex-periments and presents the corresponding results for two sets of documents, in? This research was supported by the Brazilian PIBIC-CNPq Agency.English and Portuguese; and finally, in section 4 we draw the conclusions andperspectives of this work.2 The Different Approaches to Detect SentenceBoundariesIn this section we present an overview of the three different systems used in thiswork.2.1 RE SYSTEMAs a representative of the fixed rules approach, we used a system that encodethe rules as regular expressions, and considers the context where each possibleend of sentence occurs within the document. This system was chosen becauseit has achieved results close to the MxTerminator system in a dataset of theTIPSTER collection in recent experiments [3].The system uses a database of regular expressions which denote chains thatcontain punctuation marks but don’t indicate the end of a sentence, like abbre-viations and other sequences like e-mails, www addresses, etc. The database ofregular expressions is kept on a text file, which allows easy manipulation of theexisting rules.To identify the sentences, the system scans the text until it finds the firstperiod (.); after that, it analyzes the preceding string; if this string matches someregular expression, then the system concludes that this is not an end of sentenceand advances to the next period. If the preceding string doesn’t match anyregular expression, the system verifies the string after the period, which mightbe one of the special cases and need a different treatment. If the system doesn’tfind any matching regular expression for the current string, it concludes thatthe period indicates an end of sentence, and tags the text with the appropriatemarks, in this case: <S> and </S>. The procedure is repeated until the entiredocument has been analyzed.The following special cases are treated by the system:– Decimal Numbers: the system verifies if what comes before the period is anumber, and if it is, it also verifies if the word after the dot is a number.That way it can distinguish between: “. 2003.” and “.... US$ 50.25”.– Parenthesis at the end of a sentence: one characteristic of the English lan-guage is that sentences like “( that night.)” are correct, unlike the Portugueselanguage, in which the correct form is “( that night).”.– Ellipsis: The last special case treated by the system is related to the occur-rence of ellipsis (“. . . ”). In this case, the system verifies the occurrences ofsuccessive dots until it finds the last one of them, which indicates the end ofthe sentence.To adapt the system to the Brazilian Portuguese it was necessary to add240 new regular expressions that basically denote abbreviations of the language.Since a text file describes the regular expressions, it was easy, although timeconsuming, to adapt the system.2.2 MxTerminatorThe system MxTerminator was developed by Reynar and Ratnaparkhi[1] in thePennsylvania University and uses an approach which is independent of languageor text genre. MxTerminator uses a machine learning algorithm named Maxi-mum Entropy Model to identify the sentences of a document.From a Corpus with the sentences already identified (training set) the modellearns to classify each instance of period (.), exclamation mark (!) and questionmark (?) as elements that identify what is a sentence end and what is not.The training process is robust and doesn’t need any type of fixed rules orsome other linguistic information, like part-of-speech frequencies, or even spe-cific information about the genre or domain of the texts, because during thetraining the system creates a list of induced abbreviations. This list is obtainedconsidering an abbreviation as every word in the training set that has a whitespace before and after its occurrence and contains a possible end of sentencesymbol (.,!,?), but doesn’t indicate an end of sentence.The possible sentences of the document are identified by scanning the textfor sequences of characters separated by a blank space (token) containing one ofthe symbols that indicate a possible end of sentence (.,?,!).The token that contains the symbol which denotes a possible end of sentenceis called Candidate. The system then uses the contextual information where eachCandidate occurs. The contextual information is represented by a set of featureslike the prefix, the suffix, etc.The main idea of the Maximum Entropy Model is that the probability of acertain class in this case - the sentence boundaries - in a given context, can beestimated by the joint probability distribution using a maximum entropy model.To adapt the MxTerminator to the Brazilian Portuguese language the pro-cedure was very simple, because the system uses, for training a text file of anysize that must contain one sentence per line. Another interesting factor is that,besides the training files, no other type of information was needed. For this rea-son, the MxTerminator was considered the simplest of the analyzed methods touse and adapt and use in new language.2.3 SatzThe Satz system was developed by Palmer and Hearst[2] and uses an approachthat considers the context where each punctuation mark occurs; it can be usedwith any machine learning algorithm, and the original results were tested usingNeural Networks [5] and the C4.5 Decision Tree Classifier [2].The Satz system represents the context around a possible end of sentencesymbol constructing a series of descriptor arrays, that represent an estimativeof the part-of-speech distribution for each word.Table 1. Mapped ClassesGrammatical Class Mapped TagsMiscellaneous CUR; IN; OTHER; PDEN;Noun N; N/N; N/N/N; N/VVerb V; VAUX; V|PASS; V/VArticle ARTPCP; PCP/PCP; ADV; ADV/ADJ; ADV/ADV;ADV/KC; ADV/KS; ADV-KS-REL;Modifier ADV-KS-REL/ADV-KS-REL;ADV/PREP; ADV/PROADJ;ADV/PROSUB; ADJ; ADJ/ADJ; ADJ/VConjunction KS; KS/ADJ; KC; CCPROADJ; PROADJ/ADJ;Pronoun PRO-KS; PRO-KS/PRO-KS; PROPESS;PROPESS/PROPESS; PROSUBPreposition PREP; PREP/ADJ; PREP/VProper Noun PROP; NPROP; NPROP/NPROPNumber NUMComma or Semicolon , ;Left Parentheses QUOTEL; (Right Parentheses QUOTER; )Non Punctuation Character =; *; #;Possessive $Colon or Dash -; :; –;Abbreviation ABSentence Ending Punctuation .; . ; !; ?;The use of a part-of-speech estimative considers the context in which theword occurred rather than just the word itself. This is a unique aspect of theSatz system, and according to its authors is the main factor for the high efficiencyof the system. The part-of-speech frequencies are stored in a lexicon. If a wordis not present in the lexicon, a series of heuristics are used in order to define thecorresponding frequency.The context vector contains the descriptor arrays for each word surroundingthe possible end of sentence, and is the input for the machine-learning algorithm.The output is used to indicate if a possible end of sentence mark corresponds toan end of sentence or not.To adapt the Satz system to Portuguese it was necessary to re-implement thesystem, because the version available at the UCI Repository presents problemswhen dealing with accented characters, which are very common in BrazilianPortuguese. For example, a word like “agr´ıcola” (agricultural) would be identifiedas two tokens: “agr” and “cola”.In order to re-implement the Satz system, we developed a Java-based versionof the system that produces the descriptor arrays and integrated it with theWeka Data Mining Tool [6]. We used the J4.8 (which is a Java implementationof the C4.5 algorithm) in the tests. However, this procedure alone was not enoughto adapt the system. We had to create a new lexicon using the part-of-speechinformation which is present in the Corpus. We also needed to map the BrazilianPortuguese Corpus tags to the 18 general categories of the system. Table 1 showsthe tags mapped to each category.3 Experiments and Obtained ResultsIn order to perform a comparison between the different systems using documentsin English, we used one of the databases that contains news from the Wall StreetJournal, which belongs to the TIPSTER document collection, from the TextRetrieval Conference (TREC - Reference number of the database: WSJ-910130).The database contains 156 documents at different sizes, totalizing 3.554 sen-tences. To perform the experiments, each of the documents had their sentencesdetected and tagged manually. To evaluate the performance of the systems de-scribed in section 2, we also compared their results with the baseline proposedby Palmer[5]: where each sentence is obtained using the simple criterion period(.).The results achieved by each system are presented in Table 2, where:– “Precision” indicates the percentage of correctly classified sentences of thedocuments (Number of sentence endings correctly identified / Number ofsentences identified);– “Recall” indicates the percentage of correctly classified sentences of the doc-uments regarding the number of sentences present in the original document(Number of sentence endings correctly identified / Number of sentencespresent in the original database);– “F-measure” is a unique evaluation measurement, which combines precisionand recall in a single metric: the harmonic mean.The results achieved by the different systems show that although the REsystem uses a fixed rule approach, its results are close to the other systems. Thisindicates that when the domain of the documents is well known, and no trainingCorpus is available, the use of a fixed rule system might be a good option.In order to evaluate the performance of the systems with Portuguese Doc-uments, we used a version of the Lacio-Web Corpus [7], that contains 21.822sentences.Table 2. Results achieved by the different systems in the TIPSTER (English) docu-ment collectionSystem Precision Recall F-MeasureBaseline 30,29% 50,61% 37,89%RE 92,39% 91,18% 91,78%MxTerminator 91,19% 91,25% 91,22%Satz 98,67% 85,98% 91,88%Table 3. Robustness of the different systems using the Lacio-Web (Portuguese) doc-ument collection (uncustomized versions)System Precision Recall F-MeasureBaseline 85,40% 92,25% 88,69%RE 91,80% 88,02% 89,87%MxTerminator 94,29% 95,84% 95,05%Satz 99,48% 98,81% 99,14%To evaluate the robustness of each one of the systems, i.e. their performanceusing their original configuration, without any modification in the regular ex-pressions nor any re-training, we used the Lacio-Web Corpus with unidentifiedsentences. The results achieved by each system are presented in Table 3. Forthe Satz system which is dependent on part-of-speech frequencies, the test wasperformed using only the original heuristics of the system.Finally, in order to evaluate the performance of each of the systems whencustomized, they were adapted to Brazilian Portuguese by providing the neededinformation about the language. The MxTerminator was trained using the Lacio-Web Corpus using 10-fold cross-validation [8]. To the RE system we added 240new regular expressions mostly containing abbreviations of Brazilian Portuguesewords. The Satz system was also trained using 10-fold cross-validation, buta complete lexicon was created using the part-of-speech frequencies availablewithin the Corpus. The results achieved by each system can be seen in Table 4.Table 3 shows that the results achieved by the MxTerminator are surpris-ingly good, but not as impressive as the ones achieved by Satz. The resultsindicate that the MxTerminator and Satz are robust methods, although the re-sults achieved by Satz are surprisingly positive. The results also indicate thatthe fixed rule approach, even in the form of regular expressions, is not well suitedif the domain and genre of the texts are unknown.Table 4 shows that after being adapted to Brazilian Portuguese, both machinelearning methods improved their performance, except for the fixed rule approach.However, the results achieved by Satz even without the lexicon with part-of-speech information for all the words in the documents are outstanding.Table 4. Results achieved by the different systems in the Lacio-Web (Portuguese)document collection (customized versions)System Precision Recall F-MeasureBaseline 85,40% 92,25% 88,69%RE 91,80% 88,02% 89,87%MxTerminator 96,31% 96,63% 96,46%Satz 99,59% 98,74% 99,16%4 ConclusionsIn this work, we analyzed three different systems for the task of automatic textsegmentation, in order to identify the sentence boundaries in a document. Weperformed experiments for both English and Portuguese documents, using afixed regular expression rules system, the Satz decision tree approach and theMxTerminator maximum entropy approach. The best results were achieved bythe Satz system: 91,88% of F-measure in the English document database; 99,14%in the Portuguese document database without retraining, using only heuristicsand 99,16% in the same collection with retraining.These results indicate that the part-of-speech frequencies, in the case ofBrazilian Portuguese, are not as important as it is when working with English.This is explained by the fact that English sentence construction follows more re-strictive construction patterns than the sentences in Portuguese, which is a Latinlanguage. The adaptability and robustness of each system were also evaluated.Although the RE system achieved results similar to the ones achieved by theother two systems, this was one specific case where the domain and genre of thetext was well known. The MxTerminator achieved good results and was also theeasiest system to adapt to the Brazilian Portuguese language. The Satz systemhad an outstanding performance showing that the part-of-speech informationdoes not matter to identify the sentences in Brazilian Portuguese Documents.AcknowledgmentsWe would like to thank Dr. Adwait Ratnaparkhi from the Pennsylvania Univer-sity for sending us the MxTerminator system, and to Dr. Marti A. Hearst fromthe Berkeley University for kindly sending us the original files used by Satz.References1. Reynar, J., Ratnaparkhi, A.: A maximum entropy approach to identifying sentenceboundaries. In: Proceedings of the Fifth Conference on Applied Natural LanguageProcessing. (1997) 16–192. Palmer, D.D., Hearst, M.A.: Adaptive multilingual sentence boundary disambigua-tion. Computational Linguistics 23 (1997) 241–2673. Silla Jr., C.N., Valle Jr., J.D., Kaestner, C.A.A.: Automatic sentence detectionusing regulares expressions (in portuguese). In: Proceedings of the 3rd BrazilianComputer Science Congress, Itaja´ı, SC, Brazil (2003) 548–5604. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)5. Palmer, D.D.: SATZ - an adaptive sentence segmentation system. Master’s thesis(1994)6. Witten, I.H., Frank, B.: Data Mining: Practical Machine Learning Tools and Tech-niques with Java Implementations. Wiley-Interscience, San Francisco (1999)7. Aluisio, S.M., Pinheiro, G.M., Finger, Nunes, M.G.V., Tagnin, S.E.: The lacio-webproject: overview and issues in brazilian portuguese corpora creation. In: Proceed-ings of the Corpus Linguistics 2003. Volume 16. (2003) 14–218. Mitchell, T.M.: Machine Learning. McGraw-Hill (1997)

English

A.: A maximum entropy approach to identifying sentence boundaries. In:

Adaptive multilingual sentence boundary disambiguation.

C.A.A.: Automatic sentence detection using regulares expressions (in portuguese). In:

C4.5: Programs for Machine Learning.

Data Mining:

Machine Learning.

SATZ - an adaptive sentence segmentation system.

The lacio-web project: overview and issues in brazilian portuguese corpora creation. In:

An Analysis of Sentence Boundary Detection Systems for English and Portuguese Documents

http://kar.kent.ac.uk/24119/1/AnalysisSilasJr.pdf

Spanish question answering evaluation

Abstract

Similar works

Full text

Available Versions

Crossref

Docta Complutense

Kent Academic Repository

Crossref