We compare the use of edited text in the form of newswire and unedited text in the form of discussion forum posts as sources for training material in a self-training experiment involving the Brown reranking parser and a test set of sentences from an online sports discussion forum. We find that grammars induced from the two automatically parsed corpora achieve similar Parseval f-scores, with the grammars induced from the discussion forum material being slightly superior. An error analysis reveals that the two types of grammars do behave differently

Cetinoglu, Ozlem

Foster, Jennifer

van Genabith, Josef

Wagner, Joachim

English

Irish Universities

Comparing the use of edited and unedited text in parser self-training

DCU Online Research Access Service

Comparing the Use of Edited and Unedited Text in Parser Self-TrainingJennifer Foster, ¨Ozlem C¸etinog˘lu, Joachim Wagner and Josef van GenabithNational Centre for Language Technology/Centre for Next Generation LocalisationSchool of ComputingDublin City UniversityIreland{jfoster,ocetinoglu,jwagner,josef}@computing.dcu.ieAbstractWe compare the use of edited text in theform of newswire and unedited text in theform of discussion forum posts as sourcesfor training material in a self-training exper-iment involving the Brown reranking parserand a test set of sentences from an onlinesports discussion forum. We find that gram-mars induced from the two automaticallyparsed corpora achieve similar Parseval f-scores, with the grammars induced from thediscussion forum material being slightly su-perior. An error analysis reveals that the twotypes of grammars do behave differently.1 IntroductionThere have been several successful attempts in re-cent years to employ automatically parsed datain semi- and unsupervised approaches to parserdomain adaptation (McClosky et al., 2006b; Re-ichart and Rappaport, 2007; Huang and Harper,2009; Petrov et al., 2010). We turn our attentionto adapting a Wall-Street-Journal-trained parserto user-generated content from an online sportsdiscussion forum. The sentences on the discus-sion forum are produced by a group of speakerswho are communicating with each other about ashared interest and are discussing the same events,but, who, given the open, unedited nature of themedium itself, do not follow an in-house writingstyle. Our particular aim in this paper is to com-pare the use of discussion forum comments as asource of unlabelled training material to the useof edited, professionally written sentences on thesame theme. We hypothesise that the well-formedsentences will be more suitable as training mate-rial since they are likely to be closer syntacticallyto the source domain Wall Street Journal (WSJ)sentences than the noisier discussion forum sen-tences, while at the same time, remaining lexicallyclose to the target domain, thus acting as a type of“self-training bridging corpus” (McClosky et al.,2006b).2 Related WorkMcClosky et al. (2006b) demonstrate that a WSJ-trained parser can be adapted to the fiction do-mains of the Brown corpus by performing a typeof self-training that involves the use of the two-stage Brown reranking parser (Charniak and John-son, 2005). Their training protocol is as follows:sentences from the LA Times are parsed using thefirst-stage parser (Charniak, 2000) and reranked inthe second stage. These parse trees are added tothe original WSJ training set and the first-stageparser is retrained. The sentences from the tar-get domain, in this case, Brown corpus sentencesare then parsed using the newly trained first-stageparser and reranked using the original reranker, re-sulting in a Parseval f-score increase from 85.2%to 87.8%.McClosky and Charniak (2008) later show thatthe same procedure can be used to adapt a WSJ-trained parser to biomedical text. They also try anexperiment which is very similar to the experimentdescribed in this paper. Instead of using Medlineabstracts as training material, they use sentencesfrom a biology textbook under the assumption thatthe parses produced for these sentences will bemore accurate (and thus more reliable as trainingdata) than the sentences in the abstracts since theyare closer to the source domain. They find, how-ever, that the textbook sentences are less effectivethan the target domain material. We attempt torepeat the experiment with Web 2.0 data, believ-ing that the two setups are sufficiently different forour experiment to be worthwhile — our bridgingcorpus is closely related in subject matter to ourtarget corpus (both referring to the same events)but quite different in form (professionally editedversus an unedited mix of writing styles), whereastheir bridging corpus is less closely related in con-tent (biology textbooks versus Medline abstracts)and more closely related in form (both profession-ally edited and syntactically well-formed).3 DataOur dataset, summarised in Table 1, consists of asmall treebank of hand-corrected phrase structureparse trees and two larger corpora of unannotatedsentences.Discussion Forum Treebank The treebank isan extension of that described in Foster (2010). Itcontains 481 sentences taken from two threads onthe BBC Sport 606 discussion forum in Novem-ber 2009.1 The discussion forum posts were splitinto sentences by hand. The sentences were firstparsed automatically using an implementation ofthe Collins Model 2 generative statistical parser(Bikel, 2004). They were then corrected by handusing as a reference the Penn Treebank (PTB)bracketing guidelines (Bies et al., 1995) and thePTB trees themselves (Marcus et al., 1994). Formore detail on the annotation process, see Fosteret al. (2011). The development set contains 258sentences and the test set 223. The experiments inthis paper are carried out on the development set(which we refer to as FootballDev).Discussion Forum Corpus The same discus-sion forum used to create the treebank was scrapedduring the final quarter of 2010. The content wasstripped of HTML markup and passed through anin-house sentence splitter and tokeniser, resultingin a corpus of 1,009,646 sentences. We call thisthe FootballTrainDiscussion corpus.Edited Text Corpus In order to compare the useof edited versus unedited text, we also collected acorpus of professionally written news articles onthe same theme as the discussion forum sentences,namely, the English Premier League. Content wasscraped from the online BBC sports site2 and arti-cles dating from April 2010 to February 2011 re-trieved. Similar preprocessing was carried out onthese as was carried out on the FootballTrainDis-cussion content, i.e. HTML-stripping, sentencesplitting and tokenisation. The resulting corpus,1http://www.bbc.co.uk/dna/606/F15264075?thread=7065503&show=50 and http://www.bbc.co.uk/dna/606/F15265997?thread=7066196&show=502http://www.bbc.co.uk/search/sport/football and http://www.bbc.co.uk/search/news/footballwhich we will refer to as FootballTrainEdited,contains 209,014 sentences.4 ExperimentsWe retrain the Brown parser using the self-training protocol of McClosky et al. (2006b),that is, we retrain the first-stage parser usingcombinations of trees produced by the rerank-ing parser for sentences from Sections 2 to 21of the WSJ section of the Penn Treebank andfrom FootballTrainEdited|Discussion. We thenparse the sentences in FootballDev using the re-trained first-stage parser and the original reranker.Because we have approximately five times theFigure 1: Comparing the performance of five gram-mars trained on disjoint 200k subsections of Football-TrainDiscussion in a Brown self-training experiment.Results are on FootballDev.number of sentences in FootballTrainDiscussionthan in FootballTrainEdited, we first train five dif-ferent FootballTrainDiscussion grammars. Thegraph in Figure 1 shows the results on FootballDevwhen the training data contains disjoint subsec-tions of FootballTrainDiscussion, each containing200,000 sentences, along with varying amount ofWSJ2-21 trees. This gives us an idea of the amountof variation we might expect within one train-ing set source — the f-score noise is roughly 1.5points wide (= 3 boxes in the graph).We now turn to the main experiment of the pa-per, i.e. the comparison of FootballTrainDiscus-sion and FootballTrainEdited. The graph in Fig-ure 2 compares the performance of the Football-TrainEdited grammars with the performance aver-age over the five types of FootballTrainDiscussionCorpus Name #Sen SL Mean SL Med. σFootballDev 258 17.7 14 13.9FootballTest 223 16.1 14 9.7FootballTrainDiscussion 1,009,646 15.4 12 13.3FootballTrainEdited 209,014 17.7 17 11.4Table 1: Basic Statistics on the Web 2.0 datasets: number of sentences, average sentence length, median sentencelength and standard deviationFigure 2: Comparing the use of discussion forum ma-terial (FootballTrainDiscussion) and newswire articles(FootballTrainEdited) in a Brown self-training experi-ment. Results are on FootballDev.grammars on FootballDev. Note that a baselinegrammar which is trained on one copy of WSJ02-21 and no automatically parsed data achieves aParseval f-score of 79.7. The results in Figure 2appear to refute our original hypothesis, suggest-ing that there is very little difference between thetwo corpora, with the user-generated content ofFootballTrainDiscussion emerging as slightly su-perior on our development set.3 The only time thatthe FootballTrainEdited curve is above the Foot-ballTrainDiscussion is when the size of the orig-inal WSJ training set is restricted. This is an in-tuitively appealing result — in this scenario, thesentences in the FootballTrainEdited corpus aremaking up for the lack of WSJ trained material,although it is not clear whether this is because theFootballTrainEdited sentences are slightly longerthan the FootballTrainDiscussion sentences (see3Keeping the WSJ02-21 dataset size constant, we testwhether the difference between a FootballTrainEdited gram-mar and its five corresponding FootballTrainDiscussiongrammars is statistically significant. Of the 150 pairs, 42 dif-ferences are statistically significant (p <0.05).Table 1) or because they contain more WSJ-likeconstructions.5 AnalysisWe next attempt to determine the strengths andweaknesses of the two types of training mate-rial by classifying our development set items intothose that have improved as a result of self-training, those that have remained unchanged andthose that have deteriorated. We examine alledited grammars shown in Figure 2, i.e. the thirtygrammars obtained using 200,000 sentences fromFootballTrainEdited and varying sized copies ofWSJ02-21. For the discussion grammars, weexamine the grammars trained using one of thefive disjoint 200,000-sentence subsets of Foot-ballTrainDiscussion and varying sized copies ofWSJ02-21 — we randomly choose the grammarsmarked with squares in Figure 1. Following Mc-Closky et al. (2006a), we present a breakdownof our results according to sentence length, num-ber of co-ordinating conjunctions (CC) in the sen-tence, and, number of unknown words4 in the sen-tence. The results are shown in Tables 2, 3 and 4.Sentence counts are provided along with averagef-score differences between a self-trained gram-mar and the baseline grammar.It is not possible to discern strong patterns inthe breakdown of results but we do observe thefollowing subtle differences between the two typesof grammars:• The FootballTrainEdited grammars are moreconservative than the FootballTrainDiscus-sion grammars, with a larger number of sen-tences unchanged by self-training.• The FootballTrainDiscussion grammars out-perform the FootballTrainEdited grammarsfor short sentences.4A word is considered to be unknown if it does not appearat all in WSJ02-21.Table 2: Effect of Self-Training Broken Down by Sentence LengthDiscussion 1-9 10-19 20-29 30-39 40-59 >= 60 TOTALBetter 339 (+21.4) 761 (+14.7) 643 (+11.0) 229 (+9.6) 271 (+12.5) 65 (+5.7) 2308 (+13.6)No Change 1853 1760 444 119 27 7 4210Worse 118 (-26.6) 329 (-8.3) 443 (-7.7) 192 (-7.0) 92 (-7.1) 48 (-5.3) 1222 (-9.4)TOTAL 2310 (+1.8) 2850 (+3.0) 1530 (+2.4) 540 (+1.6) 390 (+7.0) 120 (+1.0) 7740 (+2.6)Edited 1-9 10-19 20-29 30-39 40-59 >=60 TOTALBetter 237 (+20.6) 667 (+12.3) 543 (+10.8) 218 (+10.4) 213 (+9.2) 97 (+7.5) 1975 (+12.1)No Change 1985 1934 656 202 29 1 4807Worse 88 (-18.5) 249 (-6.8) 331 (-9.4) 120 (-5.8) 148 (-6.3) 22 (-2.5) 958 (-8.5)TOTAL 2310 (+1.4) 2850 (+2.3) 1530 (+1.8) 540 (+2.9) 390 (+2.6) 120 (+5.6) 7740 (+2.0)Table 3: Effect of Self-Training Broken Down by Number of Coordinating Conjunctions in a SentenceDiscussion 0 1 2 3 4 TOTALBetter 1286 (+16.5) 631 (+11.1) 312 (+9.1) 0 (—) 79 (+5.5) 2308 (+13.6)No Change 3061 981 131 30 7 4210Worse 573 (-10.8) 518 (-8.3) 97 (-8.8) 0 (—) 34 (-4.6) 1222 (-9.4)TOTAL 4920 (+3.0) 2130 (+1.3) 540 (+3.7) 30 120 (+2.3) 7740 (+2.6)Edited 0 1 2 3 4 TOTALBetter 1052 (+14.5) 561 (+10.0) 234 (+9.8) 18 (+5.2) 110 (+6.3) 1975 (+12.1)No Change 3414 1168 212 12 1 4807Worse 454 (-9.6) 401 (-8.0) 94 (-5.4) 0 (—) 9 (-3.6) 958 (-8.5)TOTAL 4920 (+2.2) 2130 (+1.1) 540 (+3.3) 30 (+3.1) 120 (+5.5) 7740 (+2.0)Table 4: Effect of Self-Training Broken Down by Number of Unknown Words in a SentenceDiscussion 0 1 2 3 4 5 >=6 TOTALBetter 488 (+16.2) 865 (+10.8) 360 (+11.2) 319 (+19.1) 91 (+15.8) 107 (+11.9) 78 (+17.2) 2308 (+13.6)No Change 2028 1277 584 223 58 30 10 4210Worse 274 (-14.0) 408 (-7.5) 406 (-9.5) 58 (-6.3) 31 (-5.3) 13 (-4.2) 32 (-5.7) 1222 (-9.4)TOTAL 2790 (+1.5) 2550 (+2.5) 1350 (+0.1) 600 (+9.6) 180 (+7.1) 150 (+8.1) 120 (+9.7) 7740 (+2.6)Edited 0 1 2 3 4 5 >=6 TOTALBetter 330 (+14.9) 758 (+11.1) 329 (+8.8) 328 (+15.7) 55 (+14.0) 94 (+9.5) 81 (+10.4) 1975 (+12.1)No Change 2291 1352 800 241 91 32 0 (—) 4807Worse 169 (-13.5) 440 (-7.0) 221 (-8.6) 31 (-3.8) 34 (-7.4) 24 (-1.6) 39 (-11.2) 958 (-8.5)TOTAL 2790 (+1.0) 2550 (+2.1) 1350 (+0.8) 600 (+8.4) 180 (+2.9) 150 (+5.7) 120 (+3.4) 7740 (+2.0)• The FootballTrainEdited grammars appear toperform better than the FootballTrainDiscus-sion grammars when there are a relativelyhigh number of coordinating conjunctions ina sentence (greater than two).• Self-training with both FootballTrainDis-cussion and FootballTrainEdited data tendsto benefit sentences containing several un-known words, with the discussion grammarsbeing superior.6 ConclusionWe compare the use of edited versus uneditedtext in the task of adapting a WSJ-trained parserto the noisy language of an online discussion fo-rum. Given the small size of our developmentset, we have to be careful how we interpret theresults. However, they do seem to suggest thatthe two corpora are performing at similar levelsof effectiveness but exhibit differences. For ex-ample, if we take the best performing Football-TrainEdited and FootballTrainDiscussion gram-mars from those used in our error analysis of Sec-tion 5, we get two grammars with a Parseval f-score of 83.2 on FootballDev. Assuming the ex-istence of a perfect classifier, which, given an in-put sentence, can predict which of the two gram-mars will produce the higher-scoring tree, the f-score for FootballDev increases from 83.2 to 85.6.When we include the baseline grammar (f-score:79.7), this increases to 86.4. This suggests that thenext step in our research is to build such a classi-fier including as features the sentential propertieswe examined in Section 5, as well as the featuresdescribed in McClosky et al. (2010) and Ravi etal. (2008).AcknowledgementsThis research has been supported by EnterpriseIreland (CFTD/2007/229) and by Science Foun-dation Ireland (Grant 07/CE/ I1142) as part of theCNGL (www.cngl.ie) at the School of Computing,DCU.ReferencesAnn Bies, Mark Ferguson, Karen Katz, and RobertMacIntyre. 1995. Bracketing guidelines forTreebank II style. Technical report, Universityof Pennsylvania.Daniel Bikel. 2004. Intricacies of Collins parsingmodel. Computational Linguistics, 30(4).Eugene Charniak and Mark Johnson. 2005.Course-to-fine n-best-parsing and maxent dis-criminative reranking. In Proceedings of the43rd ACL.Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of NAACL.Jennifer Foster, ¨Ozlem C¸etinog˘lu, Joachim Wag-ner, Joseph Le Roux, Joakim Nivre, DeirdreHogan, and Josef van Genabith. 2011. FromNews to Comment: Resources and benchmarksfor parsing the language of Web 2.0. In Pro-ceedings of IJCNLP.Jennifer Foster. 2010. “cba to check the spelling”Investigating parser performance on discussionforum posts. In Proceedings of HLT NAACL.Zhongqiang Huang and Mary Harper. 2009.Self-training PCFG grammars with latent an-notations across languages. In Proceedings ofEMNLP.Mitchell Marcus, Grace Kim, Mary AnnMarcinkiewicz, Robert MacIntyre, Ann Bies,Mark Ferguson, Karen Katz, and Britta Schas-berger. 1994. The Penn Treebank: Annotatingpredicate argument structure. In Proceedingsof the ARPA Speech and Natural LanguageWorkshop.David McClosky and Eugene Charniak. 2008.Self-training for biomedical parsing. In Pro-ceedings of ACL:HLT.David McClosky, Eugene Charniak, and MarkJohnson. 2006a. Effective self-training forparsing. In Proceedings of NAACL.David McClosky, Eugene Charniak, and MarkJohnson. 2006b. Reranking and self-trainingfor parser adaptation. In Proceedings of ACL.David McClosky, Eugene Charniak, and MarkJohnson. 2010. Automatic domain adaptationfor parsing. In Proceedings of NAACL-HLT.Slav Petrov, Pi-Chuan Chang, Michael Ringgaard,and Hiyan Alshawi. 2010. Uptraining for accu-rate deterministic question parsing. In Proceed-ings of EMNLP.Sujith Ravi, Kevin Knight, and Radu Soricut.2008. Automatic prediction of parser accuracy.In Proceedings of EMNLP.Roi Reichart and Ari Rappaport. 2007. Self-training for enhancement and domain adap-tation of statistical parsers trained on smalldatasets. In Proceedings of ACL.

A maximum-entropyinspired parser.

Automatic domain adaptation for parsing.

Automatic prediction of parser accuracy.

Bracketing guidelines for Treebank II style.

cba to check the spelling” Investigating parser performance on discussion forum posts.

Course-to-ﬁne n-best-parsing and maxent dis-criminative reranking.

Effective self-training for parsing.

From News to Comment: Resources and benchmarks for parsing the language of Web 2.0.

Intricacies of Collins parsing model.

Reranking and self-training for parser adaptation.

Self-training for biomedical parsing.

Self-training PCFG grammars with latent annotations across languages.

Selftraining for enhancement and domain adaptation of statistical parsers trained on small datasets.

The Penn Treebank: Annotating predicate argument structure.

Uptraining for accurate deterministic question parsing.

Name not available

http://doras.dcu.ie/16855/1/Foster_comparing11.pdf

Comparing the use of edited and unedited text in parser self-training

Abstract

Similar works

Full text

Available Versions

Irish Universities

DCU Online Research Access Service

Name not available