Chen Keh-Jiann

Tsai Yu-Fang

English

DSpace at Waseda University

Context-rule Model for Pos TaggingYu-Fang TsaiAcademia Sinica, Institute of InformationScience128 Sec. 2Academy Rd.Nankang, Taipei, Taiwan 115eddie@iis.sinica.edu.twKeh-Jiann ChenAcademia Sinica, Institute of InformationScience128 Sec. 2Academy Rd.Nankang, Taipei, Taiwan 115kchen@iis.sinica.edu.twAbstractPart-of-speech tagging for a large corpus is a labour intensive and time-consuming task. In orderto achieve fast and high quality tagging, algorithms should be high precision and in particular, itstagging results should require less manual proofreading. In this paper, we proposed acontext-rule model to achieve both the above goals for pos tagging.We compared the tagging precisions between Markov bi-gram model and context-ruleclassifier. According to the experiments, context-rule classifier performs better than those twoother algorithms. Also, it covers the data sparseness problem by utilizing more context features,and reduces the amount of corpus that is need to be manual proofread by introducing theconfidence measure.1	 IntroductionPart-of-speech tagging for a large corpus is a labour intensive and time-consuming task. In order toachieve fast and high quality tagging, algorithms should be high precision and in particular, its taggingresults should require less manual proofreading. There is lots of work on part-of-speech tagging such asHidden Markov Models (HMMs), Maximum Entropy Models (MEs), and Support Vector Machines(SVMs), etc. Most of works addressed on the high accuracy of tagging results only. In this paper, weproposed a context-rule model to achieve both the above goals for pos tagging.2 Tagging AlgorithmsIn this study, we are going to test two different tagging algorithms based on same training data andtesting data. The two tagging algorithms are Markov bi-gram model, and context-rule classifier. ForMarkov bi-gram model, we propose a new form named word-dependent Markov bi-gram model, whichwill be described later. The training data and testing data are extracted from Sinica corpus, a 5 millionword balanced Chinese corpus with pos tagging (Chen et al., 1996). The confidence measure will bedefined for each algorithm and the best accuracy will be estimated at the constraint of only a fixedamount of testing data being proofread.It is easier to proofread and make more consistent tagging results, if proofreading processes weredone by checking the keyword-in-context file for each ambivalence word and only the tagging results ofambivalence word need to be proofread. The words with single category need not be rechecked their postagging. For instance, in Table 1, the keyword-in-context file of the word '1111:F9: (research), which haspos-categories of verb type VE and noun type Nv, is sorted according to its left/right context. Theproofreader can see the other examples as references to determine whether or not each tagging result isright. If all of the occurrences of ambivalence word have to be rechecked, it is still too much of the work.The common terms used in the following tagging algorithms were defined as follows:Wk	 The k-th word in a sequenceCk	 The pos-category associated with k-th word wk146ilig(DE)	 *16,-(VH) fi)fR(Nv)	 vt 40(Na)(Dfa)	 wrE(VJ)	 iiffR(Nv)	 (Nv)(Na)	 frEM(Na) ifFR(VE)	 (Na)ki_1(VCL)	 w4(Nc) iffri,(VE) A(Nes).71( 3)	 ift-f4(VH) ii):FR(VE)	 (PERIODCATEGORY)1-rkit(Na)ffif4(VH) lifFR(VE)	 (PERIODCATEGORY)W(D)	 MV4(VH) ii)M(Nv)	 (PERIODCATEGORY)Z(DE)(COMMACATEGORY)1. 'tj (Nb)t{h(Na)Table 1 Sample keyword-in-context file of the words 'z sorted by its left/right context141,Cn	 A word sequence containing n words with their associated categoriesrespectively2.1 Markov Bi-gram ModelThe most widely used tagging models are part-of-speech n-gram models, in particular bi-gram andtri-gram model. In a bi-gram model, it looks at pair of categories (or words) and uses the conditionalprobability of P(ck I ck_1 ) , and the Markov assumption is that the probability of a category occurringdepends only on the one category before it.Given a word sequence wo...wn , the Markov bi-gram model calculates the probability of eachcandidate category ck for a target word wk by P(ck I ck_i ) P(wk I ck ). There are two approaches toestimate the statistical data for P(ck ck_i ) . One is to count all the occurrences in the training data,called general Markov model, and another one is to count only the occurrences in which each wk occurs,called word-dependent Markov model. We compared the two different approaches of Markov bi-grammodel with the proposed context-rule model algorithm in the experiments.2.2 Context-rule ModelThe conventional Markov n-gram models utilize the features of categories of context words and theprobability distribution of the categories of target words. In fact, for some cases the best pos-tags mightbe determined by other context features, such as context words instead of the categories. In thecontext-rule model, we extend the scope of the dependency context of a target word into its 2 by 2context windows. Therefore the context features of a target word wo can be represented by the vector of[w_2 , c_2 , w_1 , c_1 , w1 , ci ,w2 , c2 ] . Each feature vector may be associated with one or more pos-tags. Theassociation probability of the candidate category c; is P(c;Iwo , feature vector). If for some ( wo ,the value of P(co wo , feature vector) is not 1, it means that the pos of wo cannot be uniquelydetermined by its context vector. Some additional features have to be incorporated to resolve theambiguity. If for some word wo , all of its pos c; such that the value of P(c'o I wo , feature vector) is zerowhich means there is no training examples with the same context vector of wo . If the full scope of thecontext feature vector is used, data sparseness problem will seriously hurt the system performance.Therefore partial feature vectors are used instead of full feature vectors. The partial feature vectorsapplied in our context-rule classifier are w_1 , w1 , c _2c , c1 c2 , c_1 cl ,w_2 c_1 , w_l c_i , and c1 w2 .At the training stage, for each feature vector type many rule instances will be generated. Forinstance, with the above applied feature vector types, we can extract rule patterns of w _1 (%), w1 (2,c _2 c _1 (Nb, Na), ci c2 (Ng, COMMA), ... etc, associated with the category VE of target word 'FRresearch' from the following sentence:Tsou (Nb) 	  Mr (Na) ifFR research (YE) zn after (Ng) ' (COMMA)" After Mr. Tsou has done his research,"By investigating all training data, different rule patterns will be generated, and their associationprobabilities P(c'olwo , feature vector) are also derived. For instance, If we take those word sequences147Word Meaning Characteristics-r an expletive in the Chinese high frequencyg get, be about to average distribution of candidate categoriesw5f., research high inconsistence of context informationaV change simply two candidate categories,-MI- interview, gather material low frequency Nffi perform extreme low frequencyTable 2 Target words used in the experimentslisted in Table 1 as training data and c _lci as feature pattern, and set .. research' as target word, wewould train with a result containing a rule pattern = 	 PERIOD) and derive the probabilities ofP(VE (VII, PERIOD)) = 2/3 and P(.1VV I (Y1-1,PERIOD))= 1/3. Suppose that the target word wohas ambiguous categories of cl , c2 c , and the context patterns of patterni , pattern 2 ,..., pattern m,then the probability to assign tag c i to the target word wo is defined as follows:P(c1 I w, pattern),)Y=m1	 ny, y, P(ex I w, pattern),)x=1 y=1In other words, the probabilities of different patterns with the same candidate category areaccumulated and normalized by the total probability distributed to all candidates as the probability of thecandidate category. The algorithm will tag the category of the highest probability.3 Experiment ResultsThe Sinica corpus is separated into two parts as our training data and testing data. The training data israndomly generated and utilizes 90% of the corpus, while the testing data is the remaining 10% part.Some ambiguous words' frequencies in the corpus are too low so that neither the context-rule algorithmnor the word-dependent Markov model is able to tag them well. Those words should be processed byother generic tagging algorithms. Therefore, we picked up words that its frequency is equal to or greaterthan 10 only as the target words in the experiments. The six ambivalence words with differentfrequencies, listed in Table 2, were picked as our example target words to see the performance of eachtagging algorithm on words with different characteristics.Some words like .. interview' and `.. perform' have too low frequencies to have enough training data.To solve the problem of data sparseness, the Jeffreys-Perks law, or Expected Likehood Estimation(ELE), is introduced as the smoothing method for all evaluated tagging algorithms. To smooth for anunseen pattern	 wn , the probability PO411 ,...,1410 is defined as 	 , whereN+.13.1,C(wi,...,wn) is the amount that the pattern occurs in the training data, and N is the total amount of alltraining patterns, and B denotes the amount of all pattern types in training data and A denotes thedefault occurrence count for an unseen pattern. The most widely used value for A. is 0.5, which is alsoapplied in the experiments.The Markov bi-gram model was evaluated to be compared with our context-rule model. Markovbi-gram model looks the category of the target word and categories before/after the target words. That is,given a word sequence w1 ,...wn , it calculates the probability of each candidate category ck for a targetword wk by P(ck I ck_1 ) P(e lc+, I Ck) P(wk I ck ). In the experiments, we evaluate the probabilities ofP(ck Ic k_1 ) and P(c k+1 I c k ) by two different approaches. One is to train all the sequences in thetraining data, and another one is to train only the sequences in which each wk occurs.148Word General Markov Word-Depend. Markov Context-RuleT 96.95 % 97.92 % 98.87 %;f4 93.47 % 93.17 % 95.52 %FR 80.76 % 79.28 % 81.40 %EzV 87.60% 89.92% 93.02%"iii 68.06 % 63.89 % 77.78 %Affi 41.67 % 66.67 % 66.67 %Avg. of 6 words 94.56% 95.12% 96.60%Avg. of all words 91.07 % 94.07 % 95.08 %Table 3 Precision rates between evaluated tagging algorithmsThe evaluated result is shown in table 3. The comparison of two approaches to evaluating P(c k I ck_i )and P(C k+i IC k ) in Markov model shows that using word-dependent context features is better than usingall context features. The proposed context-rule model has higher precision rate than the Markov models.4 Confidence Measure and Reduction on Manual ProofreadingThe accuracy of a tagging result is usually estimated by the tagging precision of the algorithm. Howeverthe report precision of automatic tagging algorithm is about 95% to 96% (Chang et al., 1993; Lua, 1996;Liu et al., 1995). A better accuracy can be achieved if the tagging results are manually proofread. If wecan pinpoint the errors, only 4-5% of the corpus has to be revised. Since it is not known whereoccurrences of errors are, conventionally the whole corpus has to be reexamined. It is most tedious andtime consuming, since a practically useful tagged corpus is at least in the size of several million words.In order to reduce the manual editing and speed up the construction process of a large tagged corpus, apartial proofreading process has to be carried out. Only potential errors of tagging will be recheckedmanually. The problem is how we find the potential errors of the tagging and what is a reliable taggingsystem, which can provide a confidence score for each step of tagging?Since a probabilistic-based tagging method will assign a probability to each candidate pas-category,we assume that a candidate with higher probability might be more reliable. Therefore we adopt thefollowing hypothesis. If the probability P(c1 ) of the top choice candidate c 1 is much higher than theprobability P(c2 ) of the second choice candidate c2 , then the confidence value assigned for c 1 is alsohigher. Likewise if the probability P(c 1 ) is closer to the probability P(c 2 ) , then the confidence valueP(ci ) assigned for c1 is also lower. A general confidence measure was defined as the value ofP(c1)+P(c2)where P(c1 ) is the probability of the top choice category c1 assigned by the tagging algorithm andP(c2 ) is the probability of the second choice category c2 . By using this definition of confidencemeasure, one can choose a confidence score, for example, 0.6, to filter those tagged words that havescore lower than the pre-chosen confidence score, which are need manual proofreading. We like toprove the above hypothesis by empirical methods.A tagging algorithm provided with a very reliable confidence score in some sense is a goodcost-effective algorithm. A cost-effective algorithm may not be the algorithm with the highest precision.Therefore we defined below a new concept of reliability of a tagging system in term of cost-effective:Reliability	 The estimated best accuracy can be achieved by the tagging model under theconstraint that only a fixed amount of K% corpus with the lowest confidence valueis manually proofread.We carried out an experiment on the confidence measure on the context-rule tagging model. Thetarget words are the ambivalence words of frequency greater than or equal to 10. Figure 1 shows theresults. The confidence score is increased from 0.50, step in 0.01, to 1.00, to observe the curve betweenthe amount of manual proofreading and the best accuracy with manual proofreading. When a certainconfident score is chosen, some tagged words with confidence score lower than the chosen one will149Context-Rule1.00000.99500.99000.9850Uco- 0.9800u 0.97504,• 0.9700w 0.9650co0.96000.95500.95000.94500.0000 0.1000	0.2000	 0.3000	 0.4000	0.5000	0.6000Reiiew Radio--in--Context-RuleFigure 1	 Tradeoffs between amount of manual proofreading and the best accuracyshow that they might have been tagged with wrong category. If those words are all manual proofread,the tagging accuracy can be increased more efficiently, and the final tagging accuracy can be easilyestimated. For instance, if the confidence score is set 0.6, only 10.04% of tagged words have theconfidence scores less than 0.6 that cover 57.92% error tagging. Therefore the estimated best taggingaccuracy is 97.93%, if those tagged words with lower confidence score are all manual revised. The bestaccuracy is estimated by adding the amount of error reduction by manual proofreading to the originaltagging accuracy, i.e. 95.08%+4.92%*57.92%=97.93% for confidence threshold of 0.6.It is obviously that there is a trade-off between the accuracy and the amount of corpus to be manualproofread. The higher accuracy required, the larger corpus to be manual proofread. Thus, with a fixedresource of labour, one can determine the final accuracy of corpus after manual proofread is done, orhe/she can estimate how many corpus should be manual proofread to achieve the required accuracyaccording to the curve.5	 ConclusionThe proposed context-rule model utilizes a broader scope of features to tag pos and achieve a betterprecision. The target word dependent Markov model also performs better than general Markov model. Itclearly shown that to utilize more dependent features and more precise probability dependent statisticswill perform better on the pos tagging. On the other hand the sparseness of training data reduces theaccuracy of the tagging algorithm. Therefore use of more dependent features means more serious of datasparseness problem. However the context-rule model avoid the data sparseness problem by utilizing therules with higher occurrence patterns only and use the general category patterns to cope with the lowfrequency target words. The context-rule tagging models focus on the ambivalence words only since top300 ambivalence words contains 95% of tagging ambiguities according to Huang et al (2000). Thereforeusing confidence evaluation and context-rule models can drastically reduce amount of manualproofreading.150ReferencesC. H. Chang & C. D. Chen, 1993, "HMM-based Part-of-Speech Tagging for Chinese Corpora," in Proceedings ofthe Workshop on Very Large Corpora, Columbus, Ohio, pp. 40-47.C. J. Chen, M. H. Bai, & K. J. Chen, 1997, "Category Guessing for Chinese Unknown Words," in Proceedings ofNLPRS97, Phuket, Thailand, pp. 35-40.Christopher D. Manning & Hinrich Schutze, Foundations of Statistical Natural Language Processing, The MITPress, 1999, pp. 202-204.K. J. Chen, C. R. Huang, L. P. Chang, & H. L. Hsu, 1996, "Sinica Corpus: Design Methodology for BalancedCorpora," in Proceedings of PACLIC II, Seoul, Korea, pp. 167-176.K. T. Lua, 1996, "Part of Speech Tagging of Chinese Sentences Using Genetic Algorithm," in Proceedings ofICCC96, National University of Singapore, pp. 45-49.P. Kveton & K. Oliva, 2002, "(Semi-) Automatic Detection of Errors in PoS-Tagged Corpora," in Proceedings ofColing 2002, Taipei, Tai-wan, pp. 509-515.S. H. Liu, K. J. Chen, L. P. Chang, & Y. H. Chin, 1995, "Automatic Part-of-Speech Tagging for Chinese Corpora,"on Computer Proceeding of Oriental Languages, Hawaii, Vol. 9, pp.31-48.151

Context-rule Model for Pos Tagging

Tsai, Yu-Fang

49724

Chen, Keh-Jiann

49725

Waseda University Repository

https://waseda.repo.nii.ac.jp/?action=repository_action_common_download&item_id=28774&item_no=1&attribute_id=101&file_no=1

Context-rule Model for Pos Tagging

Abstract

Similar works

Full text

Available Versions

DSpace at Waseda University

Waseda University Repository