He Ting ting

Luo Chang Ri

English

DSpace at Waseda University

A Vector-Based Algorithm for Chinese Text ClassificationEllLuo Chang riDepartment of Computer ScienceCentral China Normal UniversityWuhan 430079,PR. ChinaLuocr2@hotmail.comHe Ting tingDepartment of Computer ScienceCentral China Normal UniversityWuhan 430079,PR. Chinahett@l63.netAbstractIn this paper, vector-distance-weighted algorithm and representative-vector-distance algorithm aredescribed and used to implement the process of automatic text classification. Two experiments havebeen done by means of the algorithms (experiment) is based on vector-distance-weighted algorithmand experiment2 is based on representative-vector-distance algorithm). Characters are selected asfeatures. The average precision of experiment) and experiment2 is 80.36% and 69.27%, respectively.Comparing the two experiments, it can be concluded that the efficiency of text classification can beimproved by means of vector-distance-weighted algorithm.Keywords: text classification, vector-distance-weighted algorithm, Natural Language Process.1. IntroductionText classification is to assign one or more appropriate class-types for a text basedon its content. In western countries, the study about text classification and its relative fields begins asearly as 60 or 70s last century, when Salton put forward the VSM (Vector Space Model) theory andthe VSM is used successfully in application. The study on Chinese text classification by means ofcomputer starts in the 90s. The research has achieved a lot, for example, FuDan University andInstitute of Computing Technology of Chinese Academy of Sciences have tracked and studied theTREC test. Early on PeKing University and TingHua University have made a study of the technologyof web classification on their search engine "Network Sky" and "The guide of Network" respectively[LIU Bin, HUANG Tie Jun, CHENG Jun, GAO Wen 2002]. The results of these fields are not assatisfactory as those in English research because of the uniqueness of Chinese language.There are two methods for text classification. One is the rule method. This method is appliedearlier. For example, the Construe [Church , K.W.Lisa.F.Rau 1995] is developed based on it. The othermethod is based on statistics, such as Bayes, VSM, KNN, SVM and so on. In the following section,the authors will discuss the expression of texts in VSM, computation of the feature weight, thesimilarity between text and classes, and the vector-distance algorithm. At the same time, the results ofexperiments are figured out and analyzed.2. Vector Space ModelVector Space Model (VSM) is widely applied in 111. system for its simple conceptions and itssimulation of close space to close meanings. The classification method used in texts is introduced fromIR system.2.1. Vector expression of training textsA text is composed of characters, words and phrases which are termed as the features of text.According to "Bayes hypothesis", presuming the effects of features to the class adscription areEll This paper is supported by Nature Science Foundation of Hubei, China (ID:2001ABBol2)235independent, the text can be expressed as the vector of feature collection. After the handling of thetraining set, "Term-Documents" matrix space Aca could be obtained. The row and column areexpressed by the feature and text respectively. So we can get the training set vectors space showing infigurel.doc1 doc2 doc3 	doc dti a1,1ai,t  • • • atdFigurel.In general, A is a sparse matrix, it can be compressed by using the method described in [Than xuegang, Lin Hongfei, Yao Tianshun 1999].2.2. Feature weightingFrom section 2.1, the matrix Aba can be inferred .If au is used to express delegate thenon-zero element, then A =[ai,i ]d . In order to show the importance of the features in texts, al,;can't be expressed by the frequency of the features that occur in texts. Generally, feature weight iscomputed and normalized. TF-IDF model is used to compute the term weight as follows: [21Local(i, j) * Global(i) (1)Where, air; is the term weight; Local(i, j) = log2 (1 + t si,./ ) , bi, i is the frequency of term i indocument j. GlobaAi) = loge an I dfi ) + 1) , n is the number of the train set, dfi is the number ofdocuments containing term i.A new document, formula (1) can be used to calculate the term weight. Because n.=1 and dfi =1,so Globa(i) = 1 . In the experiments, formula (2) is used to compute the term weight and formula (3)is used to normalize it:(1+10	 nlog(tfi, j)1 + logqi) 	 (2) Sti,i = .44 (Si, j)2 	Ls, i j =10* 	 	 (3)\	 IWhere, Si,; is the weight of term i in document j, l i is the length of the document j, and the[2] about other model please read the paper[Diao Qian, Wang Yongcheng, Zhang Huihui, He Ji 2000] and the chapter 15 of[Christopher D.Manning, Hinrich Schutze]a - •,, = 11±(Local(i,j)*Global(i)Yi=1236frequency of terms in document is summed up as the document length.2.3. Feature selectingThere are multi-selection for features, such as characters, words, phrases or their compounds. It isuniversally accepted that using words as features is superior to other selections. Words can beextracted directly from texts. Because automatic extracting of words is not satisfactory, segmenttraining set is then the first choice. After segmentation, mutual information method can be used tofilter the feature. In literature [LIU Bin, HUANG Tie Jun, CHENG Jun, GAO Wen 2002], the authorsemploy characters and characters plus words as the features respectively, and compare the results. Theresults improved little as shown, after adding more than 200 000 words as features. So, employing thecharacters as the features for text classification study is meaningful.In experiments, characters are selected as the features, but not all the 6763 Chinese charactersdefined in GB2312-80. The characters of training set are selected as features. The number is less thanGB2312-80. There are 5468 characters in the training set.3. Vector-distance algorithmThere are many algorithms which are based on vector space, such as Support Vector Machine,Nerval Network, KNN, Bayse, Vector-distance and so on. In this paper, Vector-distance algorithm isemployed. The simple vector-distance algorithm is used in the vector space model. Simply speaking,this algorithm is used to compute the vector distance between the document to be classified and theclasses of training set. Two methods are used in the experiments.Method I : vector-distance weighted algorithm.This is an algorithm to compute the weighted similarity, that is, handling every text of training setto get training set matrix, handling the text to be classified to get vector, computing the similaritybetween the texts in training set and the text to be classified, and then weighting the similarity, if thetraining texts have the same class. The main steps are as follows:Step 1: Formula (1) is used to handle the training set text vector to get the training set matrixspace;Step2: Formula (2) is used to handle the new document vector, while formula (3) is used tonormalize the vector;Step3: Formula (4) is used to figure out the similarities;n	 nCos (c1	s' J ,k	 a ",k x j ,k 	  	 (4)Where, di is the feature vector of the new text, di is the vector of jth document intraining set, n is feature number;Step4: The same class training set texts are judged in order to calculate the weighted similarities,using formula (5):SumSim	 i, C  =	 Cos (di,d,) *T (Ci t ,C i)	  	 ( 5)t=1where, d; is the new document, Ca(clocOis as the same as the formula (4), n isk=1	 k=1 k=1237documents' number of the collection , T(c1„Cf ) is the class determine function. Suppose4 belongs to class C j , then T(d,Cf ). 1 , otherwise Rdt ,Cd= 0.Steps: The classes are sorted in descending according to the similarities computed in step4, andare outputted; the first one is the class the new document will belong to.Method II: representative-vector-distance algorithmIn this method, the representative vector vc i is formed by the mergence of every same classtraining texts. When there is a new document, construct a vector di for it. Then calculate thecosine-distance (similarity) of d i and vci , then sort the similarity and output the result, the mainsteps are as follows:Step 1: Handle every kind of texts of the training set to get all the kinds of representative vectors.Furthermore, get the representative vector matrix of the whole collection, to get thenormalization matrix, using formula (1);Step2: The vector of new document is gotten by using formula (2) and (3);Step3: The similarities are calculated by using formula (4);Step4: At last, get the class that the new document belongs to, according to the size of thesimilarities.4. EvaluationThe effectiveness of text classification is measured as Recall and Precision calculated by the followingequations:the number of classification that arePrecision  correctly assigned to documents r c the number of training setRecall =the number of classification that arecorrectly assigned to documents the number of classification that are correctlyassigned to documents in whole training setIn order to synthetically consider the effectiveness of text classification, F 1 test value is used asfollows:Fl test value  Pr ecision x Re call x 2 Pr ecision + Re call5. Results and Discussions,238In experiments, the authors employ the Modern Chinese Corpus of State Language Commissionas training set and use vector-distance algorithm to implement the process of automatic textclassification. The corpus is a balanced corpus and has been manually classified. It includes three parts,namely: Human& Society Science, Natural Science and Integration. Politics, history, society, economy,arts, literature, military affairs& gym, and life, are the 8 classes included in the first part; mathematics& physics, biology & chemistry, astronomy & geography, medicine& sanitation, agriculture &forests,ocean & weather are included in Natural Science, and Integration part contains application documentsand others two classes. The size of every text is about 3-4kB, and the content of the texts is selectedfrom newspapers, books and general magazines.In the experiments, 11 classes are randomly selected, including literature-colloquialism,politics-law, society-education, mathematics & physics, biology & chemistry, military affairs& gym,astronomy & geography, medicine& sanitation, arts, agriculture &forests, ocean & weather, from eachof which 100 passages are selected. The total amount is 1100. Half of them are selected as training set,and the rest for test. Two experiments have been done with them separately. Experiment 1 is done withmethod I, and experiment 2 with method II. The results of experiment 1 and experiment 2 are revealedin table 1 and table 2 respectively [Note: Literature is for Literature-colloquialism, politics is forpolitics-law, society is for society-education, mathematics is for mathematics & Physics... and so on].Tablel. The data of experiment 1Right Recall% Precision%Literature 39 78 98Politics 49 98 87.46Arts 16 32 93.82Medicine 18 36 94Astronomy 38 76 96.55Mathematics 40 80 97.82Biology 23 46 94.91Society 44 88 86.18Agriculture 20 40 94.55Military 4 8 91.64Ocean 50 100 89.09Tablet. The data of experiment 2Right Recall% Precision%Literature 25 50 95.46Politics 50 100 83.09Arts 11 22 92.09Medicine 12 24 93.09Astronomy 33 66 95.46Mathematics 50 100 78.18Biology 5 10 91.82Society 32 64 89.64Agriculture 14 28 93.46Military 7 14 92.18Ocean 34 68 95.27239Table 3.The comparison of experiment 1 &experiment 2Correct pages Precision Recall Fl valueExperiment 1 341 62% 62% 62%Experiment 2 273 49.64% 49.64% 49.64%It is obviously shown in the above figures that:1. The effectiveness of experiment) is better than that of experiment2. That is to say, themerging of the same class training texts to get representative vector for classificationeffectiveness does not work well.2. In experiment), the classification effects of literature-colloquialism, politics-law,society-education, Mathematics & physics, astronomy & geography, ocean & weather arebetter and more stable than others, especially politics-law, society-education, mathematics &physics, ocean & weather. The Recall of them achieved as 80% or even more. The mainreason is that the features of these classes are distinct, thus little influence from otheroverlapping classes is found.3. The Recall of biology & chemistry, military affairs& gym, medicine& sanitation, arts andagriculture &forests is low, especially that of military affairs& gym, only about 10%. Afteranalyzing the corpus and the outcome of the experiment on these classes, it is found that:0 The boundaries of classes are seriously overlapped. The discrimination becomes lower,when using the characters as the features.Some samples in corpus are excerpts of one or several passages. Then, it is likely that thetitle of the sample is about medicine and is classified into medicine& sanitation, however, thecontent of the text is about chemical components and chemical reactions of the medicine, sothe text is judged as biology & chemistry class by machine.© The corpus of military affairs& gym contains military affairs and gym. After the corpusis analyzed, it is found that the majority of military documents contain historical and militaryfacts, and they overlap with politics-law. The samples of gym overlap with those ofmedicine& sanitation and education in content. The samples which overlap with otherclasses' samples are classified correctly, but they are classified wrongly by machine. Thesame reason is found in other classes whose Recall is low.For above problems, the effectiveness of case Q can be improved by means of changing thefeature selection, such as selecting word as the feature. As for the case CD, it is difficult to improve theeffectiveness by statistical method only. Semantic comprehension should be added to help us improvethe classification effectiveness.4. The authors select characters as features and don't filter the feature set, this probably leads tonoise [Than xue gang, Lin Hongfei, Yao Tianshun 1999, Schutze H, Hull D, Pedersen J. 1996,ZHOU Shui-Geng, GUAN Ji-Hong, HU Yun-Fa, ZHOU Ao-Ying 2001], for example, theinfluence from number, and make the classification effectiveness decline. In the experiments,after the handling of the texts to Unicode, then every number code becomes a feature, so thefunction of number is larger than before.5. The data of table 1, 2, 3 are the numbers of successfully classified documents, and each ofthem has the largest similarity with training set class, and into which they are classified. Inthe analysis of the corpus, in each class it is found that the categories of some texts judged by240machine are different from their original categories which are classified manually. The resultsjudged by machine are regarded as correct, by means of artificial discrimination. Therefore,the results of classification are correct if they are of this case. According to this principle, thenumber behind "+" indicates the increasing correct result.Table4.The adjusted data of experiment 1	 Tables. The adjusted data of experiment 2(based on table 1)	 (based on table 2 )Correct WrongLiterature 39+1 10Politics 49+1 0Arts 16+11 23Medicine 18+3 29Astronomy 38+1 11Mathematics 40+1 9Biology 23+2 25Society 44+4 2Agriculture 20+1 29Military 4+23 23Ocean 50 0Total 341+48 161Correct WrongLiterature 25+2 23Politics 50 0Arts 11+12 27Medicine 12+5 33Astronomy 33+7 10Mathematics 50 0Biology 5+16 29Society 32+9 9Agriculture 14+7 29Military 7+19 24Ocean 34+7 9Total 273+84 193Table6. The comparison of experiment 1 &experiment 2 after the first adjustCorrect pages Precision Recall F 1 valueExperiment 1 389 70.73% 70.73% 70.73%Experiment 2 357 64.91% 64.91% 64.91%. According to the analysis of the experiments, the result of classification in military affairs &gym, medicine & sanitation, arts, biology& chemistry, agriculture& forests is unsatisfactory. Itis partly due to the overlapping of the feature set or class boundary. According to the results ofexperiments and analysis of the corpus, the authors admit a document can be classified intomore than one class. So, in class similarities sort list, the first-three classes of the newdocument are observed. If any one of them among the three is the same as the original class, itmeans that the classification of the new document is successful. (This adjustment is only formilitary affairs & gym, medicine & sanitation, arts, biology& chemistry, and agriculture&241forests.) The reasons to do so are: first, the documents in corpus may have several classes;second, the cosine distance discrepancy is very small among the first-three when comparedwith others; third, using this method, setting the threshold can be avoided. Table7, 8 arerelated data.Table7. The data of experiment 1 before and after the second adjustment (based on tablel)Arts Biology Military Medicine Agriculture Correct Precision RecallBeforeadjust16 23 4 18 20 81 62% 62%Afteradjust48 37 33 30  34 182 80.36% 80.36%Table8. The data of experiment 2 before and after the second adjustment (based on tablet)Arts Biology Military Medicine Agriculture Correct Precision RecallBeforeadjust11 5 7 12 14 49 49.64% 49.64%Afteradjust38 34 32 24.30 158 69.27% 69.27%The results in tablel 8 is better, including that of literature-colloquialism, politics-law,society-education, mathematics & physics, astronomy & geography and ocean & weather, of whichthe data are not adjusted yet. The average precision of experiment) and experiment2 is 80.36% and69.27%, respectively.Comparing the two experiments, it can be concluded that the efficiency of text classification canbe improved by means of vector-distance weighted algorithm.6. References[LIU Bin, HUANG Tie Jun, CHENG Jun, GAO Wen 2002] A New Statistical-based Method inAutomatic Text Classification, Journal of Chinese Information Processing Vol.16 No.6.[Church , K.W.Lisa.F.Rau 1995] Commercial Applications of Natural Language Processing ,Communications of ACM,Vol.38.No.11[Than xue gang, Lin Hongfei, Yao Tianshun 1999] Hierarchical Method for Chinese DocumentClassification. Journal of Chinese Information Processing Vol.13 No.6 1999.[Diao Qian, Wang Yongcheng, Zhang Huihui, He Ji 2000] Term Weighting and ClassificationAlgorithms. Journal of Chinese Information Processing, Vol.14 No.3 2000.[Christopher D.Manning, Hinrich Schutze] Foundations of Statistical Natural Language Processing. TheMIT Press Cambridge,Massachusetts London, England.[Schutze H, Hull D, Pedersen J.1996] A Comparison of Selective Bayesian Network Classifiers.ICML-96,1996.[ZHOU Shui-Geng, GUAN Ji-Hong, HU Yun-Fa, and ZHOU Ao-Ying 2001] A Chinese DoucumentCategorization System without Dictionary Support and Segmentation Processing, Journal of ComputerResearch & Development Vol.38 No.7 2001.242

A Vector-Based Algorithm for Chinese Text Classification

Luo, Chang Ri

49746

He, Ting ting

49747

Waseda University Repository

https://waseda.repo.nii.ac.jp/?action=repository_action_common_download&item_id=28786&item_no=1&attribute_id=101&file_no=1

A Vector-Based Algorithm for Chinese Text Classification

Abstract

Similar works

Full text

Available Versions

DSpace at Waseda University

Waseda University Repository