Koutraki, Maria

Sack, Harald

Türker, Rima

Zhang, Lei

English

KITopen

“The Less Is More” for Text ClassificationRima Türker1,2, Lei Zhang1, Maria Koutraki1,2, and Harald Sack1,21 FIZ Karlsruhe – Leibniz Institute for Information Infrastructure, Germany2 Karlsruhe Institute of Technology, Institute AIFB, Germany{firstname.lastname}@fiz-karlsruhe.de{firstname.lastname}@kit.edu1 IntroductionText Classification [2,5] is gaining more attention due to the availability of ahuge number of text data, such as blog articles and news data. Traditional textclassification methods [1] use all the words present in a given text to representa document. However, the high number of words mentioned in documents cantremendously increase the complexity of the classification task and subsequentlymake it very costly. Moreover, long (natural language text) documents usuallyinclude a different variety of information related to the topic of a document. Forexample, encyclopedic articles such as the life of a scientist3, contain besides topicrelated content also detailed biographical information. Often, in such articlesafter the first paragraph (or first a few sentences), words or entities appear, whichare not related to the main topic (or category4) of the article. We assume thatthe most informative part of such articles is limited to a few starting sentences.In other words, instead of considering the complete document, only its beginningcan be exploited to classify a document accurately.In this study, we design a Knowledge Based Text Classification method, whichis able to classify a document by using only a few starting sentences of the article.Since the length of the considered text is rather limited, ambiguous words mightlead to inaccurate classification results. Therefore, instead of words, we considerentities to represent a document. In addition, entities and categories are embed-ded into a common vector space, which allows capturing the semantic similaritybetween them. Moreover, the similarity based approach does not require any la-beled training data as a prerequisite. Instead, it relies on the semantic similaritybetween a set of predefined categories and a given document to determine whichcategory the given document belongs to. The study has been validated with pre-liminary experiments on text classification for encyclopedic articles, which showthat our method achieves comparable and even better results using only the firstfew sentences of a document than using the entire document.2 Knowledge-Based Text Classification (KBTC)Given a Knowledge Base KB containing a set of entities E = {e1, e2, .., en} anda set of hierarchically related categories C = {c1, c2, .., cm}, where each entity3 http://scihi.org/albert-einstein-revolutionized-physics/4 https://en.wikipedia.org/wiki/Category:Physics2 R. Türker et al.Fig. 1. The work flow of the Knowledge Based Text Classification approach (bestviewed in color)ei ∈ E is associated with a set of categories C ′ ⊆ C. The input is a text t, whichcontains a set of mentions Mt = {m1, . . . ,mk} that uniquely refer to a set ofentities. Then, the output is the most relevant category ci ∈ C ′ for the giventext t.KBTC Overview. The general work flow of Knowledge Based Text Classifica-tion is shown in Figure 1. The first step is “Mention Detection Based on Anchor-Text Dictionary”, where each entity mention present in t is detected based ona “Anchor-Text Dictionary” prefabricated from Wikipedia. The Anchor-TextDictionary contains all mentions and their corresponding Wikipedia entities. Inorder to construct an Anchor-Text Dictionary all the anchor texts of hyper-links in Wikipedia articles referring to another Wikipedia article are extracted,whereby the anchor texts serve as mentions and the Wikipedia article links referto the corresponding entities. In the second step, for each detected mention inthe given input text candidate entities are generated based on the Anchor-TextDictionary. In our example these are “Motorola”, “Hewlett-Packard” and “Linux”.Likewise, the predefined categories are mapped to Wikipedia categories. Finally,with the help of entity and category embeddings [3] that have been precom-puted from Wikipedia, the output is the semantically most related category forthe given entities. Thereby, in the given example the category Technology willbe determined.Probabilistic Model. The proposed classification task is formalized as estimat-ing the probability of P (c|t) of each predefined category c and an input text t.Based on Bayes’ theorem, the probability P (c|t) can be rewritten as follows:P (c|t) = P (c, t)P (t)∝ P (c, t) (1)“The Less Is More” for Text Classification 3where the denominator P (t) has no impact on the ranking of the categories. Foran input text t, a mention is a term in t that can refer to an entity e and thecontext of e is the set of all other mentions in t except the one for e. For eachcandidate entity e in t, the input text t can be decomposed into the mention andcontext of e, denoted by me and Ce, respectively. Based on the above introducedconcepts, the joint probability P (c, t) is given as follows:P (c, t) =∑e∈EtP (e, c, t) =∑e∈EtP (e, c,me, Ce)=∑e∈EtP (e)P (c|e)P (me|e)P (Ce|e) (2)where Et represents the set of all possible entities contained in the input text t.Here, we simply apply a uniform distribution to calculate P (e) for each en-tity e. The probability P (c|e) models the relatedness between an entity e and acategory c, which is estimated by using the prefabricated entity-category embed-dings. Moreover, the probability of P (me|e) is calculated based on the anchortext dictionary. Finally, the probability P (Ce|e) models the relatedness betweenthe entity e and its context Ce. Each mention in Ce refers to a context entityec from the given knowledge base. The probability P (Ce|e) can be calculatedwith the help of entity-category embeddings. More details about the probabilityestimation can be found in [4].3 Results and DiscussionDataset. The proposed text classification approach is evaluated on articles ofSciHi5, a web blog on the history of science. From that dataset6 1452 articlesassociated to a single category have been considered. The different categoriessupported in the dataset are 45 and the average number of sentences per articleis 32.96.1 2 3 5 10 All0.590.60.610.62Number of SentencesAccuracyFig. 2. Performance of the approach fordifferent size of sampled sentences.Number Of Sentences Execution Time1 18 min2 23 min3 28 min5 38 min10 64 minAll 215 minTable 1. Execution Time for differentsize of sampled sentences.5 http://scihi.org/6 https://github.com/ISE-FIZKarlsruhe/Submission-under-review4 R. Türker et al.Experimental Results. The proposed approach does not require any train-ing phase. Therefore, only test sets are generated for the classification task fromSciHi data. To show the impact of the number of the starting sentences of thearticles on the classification accuracy, the data set has been sampled in differentsizes. From each article, the first sentence, first 2, first 3, first 5, first 10 sen-tences and complete documents have been collected. For each sampled datasetsthe proposed approach has been applied to the classification task. The resultsare depicted in Fig. 2. The results show that a few starting sentences (in this case3 sentences) are rather informative and have huge impact on the classificationaccuracy. During the experiments it has been observed that most of the timesirrelevant entities to the corresponding category tend to appear after the first 2or 3 sentences. Hence, after the 3rd sentence the accuracy starts to drop (Fig. 2).Note that usually in such documents the frequency of relevant entities is higherin comparison to irrelevant entities. Therefore, complete documents help to ob-tain reasonable classification accuracy. However, the classification of completedocuments is computationally very expensive (cp. Table 1). The classificationof the whole documents takes 215 minutes while the classification of a sentencerequires no more than 18 minutes for the entire dataset. The best results havebeen obtained with first 3 sentences (Fig. 2), where the execution time was 23minutes, which is almost 90% faster. As expected, the complexity significantlyincreases when the number of sentences is increased.4 Conclusion and Future WorkIn this study, a probabilistic text classification approach has been used to an-alyze the influence of the text length for a text classification task. Based onthe obtained results we can conclude that considering complete document doesnot always increase the classification accuracy. Instead, the accuracy dependson the nature of the considered part of the documents. In this study, it hasbeen observed that the most informative part of encyclopedic documents is thefirst 3 sentences for the classification based on entity and category embeddings.Moreover, as anticipated, the complexity of the classification task decreases byconsidering only a few starting sentences. As for future work, we plan to applythe proposed approach to the different domains such as patent data to be ableto classify patents.References1. Joachims, T.: Text categorization with support vector machines: Learning withmany relevant features. Machine learning: ECML-98 pp. 137–142 (1998)2. Song, Y., Roth, D.: On dataless hierarchical text classification. In: AAAI (2014)3. Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., Mei, Q.: Line: Large-scale infor-mation network embedding. CoRR (2015)4. Türker, R., Zhang, L., Koutraki, M., Sack, H.: Short text categorization usingjoint entity and category embeddings - (under review), https://github.com/ISE-FIZKarlsruhe/Submission-under-review5. Zhang, X., Zhao, J.J., LeCun, Y.: Character-level convolutional networks for textclassification. In: NIPS (2015)

"The Less Is More" for Text Classification

https://publikationen.bibliothek.kit.edu/1000092892/25813508

"The Less Is More" for Text Classification

Abstract

Similar works

Full text

Available Versions

KITopen