In information retrieval (IR) systems, there are a query and a collection of documents compared with this query and ranked according to a particular similarity measure. Since texts with the same content can be written by different authors, the writing styles of the documents change as well accordingly. This observation brings the idea of investigating text by means of style. In this paper, we analyze text documents in terms of stylistic features of the written text and measure effectiveness of these features in an IR system. Our main focus is on Turkish text documents. Although there are many studies about broadening IR systems with style based enhancement, there is no similar application for Turkish which performs retrieval depending purely on style. © 2009 IEEE

Can F.

Kalaycilar F.

Kale, A.

Pehlivan, S.

Zamalieva, D.

Bilkent University Institutional Repository

978-1-4244-5023-7/09/$25.00 ©2009 IEEESeptember 14-16, 2009METUNorthern Cyprus Campus710Stylistic Document Retrieval for TurkishDaniya Zamalieva, Fırat Kalaycılar, Aslı Kale, Selen Pehlivan, Fazlı CanDepartment of Computer EngineeringBilkent University06800 Bilkent Ankara, TurkeyEmail: {daniya,firatk,akale,pselen,canf}@cs.bilkent.edu.trAbstract—In information retrieval (IR) systems, there are aquery and a collection of documents compared with this queryand ranked according to a particular similarity measure. Sincetexts with the same content can be written by different authors,the writing styles of the documents change as well accordingly.This observation brings the idea of investigating text by meansof style. In this paper, we analyze text documents in terms ofstylistic features of the written text and measure effectiveness ofthese features in an IR system. Our main focus is on Turkish textdocuments. Although there are many studies about broadeningIR systems with style based enhancement, there is no similarapplication for Turkish which performs retrieval dependingpurely on style.I. INTRODUCTIONDocument retrieval systems try to match given queries witha collection of unstructured text documents. In general, theypresent a list of documents ordered from most relevant to lessrelevant. Distances between the query and each document arecomputed using several features. One common feature is thenumber of indexing terms shared by the query and a document.This shows that classical retrieval systems are mainly based onthe similarity measurements such as frequencies of commontextual units like words, phrases, indexing terms etc.[1]In this study, we concentrate on the style of documentsrather than their contents. Style comprises the structural andsyntactic choices of an author that are independent fromthe subjects of writings. This indicates the uniqueness ofauthor in terms of linguistic tendencies. Since there is noformal definition for document style, firstly the investigation ofcandidate features representing document style was conducted.Based on this notion, we present a Turkish documentretrieval approach relying on stylistic features. In this system,we model stylistic tendencies in terms of certain measurementsand propose a numerical representation corresponding to doc-ument style. Input of the system is a textual unit, preferablya query document or a query paragraph. During retrieval,in contrast to the classical methods, this system computesdistances between the query and the documents using theirnumerical style representations. Therefore the resulting listcontains documents with similar stylistic tendencies.One possible use of our system can be discovery of theprobable author of an anonymous text like poem, short story,and newspaper columns. For example, there are a lot ofliterary works whose authors are unknown. We suggest thatby comparison of these anonymous works with the writingsof certain authors in terms of style, the probable author can bedetermined. In addition, this system can serve the users whoseek for documents that have resemblance in style with theone they are interested in. By this way, people can discoverother authors which create literary works they would probablybe attracted to.The document collection we used consists of newspapercolumns written by different authors. We expect that for anew document (query) written by an author who has severalarticles in the collection, our system will primarily retrieve theexisting documents of that writer.The rest of the paper is organized as follows. In Section II, abrief discussion of related work is given. Section III containsa detailed description of the retrieval approach and stylisticfeatures. Section IV presents performance evaluation using acollection of newspaper columns. Finally, Section V providesa summary and lists the contributions.II. RELATED WORKMost of the effort on information retrieval systems coversenhancement strategies to increase retrieval performance. Apart of the research is based on methods that enhance retrievalby including topic information. These approaches put topicrelevance into consideration during retrieval. On the otherhand, [2] states that deviations among documents are not onlytopical but also stylistic.Analyzing subjective values of a given text in IR systems isa multi-disciplinary area including linguistics and computerscience. Generally, each author has his own characteristicwriting style that is independent of the topic. This inspirationexpands approaches to improve IR systems with stylisticfeatures. [2] describes a prototype which introduces stylisticitems used for measurements of document style. Measure-ments with respect to these items are then merged using non-parametric multivariate method, such as decision tree learningapproach. Study is tested on The Wall Street Journal articlesand materials from Internet. According to [2], stylistic itemsare divided into two sets: lexical statistics including averageword length, long word counts, number of pronouns, numberof digits, and syntactic statistics such as average length of asentence.Similar work is presented by Argamon et. al [3]. Themain aim of the study is to find a computationally efficientformulation of linguistic features to classify text styles. Then,machine learning techniques are applied to build a model todiscriminate styles.663711Fig. 1. System Overview.Some other studies analyzing text style are based on authorattribution [4], [5], [6], genre attribution [7] and semanticorientation [8]. For example, Argamon et al. [5] learn themodels for discrimination of different authors by applyingseveral multiclass variants of the Winnow [9] algorithm to thefeature vectors corresponding to texts. Finn et al. [7] discussesand compares three different approaches for classification ofdocuments by genre: bag of words techniques, part-of-speechstatistics, and hand-crafted shallow linguistic features. Turneyand Littman [8] introduce an algorithm for unsupervisedlearning of semantic orientation, the evaluative character of aword, from text. Similarly, Wiebe et al. [10] presents a corpusannotation project to investigate issues in manual annotationof private states in language such as opinions, emotions,sentiments, speculations, and evaluations.Motivated by the previous research work discussed aboveand the lack of studies on document retrieval systems forTurkish, our work presents a style oriented method for TurkishIR systems.III. STYLISTIC DOCUMENT RETRIEVALIn this study, we want to represent each document interms of a set of stylistic features. Firstly, the investigation ofcandidate features that can adequately describe the documentstyle is performed. Details regarding the determined features,their properties, and feature extraction are presented further inthis section.The retrieval mechanism can be explained in the followingway: first, the features are extracted from each documentof the collection and stored. Then, the same features areextracted from a query document and its feature vector iscompared with those obtained from the document collection.The similarity between two feature vectors can be calculatedusing the Mahalanobis distance [11]:Similarity(xi,xj ,H) = (xi − xj)T H−1(xi − xj) (1)where xi and xj are feature vectors, H is covariance matrix.For example, when H is chosen to be an identity matrix,the similarity measure corresponds to the Euclidean distance.TABLE IMEASUREMENTS USED FOR FEATURE CALCULATIONS.Abbreviation DescriptionC Number of conjunctionsCS Number of certainty suffixesCW Number of certainty wordsFP Number of formal pronounsFW Number of formal wordsNA Number of negative adjective/adverbP Number of pronounsPA Number of positive adjective/adverbQ Number of questionsS Number of sentencesSW Number of stop wordsSYL Number of syllablesUS Number of uncertainty suffixesUW Number of uncertainty wordsW Number of wordsHowever, since the Euclidean distance treats all feature compo-nents evenly, some problems can arise. The difference betweenfeature components with high variance can dominate overthose of other components and drastically affect the resultantsimilarity value. Therefore, in order to avoid this problem,each feature component should be normalized by using theirvariances. Hence, we define H as diagonal matrix whoseentries are variances estimated from the document collection.When the Mahalanobis distance is employed, the featurecomponent ranges are equalized.After the similarity values between the query documentand document collection are computed, the documents of thecollection are ranked accordingly from the most similar to leastsimilar. The overview of the implemented system is shown inFigure 1.Below we discuss the features that are significant for stylerepresentation of documents. The measurements used forfeature calculation and their abbreviations are listed in Table I.Formality Measure: This feature defines a measure for theauthor’s writing style in terms of formality. Our aim is todiscover the structures that define formality of the document.It can be assumed that some authors tend to use formalexpressions more frequently. Especially this is apparent whenwriter addresses the reader. In Turkish, some pronouns andsome word suffixes can be used to identify formality measureof the texts.The formality score is assigned according to the frequencyof formal structures in the current document. In Turkish, sizpronoun can be used for politeness, addressing strangers, andshowing respect. In addition, there are some words generatedby adding a formal suffix -iniz that reflect formality. Based onthese assumptions, the formality measure can be calculated as664712TABLE IISAMPLE ENTRIES OF TERM AND SUFFIX LISTS.Positive Terms Negative Terms Certainty Terms Uncertainty Terms Conjunctions Formality Suff. Certainity Suff. Uncertainty Suff.güzel çirkin şüphesiz belirsiz ama -iniz* -malı* -ebilir*mutlu mutsuz bariz sanki fakatdostça iğrenç muhakkak belki lakincandan berbat mutlaka muallak vecici nankör kesinlikle meçhul veya. . . . . . . . . . . . . . .431 terms 737 terms 12 terms 11 terms 50 terms 1 suffix 1 suffix 1 suffix* and derivatives such as -ınız, -meli and -abilir.Formality =12(FPP+FWW − SW). (2)We formulated the equations by dividing the numberof formal words in the document to the total number ofwords except the stop words. We discarded the number ofstop words used in the document to reduce bias. This leadsto a better evaluation of the text in terms of proposed features.Positivity and Negativity Measure: We seek for a way tounderstand the general style of an author in terms of mood andemotion. We observed that some authors tend to favor positivewords while others prefer to use negative words. Thus, weassume that evaluation of positivity and negativity in the wholetext provides the idea about the general emotional tendency ofthe author.Based on this assumption, the positive (negative) terms listthat contains adjectives representing the positive (negative) at-titude is constructed. The words are selected manually from theadjectives and adverbs published in the TDK web dictionary[12]. To provide objectivity, we select the words that are well-known as positive (negative) in Turkish. Refer to Table II forsample terms. Then, these lists are used to assign a positivity(negativity) score to the document as follows:Positivity =PAW − SW(3)Negativity =NAW − SW. (4)Certainty and Uncertainty Measure: It can be observed thatsome authors prefer to write in a more certain (uncertain) man-ner than others. This attitude is expressed both by words andsuffixes reflecting certainty (uncertainty). To assign a certainty(uncertainty) score based on the certain (uncertain) wordsusage, the certainty (uncertainty) terms list containing thewords that represent the sureness and definiteness (unsurenessand doubtfulness) is prepared and the frequency of occurrenceof these terms in the current document is calculated as follows:Certainty =12(CWW − SW+CSW − SW)(5)Uncertainty =12(UWW − SW+USW − SW). (6)Question Usage Measure: Some authors prefer to developtheir writings by interacting with readers by asking questions.Since questions can be located by question marks, they arecounted in the text to calculate the number of questions:QuestionUsage =QS(7)Stop Word Usage Measure: The general tendency in textanalysis is to eliminate stop words prior to processing thedocuments. However, usage of stop words in the text cangive a clue about author style. For example, some authors usemany stop words in their texts while others avoid using themfrequently. We obtain the Turkish stop word list containing 114words from [13]. By using the list, the occurrence frequencyof these terms in the document is calculated by:StopWordUsage =SWW(8)Conjunction Usage Measure: Conjunctions are basic textualunits that generally connect two phrases. Hence, the overuseof conjunctions implies complex and long sentences that arealso distinctive features of document style. By using theconjunction term list from TDK web dictionary [12], themeasure is evaluated asConjunctionUsage =CW(9)Word Length Measure: Since word length measure is widelyused in document style analysis, we include it in our featureset. We measure the average length of the words in terms ofsyllable as follows:665713Fig. 2. Average Sentence Length of Documents Written by Different AuthorsWordLength =SYLW. (10)Sentence Length Measure: A significant property revealingstylistic characteristics of a document is the average lengthof the sentences. We suggest that this measure is adequatefor style discrimination. For example, long sentences gener-ally correspond to complex expressions whereas shorter onesconvey simpler ideas. The formulation of this measure is givenasSentenceLength =WS. (11)IV. EXPERIMENTAL WORKA. Document CollectionWe evaluate the proposed system by using the documentcollection that consists of Turkish newspaper articles writtenby different authors. There are 13 authors and 60 articles foreach. The authors are Taha Akyol, Melih Aşık, Fikret Bila,Berrin Cankat, Hasan Cemal, Güneri Civaoğlu, Abbas Güçlü,Hurşit Güneş, Sami Kohen, Hasan Pulur, Derya Sazak, OsmanUlagay, and Mehmet Ali Birand.B. Author Style AnalysisWhen all authors are compared according to a stylisticfeature component, we can examine that some stylistic featuresare discriminative for a subset of authors. For example, OsmanUlagay prefers to use longer sentences than Berrin Cankat ascan be seen from Figure 2. Moreover, authors have differentuncertainty degrees in their writings as illustrated in Figure3. However, some feature components have similar values formost of the authors so they are not distinctive enough likeword length shown in Figure 4.Fig. 3. Author vs. Average Value of UncertaintyFig. 4. Average Word Length of Documents Written by Different AuthorsC. System EvaluationThe performance of the system is evaluated by precisionmeasure. Precision is the percent of retrieved documents thatare relevant to the query. For using this measure, the relevanceinformation between a query document and the documentcollection is required. There is no available data set forTurkish stylistic document retrieval systems with appropriategroundtruth data. Thus, we assume that most authors tend tohave their own style and two documents are considered asrelevant if they are written by the same author.We tested the system by using 780 query documents fromall authors. After ranking the documents as explained inSection III, the corresponding precision values are calculatedfor the first 60 retrieved documents. By taking average ofprecision values of all these queries, we obtain the precisiongraph shown in Figure 5 (corresponding to solid black curvelabeled as all). When we consider the first 5 and 10 retrieveddocuments, the precision of the system is about 0.84 and 0.77respectively. These results show that although document styleis a fuzzy and subjective concept, the performance achievedis high. Thus, it can be concluded that chosen features aresuccessful enough to capture document style. Notice that thelowest precision obtained is 0.53.666714Fig. 5. Number of Retrieved Documents vs. Average PrecisionIn addition, we perform retrieval based solely on eachfeature component to investigate the effectiveness of it. Theprecision corresponding to each feature is presented on Figure5. It is obvious that none of the feature components couldoutperform the combination of them. Besides this, we can inferthat the most discriminative features are formality measure andsentence length. This confirms that authors tend to preservethe formality level and sentence length in their writings. Incontrast, since word length and positivity features show lowretrieval performance, we can conclude that they are lessdiscriminative.V. CONCLUSIONSIn this study, we develop a stylistic document retrievalapproach for Turkish. We investigate the features that are ableto capture the style of the document and use them for retrievalof the documents that are similar in style. Although generallystyle is used to enhance the existing retrieval systems, weimplement a retrieval approach that is based purely on stylisticfeatures. It is specialized for Turkish by construction of theterm lists consisting of Turkish words.As future work, we will examine other features which couldenhance our retrieval performance. For example, usage ofOttoman Turkish words or slang existence can be investigatedas new stylistic features. Moreover, the proposed system canbe embedded to other retrieval systems in order to enhancetheir retrieval precision.REFERENCES[1] G. Salton, “Another look at automatic text-retrieval systems,” Commu-nications of the ACM, vol. 29, no. 7, pp. 648–656, 1986.[2] J. Karlgren, “Stylistic experiments for information retrieval,” 2000.[3] S. Argamon, C. Whitelaw, P. Chase, S. Hota, N. Garg, and S. Levitan,“Stylistic text classification using functional lexical features,” Journal ofthe American Society for Information Science and Technology, vol. 58,no. 6, pp. 802–822, 2007.[4] A. McEnery and M. Oakes, “Authorship studies/textual statistics.” 2000.[5] S. Argamon, M. Šarić, and S. Stein, “Style mining of electronic mes-sages for multiple authorship discrimination: first results,” in Proceed-ings of the ninth ACM SIGKDD international conference on Knowledgediscovery and data mining. ACM New York, NY, USA, 2003, pp.475–480.[6] D. Madigan, A. Genkin, D. Lewis, S. Argamon, D. Fradkin, and L. Ye,“Author identification on the large scale,” in Proc. of the Meeting of theClassification Society of North America, 2005.[7] A. Finn, N. Kushmerick, and B. Smyth, “Genre classification anddomain transfer for information filtering,” Lecture notes in computerscience, pp. 353–362, 2002.[8] P. Turney and M. Littman, “Unsupervised learning of seman-tic orientation from a hundred-billion-word corpus,” Arxiv preprintcs.LG/0212012, 2002.[9] N. Littlestone, “Learning quickly when irrelevant attributes abound: Anew linear-threshold algorithm,” Machine learning, vol. 2, no. 4, pp.285–318, 1988.[10] J. Wiebe, T. Wilson, and C. Cardie, “Annotating expressions of opinionsand emotions in language,” Language Resources and Evaluation, vol. 39,no. 2, pp. 165–210, 2005.[11] P. Mahalanobis, “On the generalized distance in statistics,” in Proceed-ings of the National Institute of Science of India, vol. 12, no. 1, 1936,pp. 49–55.[12] TDK. Türk dil kurumu. [Online]. Available: http://www.tdk.gov.tr[13] Turkish stopwords. [Online]. Available:http://www.ranks.nl/stopwords/turkish.html667

Stylistic document retrieval for Turkish

http://repository.bilkent.edu.tr/bitstream/11693/28655/1/Stylistic%20document%20retrieval%20for%20Turkish.pdf

Stylistic document retrieval for Turkish

Abstract

Similar works

Full text

Available Versions

Bilkent University Institutional Repository