Kajiwara Tomoyuki

Yamamoto Kazuhide

English

DSpace at Waseda University

PACLIC 28!644Noun Paraphrasing Based on a Variety of ContextsTomoyuki KajiwaraDepartment of Electrical EngineeringNagaoka University of TechnologyNagaoka City, Niigata, Japankajiwara@jnlp.orgKazuhide YamamotoDepartment of Electrical EngineeringNagaoka University of TechnologyNagaoka City, Niigata, Japanyamamoto@jnlp.orgAbstractWe paraphrase nouns along the contexts ofsentence input on the basis of a variety ofcontexts obtained from a large-scale corpus.The proposed method only uses the numberof types of context, not word frequency or co-occurrence frequency features. This method isbased on the notion that paraphrase candidatesappear more commonly with target words inthe same context. The results of our experi-ment demonstrate that the approach can pro-duce more appropriate paraphrases than ap-proaches based on co-occurrence frequencyand pointwise mutual information.1 IntroductionAlthough extensive and various forms of text dataare easily available in the present age, in order forreaders to gather information effectively, they needtechnology that overcomes any differences in theirlinguistic competence. For example, technology thatburies the difference in the linguistic competence offoreign language learners, children, the elderly, anddisabled persons is useful (Inui and Fujita, 2004).We present our research on paraphrasing to controllanguage at the elementary school level in order tosimplify texts for children. We believe that vocab-ulary simplification for children can be realized byparaphrasing text according to Basic Vocabulary toLearn (BVL) (Kai and Matsukawa, 2002) . BVLis a collection of words selected on the basis on alexical analysis of elementary school textbooks. Itcontains 5,404 words that can help children writeexpressively.As previous work indicated, there are lexical para-phrases that define statements from a Japanese dic-tionary (Kajiwara et al., 2013). The definition state-ments from the Japanese dictionary explain a givenheadword in several easy words. Therefore, lexi-cal simplification and paraphrasing that conserves aparticular meaning are expected by paraphrasing theheadword with the words in the definitions. How-ever, definition statements are short sentences thatconsist of several words. Consequently, there arefew paraphrase candidates, and natural paraphras-ing is difficult even if we use certain dictionariestogether. In addition, the definition statement as awhole is equivalent to the headword; there is noguarantee that any individual word extracted fromthe definition statement can paraphrase the head-word.We propose lexical paraphrasing based on a vari-ety of contexts obtained from a large corpus withoutdepending on existing lexical resources from such abackground. The proposed method is not dependenton language, thus it can perform lexical paraphrasesusing a corpus of arbitrary languages. In this paperwe examine and report on Japanese nouns.2 Related WorksAs paraphrase acquisition from a corpus, a studywith a parallel corpus and comparable corpus hasbeen performed. Barzilay and McKeown paraphrasetext using plural English translations made fromthe same document (Barzilay and McKeown, 2001).In addition, Shinyama and Sekine paraphrase us-ing plural newspaper articles that report the sameevent (Shinyama and Sekine, 2003). In a text sim-Copyright 2014 by Tomoyuki Kajiwara and Kazuhide Yamamoto 28th Pacific Asia Conference on Language, Information and Computation pages 644–649PACLIC 28!645plification task, Coster and Kauchak create a parallelcorpus that matches English Wikipedia and SimpleEnglish Wikipedia, and they perform text simplifi-cation using the framework of statistical machinetranslation (Coster and Kauchak, 2011). However,the technique of using these parallel corpora andcomparable corpora is problematic in terms of theaccuracy of alignment of corresponding expressionsand quantity of the corpora that can be used. Forexample, for Japanese, there is no large-scale par-allel corpus in which simplification is possible foruse in the framework of statistical machine transla-tion. In this paper, we generate paraphrases usingonly a single-language corpus so as not to come un-der these influences.In their research with paraphrasing based onthe similarity of the context obtained from a non-parallel corpus, Marton et al. propose a method forparaphrasing unknown words to improve machinetranslation systems (Marton et al., 2009). They se-lect candidate words with a context common to thesubject. Moreover, they calculate cosine similaritiesof their feature vectors based on the co-occurrencefrequency of subjects. Bhagat and Ravichandran ex-tract paraphrases from a massive, 25-billion wordcorpus (Bhagat and Ravichandran, 2008). They re-gard English word 5-gram as one phrase, and theygenerate feature vectors using pointwise mutual in-formation (PMI) scores. They then select the bestphrase-paraphrase pairs based on their cosine simi-larity.Our proposed method is different from thesemethods in that it does not use co-occurrence fre-quency or word frequency of conventional features.We focus on the variety of context. Assuming thatsuccessful paraphrases have context that is commonwith their subject, we select paraphrases based onlyon the number of types of context.3 Proposed MethodIn this paper, noun paraphrasing is achieved basedon the variety of contexts extracted from a large cor-pus. According to Harris’s Distributional Hypoth-esis (Harris, 1954), first, the nouns used in a con-text similar to the input sentence are extracted fromthe corpus. Then, the context similarity for each ex-tracted noun and the noun in the input sentence isFigure 1: Noun paraphrasing in the proposed method.calculated utilizing the case-frame dictionary. Anabstract of the proposed method is illustrated in Fig-ure 1.3.1 Extraction of Paraphrase CandidatesIn this method, we hypothetically define the pre-phrase and post-phrase of the target noun as the con-text; nouns used in a similar context are extractedfrom the corpus.First, the input sentence is divided into two dif-ferent contexts: pre-context and post-context. Then,the input sentence is searched through each corpus.The common nouns found at the end (tail) of the pre-context and at the start (head) of the post-context areextracted.For example, when the phrase look for the ac-cess to the airport is given as an input sentence andthe word access is the paraphrase target word, thepre-context is look for X and the post-context is Xto the airport. Both contexts are searched throughthe corpus for any phrases that have the exact samephrases next to the X for any other nouns, and the re-placeable nouns for X are extracted. In the exampleshown in Figure 1, the pre-context and post-contexthave the words transfer, fee, and way in common.3.2 Selection of Paraphrase CandidatesThis paper forms two hypotheses and defines Equa-tion (1) to obtain high values for similar contextnouns to paraphrase a given target word.PACLIC 28!646sim(nt, nc) = com(nt, nc) ∗ log( Nvar(nc)) (1)cooccurrence(wi, wj) =∑sn∈Sfreqn(wi, wj) (2)pmi(wi, wj) = log(cooccurrence(wi, wj)∑sn∈S∑wm∈sn freqn(wm)∑sn∈S freqn(wi)∑sn∈S freqn(wj)) (3)cos(u⃗, v⃗) =u⃗ · v⃗|u⃗||v⃗| (4)1. When the paraphrased target word and theparaphrase candidate have the maximum pos-sible number of common contexts, the para-phrasability increases.2. When the paraphrase candidates have sev-eral different contexts, the paraphrasability de-creases.In the Equation (1), nt is the paraphrase tar-get noun, nc is the paraphrase candidate noun,com(nt, nc) is the number of types of common con-texts, N is the sum of the number of contexts, andvar(nc) is the unique number of contexts in whichnc is used. For the first term, if the number ofdifferent common contexts is large, the value alsobecomes larger. For the latter term, the fewer thenumber of contexts for the paraphrase candidate, thelarger its value becomes. Hence, a high sim(nt, nc)indicates that two contexts are similar.According to the distribution hypothesis, the wordof the similar meaning is used in the similar con-text. The first term of the Equation (1) expressesthat context is similar so that there is much commoncontext. However, the word used in many contexts,such as boss and start, cannot be said to be that thecontext resembles the paraphrase target noun even ifcom(nt, nc) are large. Therefore we filter it in thelatter term of the Equation (1) and lower score of theparaphrase candidate noun used in much context.4 Experiment4.1 Experimental ObjectTo test our proposed method, we conducted an ex-periment using theWeb Japanese N-gram (Kudo andKazawa, 2007). The Web Japanese N-gram includesthe word N (1 to 7)-grams parsed by the Japaneselanguage morphological analyzer MeCab (Kudo etal., 2004). Each N-gram appears more than 20 timesin 20 billion sentences in Web text. We consideredthat the longest 7-gram data is a sentence and usedall 570,204,252 sentences. In addition, we selected1,365,705 sentences where the head was a noun andthe tail was the original form of a verb. In the ex-periment we used most-frequent 200 sentences as atarget. Also, nouns at the beginning of sentences areexcluded. In addition, we used MeCab to determinethe parts of speech.4.2 Experimental ProcedureWe calculated distributional similarity using the Ky-oto University case frame (KCF) (Kawahara andKurohashi, 2009) data on the extracted nouns. KCFis the predicate and noun pair that has a case re-lationship, and it is built automatically (Kawaharaand Kurohashi, 2005) from 1.6 billion Web texts. Inthe experiment, we used all 34,059 predicates and824,639 nouns. In addition, we assumed that thesepredicates are contexts and calculated their distribu-tional similarity using Equation (1).4.3 EvaluationTo evaluate the proposed method, we compared itwith related paraphrasing methods based on distri-butional similarity. We selected nouns included inthe top 10 similarities from 200 input sentences; inaddition, we extracted the paraphrasing target as de-scribed in Section 4.1 using our proposed method,the method by Marton et al. (2009), and the methodby Bhagat and Ravichandran (2008). Three evalu-ators selected one noun each to paraphrase with aparaphrasing target in an input sentence.PACLIC 28!647Figure 2: Number of paraphrasable nouns to first place ofsimilarity.Marton et al. (2009) produce a feature vector byco-occurrence frequency with a noun and the con-text, and they calculate vector similarity by cosine.On the other hand, Bhagat and Ravichandran (2008)produce a feature vector by PMI with a noun and thecontext and calculates vector similarity by cosine.Both methods define nouns and verbs in dependencyrelationships to the context and produce feature vec-tors using Web Japanese N-gram. We define the co-occurrence frequency in Equation (2), PMI in Equa-tion (3), and cosine similarity in Equation (4).In the equations, sn ∈ S, wm ∈ sn, wm ∈ W ,S is the set of sentences, W is the set of words,freqn(wm) is the appearance frequency of wordwmin sentence n, freqn(wi, wj) is the co-occurrencefrequency of word wi and wj in sentence n and u⃗and v⃗ are the feature vectors.5 Experiment ResultsFigure 2 and Figure 3 show the evaluation results ofthe experiment described in Section 4, with a para-phrase of 200 sentences. The Fleiss’s Kappa coeffi-cient of three evaluators is 0.61. Thus, the agreementdegree between raters is high enough.Figure 2 shows the number of nouns evaluated asthe possible paraphrase for each method.On one hand, (Marton et al., 2009) applied theidea that the frequently co-occurring context is theimportant context. On the other hand, (Bhagat andRavichandran, 2008) argued that the biasedly co-occurring context is important. Therefore, (Mar-ton et al., 2009)’s method depends solely on highfrequency words, whereas (Bhagat and Ravichan-Figure 3: Number of paraphrasable nouns to the 10thplace of similarity.Figure 4: Relationships by order of similarity and numberof paraphrasable nouns.dran, 2008)’s method relies on low frequency words.Hence, for (Marton et al., 2009)’s method, theword thing is suggested as the paraphrase candi-date for 100 combinations out of 200 combinations.For (Bhagat and Ravichandran, 2008), the counterwords, which are words that describe the number ofitems, are suggested as paraphrase candidates a sig-nificant number of times.The proposed method does not rely on the fre-quency of the context; therefore, such an effect isdisregarded as possible, and as a result, our methodobtains high scores.Figure 3 shows the number of nouns evaluatedas possible candidates for paraphrases for the top10 nouns of similarity. When observing the top10 nouns, the results of (Bhagat and Ravichandran,2008)’s method are close to the results of the pro-posed method. Figure 4 shows the rankings of simi-larities and the relationship of the number of possi-PACLIC 28!648Table 1: English translation of paraphrases generated by the proposed method.Owner’s [ recognition? permission ] is required.Proceeding the [ subject? problem ] as important matter.Generous [ fee? price ] is offered.National agriculture’s [ advance? growth ] is obstructed.Education’s [ expansion? strengthening ] are the examples.Figure 5: Relationships by order of similarity and numberof paraphrasable nouns.ble paraphrase candidates. Although Figure 4 showsthe results for Evaluator A, the tendency is the sameas for Evaluator B (Figure 5) and Evaluator C (Fig-ure 6). In the results of the proposed method, thereis a significant gap in the numbers of first-rankedand second-ranked nouns. However, in the resultsof (Bhagat and Ravichandran, 2008)’s method, thegap is insignificant. This is because the proposedmethod strictly applies the paraphrase process tonouns that are exactly in the context in which theyare used in the input sentence. Because (Bhagat andRavichandran, 2008)’s method does not consider thecontext of the input sentence, the quality is not al-ways guaranteed to obtain the possible best score.For instance, given an input sentence such asassign a maximum [penalty] of $, the paraphraseprocess for [penalty] in both (Marton et al., 2009)and (Bhagat and Ravichandran, 2008) grants impris-onment the highest score. On the other hand, theproposed method shows paying penaltywith the bestscore, followed by correctional fine; imprisonmentdoes not even appear as a candidate.For the input sentence, reduce the [burdens]on the back, in the case of paraphrasing [bur-dens], (Bhagat and Ravichandran, 2008)’s methodFigure 6: Relationships by order of similarity and numberof paraphrasable nouns.suggests cost, expenses, and actual cost, all of whichare money-related; any words listed within the top10 are not appropriate paraphrase candidates.Meanwhile, the proposed method suggests loads,stress, damage, exhaustion, tense, impact, etc., allof which are considerably appropriate for paraphras-ing. Table 1 presents a list of successful examples.6 Conclusion and Future WorkIn this paper, we showed the effectiveness of themethod of paraphrasing a noun along the context ofa given input sentence based on the variety of con-texts obtained from a large-scale corpus. Our pro-posed method can paraphrase nouns depending onthe context of the input sentence, and we can ob-tain the appropriate paraphrase independently of theappearance frequency and co-occurrence frequencyof the word. This is because we select a noun thatshares more contexts with the paraphrasing target inthe paraphrase.This paper discussed the validity of paraphrasesusing a different statistics value from frequencycalled the number of types of the context. Our goal isto simplify vocabulary by paraphrasing, and it con-siders the restriction to plain vocabularies, such asPACLIC 28!649the Basic Vocabulary to Learn, to maintain the accu-racy and comprehensibility of lexical paraphrasing.ReferencesRegina Barzilay and Kathleen R. McKeown. 2001. Ex-tracting Paraphrases from a Parallel Corpus. In Pro-ceedings of the 39th Annual Meeting of the Associa-tion for Computational Linguistics (ACL 2001), pages50–57.Rahul Bhagat and Deepak Ravichandran. 2008. LargeScale Acquisition of Paraphrases for Learning SurfacePatterns. In Proceedings of the 46th Annual Meetingof the Association for Computational Linguistics (ACL2008), pages 674–682.William Coster and David Kauchak. 2011. SimpleWikipedia: A New Simplification Task. In Proceed-ings of the 49th Annual Meeting of the Association forComputational Linguistics (ACL 2011), pages 665–669.Zellig S. Harris. 1954. Distributional Structure. Word,10(23):146–162.Kentaro Inui and Atsushi Fujita. 2004. A Survey onParaphrase Generation and Recognition. Journal ofNatural Language Processing, 11(5):151–198.Mutsuro Kai and Toshihiro Matsukawa. 2002. Methodof Vocabulary Teaching: Vocabulary Table version.Mitsumura Tosho Publishing Co., Ltd.Tomoyuki Kajiwara, Hiroshi Matsumoto and KazuhideYamamoto. 2013. Selecting Proper Lexical Para-phrase for Children. In Proceedings of the Twenty-Fifth Conference on Computational Linguistics andSpeech Processing (ROCLING 2013), pages 59–73.Daisuke Kawahara and Sadao Kurohashi. 2005. Grad-ual Fertilization of Case Frames. Journal of NaturalLanguage Processing, 12(2):109–131.Daisuke Kawahara and Sadao Kurohashi. 2009. KyotoUniversity?s Case Frame Data ver 1.0. Gengo ShigenKyokai.Taku Kudo and Hideto Kazawa. 2007. Web JapaneseN-gram Version 1. Gengo Shigen Kyokai.Taku Kudo, Kaoru Yamamoto and Yuji Matsumoto.2004. Applying Conditional Random Fields toJapanese Morphological Analysis. In Proceedings ofthe Conference on Empirical Methods in Natural Lan-guage Processing (EMNLP 2004)Yuval Marton, Chris Callison-Burch and Philip Resnik.2009. Improved Statistical Machine Translation Us-ing Monolingually-Derived Paraphrases. In Proceed-ings of the 2009 Conference on Empirical Methods inNatural Language Processing (EMNLP 2009), pages381–390.Yusuke Shinyama and Satoshi Sekine. 2003. ParaphraseAcquisition for Information Extraction. In Proceed-ings of the 2nd International Workshop on Paraphras-ing: Paraphrase Acquisition and Applications (IWP2003), pages 65–71.

Noun Paraphrasing Based on a Variety of Contexts

Kajiwara, Tomoyuki

51702

Yamamoto, Kazuhide

51703

1000040359708

Waseda University Repository

https://waseda.repo.nii.ac.jp/?action=repository_action_common_download&item_id=29510&item_no=1&attribute_id=101&file_no=1

Noun Paraphrasing Based on a Variety of Contexts

Abstract

Similar works

Full text

Available Versions

DSpace at Waseda University

Waseda University Repository