Parallel corpora have driven great progress in the field of Text Simplification. However, most sentence alignment algorithms either offer a limited range of alignment types supported, or simply ignore valuable clues present in comparable documents. We address this problem by introducing a new set of flexible vicinity-driven paragraph and sentence alignment algorithms that 1-N, N-1, N-N and long distance null alignments without the need for hard-to-replicate supervised models

Paetzold, GH

Specia, L

English

arXiv

Spiral - Imperial College Digital Repository

arXiv:1612.04113v1  [cs.CL]  13 Dec 2016Vicinity-Driven Paragraph and Sentence Alignmentfor Comparable CorporaGustavo Henrique Paetzold and Lucia SpeciaDepartment of Computer ScienceUniversity of Sheffield, UK{g.h.paetzold,l.specia}@sheffield.ac.ukAbstractParallel corpora have driven great progressin the field of Text Simplification. How-ever, most sentence alignment algorithmseither offer a limited range of alignmenttypes supported, or simply ignore valuableclues present in comparable documents.We address this problem by introducinga new set of flexible vicinity-driven para-graph and sentence alignment algorithmsthat 1-N, N-1, N-N and long distance nullalignments without the need for hard-to-replicate supervised models.1 IntroductionEarly Text Simplification (TS) approaches reliedheavily on manually-crafted simplification rules(Chandrasekar et al., 1996; Siddharthan, 2014)due to a lack of resources from which to automat-ically learn or extract them. Complex-to-simpleparallel corpora effectively changed the nature ofcontributions pertaining to TS upon their arrival.The most widely used resource of this kindis the Wikipedia and Simple Wikipedia parallelcorpus (Kauchak and Barzilay, 2006). Using thiscorpus, authors have devised Syntactic Simplifi-cation approaches that learn simplification rulesfrom aligned sentences (Siddharthan, 2006;Woodsend and Lapata, 2011), MachineTranslation approaches that translate fromcomplex to simple text (Zhu et al., 2010;Bach et al., 2011), Lexical Simplifiers thatextract complex-to-simple word correspon-dences from word alignments (Biran et al., 2011;Horn et al., 2014), and even hybrid lexico-syntactic simplifiers (Paetzold and Specia, 2013;Feblowitz and Kauchak, 2013). Other ex-amples of parallel corpora that have beenused in TS are the medical-domain corpus of(Deléger and Zweigenbaum, 2009), and the newlyintroduced Newsela corpus (Xu et al., 2015).Most of these corpora come, however, as a setof complex-to-simple articles aligned at documentlevel. In order for them to be used by MachineTranslation systems, Lexical Simplification strate-gies and Tree Transduction frameworks, for exam-ple, these documents need to be aligned at sen-tence level.Various strategies have been created to ad-dress this problem. Barzilay and Elhadad (2003)present an approach that first aligns para-graphs with sophisticated clusterers and classifierslearned from manually annotated instances, thenproduces sentence alignments through a dynamicprogramming algorithm that finds an alignmentpath between the sentences in aligned paragraphs.Coster and Kauchak (2011) use the same sentencealignment approach, but opt instead for a muchsimpler paragraph alignment algorithm: they sim-ply align any pair of paragraphs of which the TF-IDF similarity is higher than a manually set thresh-old of 0.5. Similarly, Bott and Saggion (2011)also make use of an elaborate, difficult to repli-cate supervised approach in order to learn thelikelihood of two portions of text being aligned.Smith et al. (2010) bypass the step of paragraphalignment and use instead a supervised Condi-tional Random Fields ranker for sentence align-ment directly.Although these algorithms have been provenuseful, they do not exploit the comparability be-tween the aligned documents’ paragraphs and sen-tences, hence the need for elaborate supervisedclassifiers and rankers. The sentence alignerused by both Barzilay and Elhadad (2003) andCoster and Kauchak (2011) suffers from yet an-other limitation: it does not allow for multipleconsecutive sentence skips, which can lead to in-correct alignments in scenarios where the numberof sentences in a pair of aligned paragraphs is notsimilar.In this paper, we introduce a new set of para-graph and sentence alignment algorithms designedto address these limitations.2 Clues in Comparable CorporaBy inspecting a handful of documents from theWikipedia-Simple Wikipedia and the Newselacorpus, we noticed that they are divided in numer-ous paragraphs, and that the order with which theinformation is presented is consistent throughoutdifferent versions of the same article. This meansthat, most of the time, one can safely assume that,if the ith andjth paragraph of a pair of documentsis aligned, then the next alignment(x, y) will besuch thatx≥ i andy≥j.Since paragraphs offer valuable alignmentclues, we believe that algorithms that by-pass paragraph alignment, such as that ofSmith et al. (2010), are not suitable for this task.We also noticed that the order of the infor-mation within aligned paragraphs is also con-sistent, but that often times aligned paragraphswill have a noticeably large disparity in num-ber of sentences they have. Consequently, sen-tence alignment algorithms that do not allowfor multiple consecutive skips, such as that ofBarzilay and Elhadad (2003), may not be suitablefor this task either.Xu et al. (2015) produced sentence alignmentsfor the Newsela corpus by simply calculating theJaccard similarity score of each pair of sentencesin a set of documents, then selecting any pairs thatachieve a similarity score above a manually setthreshold. This approach, however, suffers fromseveral limitations also, since it does not exploitthe aforementioned observations in any capacity,nor does it allow for N-1 or N-N alignments.We address this problem by introducing new,flexible paragraph and sentence alignment algo-rithms that better document-aligned corpora. Wedescribe them in what follows.3 Paragraph Alignment AlgorithmIn order to produce paragraph alignments, we em-ploy a search algorithm that allows 1-1, 1-N, N-1, N-N and null alignments (i.e. unaligned para-graphs between alignments).Algorithm 1 exploits a matrixM ∈R‖D1‖×‖D2‖where‖D1‖ represents the number of paragraphsin a documentD1, ‖D2‖ the number of para-graphs in a documentD2, andM(i, j) the sim-ilarity between theith paragraph inD1 and thejth paragraph inD2. As a similarity metric, weuse the maximum TF-IDF cosine similarity be-tween all possible pairs of sentences in[i, j] i.e.the similarity between two paragraphs is equal tothat of the most similar pair of sentences in them.We choose this metric due to the fact that, eventhough the sentences in equivalent paragraphs ofdocuments with different reading levels are usu-ally very distinct in both form and vocabulary, of-ten times paragraphs will share at least one verysimilar (or even identical) sentence between them,which is a strong indicator of a good paragraphalignment. These similar sentences are often sobecause of long quotes from subjects interviewedfor the news articles.The goal of Algorithm 1 is to find a pathA ∈Z‖D1‖×‖D2‖ whereA(i, j)=1 if there is an align-ment between paragraphs[i, j] ∈ D1×D2, andA(i, j) = 0 otherwise. The search for pathAstarts from the assumption thatA(1, 1) = 1, orin other words, the first paragraph ofD1 andD2are aligned. We exploit this assumption becausethe first paragraph in most document-aligned cor-pora available refer to the articles’ titles. The al-gorithm then initializes a control set of coordi-nates[cx, cy] = [1, 1] that represents the pointfrom which to search for the next alignment inA. The next alignment is then searched for in thefirst vicinity V1, which represents 1-1, 1-N andN-1 alignments. If there is not a paragraph pair[nx, ny] in V1 such thatM(nx, ny) ≥ α then thenext alignment is searched for in the second vicin-ity V2, which represents single paragraph skips.If no pair [nx, ny] in V2 hasM(nx, ny) ≥ α,then the algorithm searches for the the next pair[nx, ny] with M(nx, ny)≥α with the shortest eu-clidean distance to[cx, cy] in the third and finalvicinity V3, which represents long-distance para-graph skips. Finally, the update[cx, cy] = [nx, ny]is made, and the alignmentA(cx, cy)=1 is added.This process is repeated until there are no moreparagraphs left to be aligned, or until the algorithmfinds no suitable[nx, ny] to follow current align-ment[cx, cy] in vicinity V3. Notice that, althoughwe use only three vicinities, the algorithm can beeasily adapted to support as many distinct vicini-ties as suitable. AfterA is found, the paragraphs inall 1-N, N-1 and N-N alignments are concatenatedso that our sentence alignment algorithm can moreeasily search for equivalent sentences in them.Algorithm 1: Paragraph Alignmentinput: M, D1, D2, α;output:A;A← zeros (‖D1‖ , ‖D2‖);[cx, cy]← [1, 1];while [cx, cy ] 6= nil doA(cx, cy)← 1;V1← {[cx, cy+1] , [cx+1, cy] , [cx+1, cy+1]};V2← {[cx+2, cy+1] , [cx+1, cy+2]};V3← {[x, y] | lx ≥ x ≥ cx and ly ≥ y ≥ cy};[nx, ny]← [x, y] with bestM(x, y) in V1;if M(nx, ny) < α then[nx, ny]← [x, y] with bestM(x, y) in V2;if M(nx, ny) < α then[nx, ny ]← [x, y] with M(x, y)≥α in V3with shortest euclidean distance to[cx, cy];endend[cx, cy]← [nx, ny ];end4 Sentence Alignment AlgorithmOur sentence alignment algorithm employs thesame principles behind our paragraph alignmentalgorithm: it also allows for 1-1, 1-N, N-1 and nullalignments, and works under the assumption thatN-N alignments can be inferred from consecutive1-1 alignments.Algorithm 2 exploits a matrixM ∈R‖P1‖×‖P2‖where‖P1‖ represents the number of sentences ina paragraphP1, ‖P2‖ the number of sentences in aparagraphP2, andM(i, j) the similarity betweenthe ith sentence inP1 and thejth sentence inP2.As a metric, we use TF-IDF cosine similarity.In search of pathA ∈ Z‖P1‖×‖P2‖, our algo-rithm first finds the initial alignment point[cx, cy ]with M(cx, cy) ≥ α with the shortest euclideandistance to[0, 0]. Notice that this algorithm can-not exploit the assumption that the first sentencesin a pair of paragraphs will be aligned. The nextalignment[nx, ny] is then searched for in the im-mediate vicinity V1. IfM(nx, ny) is smaller thana minimumα, then it searches for the the nextpair [nx, ny] with M(nx, ny)≥ α with the short-est euclidean distance to[cx, cy] in vicinity V2.Otherwise, if[nx, ny] = [cx+1, cy+1], then thenew 1-1 alignment is added toA. However, if a1-N ([nx, ny] = [cx, cy+1]) or N-1 ([nx, ny] =[cx+1, cy]) alignment is found, then the algorithmperforms a secondary loop in order to find the sizeAlgorithm 2: Sentence Alignmentinput: M, P1, P2, α, β;output:A;A← zeros (‖P1‖ , ‖P2‖);[cx, cy ]← [x, y] with M(x, y)≥αwith shortest euclidean distance to[0, 0];if [cx, cy ] 6= nil thenA(cx, cy)← 1;endwhile [cx, cy ] 6= nil doV1← {[cx, cy+1] , [cx+1, cy ] , [cx+1, cy+1]};V2← {[x, y] | lx ≥ x ≥ cx and ly ≥ y ≥ cy};[nx, ny ]← [x, y] with bestM(x, y) in V1;case M(nx, ny) < α do[nx, ny ]← [x, y] with M(x, y)≥α in V2with shortest euclidean distance to[cx, cy ];[cx, cy ]← [nx, ny ];A(cx, cy)← 1;case [nx, ny ] = [cx+1, cy+1] do[cx, cy ]← [nx, ny ];A(cx, cy)← 1;case [nx, ny ] = [cx, cy+1] doA(nx, ny)← 1;N ← 1;whileM(nx, ny :ny+N)>M(nx, ny :ny+N−1)−βandM(nx, ny :ny+N)>M(nx+1, ny :ny+N)doA(nx, ny+N)← 1;N ← N+1;end[cx, cy ]← [nx, ny+N−1];case [nx, ny ] = [cx+1, cy] doA(nx, ny)← 1;N ← 1;whileM(nx :nx+N,ny)>M(nx :nx+N−1, ny)−βandM(nx :nx+N, ny)>M(nx :nx+N, ny+1)doA(nx+N,ny)← 1;N ← N+1;end[cx, cy ]← [nx+N−1, ny ];endendof N . N is incremented until the accumulatedsimilarity for the concatenated sentences given thecurrent size ofN is smaller than the similarity forN−1 minus a slack valueβ, or until the similarityof the adjacent diagonal (M(nx+1, ny :ny+N)or M(nx :nx+N,ny+1)) is larger than the accu-mulated similarity. This process is repeated untilthere are no more paragraphs left to be aligned, oruntil the algorithm finds no suitable new alignmentcandidates.5 Final RemarksWe presented a new set of vicinity-driven para-graph and sentence alignment algorithms fordocument-aligned corpora. Our algorithms ad-dresses the limitations of previous approaches byexploiting clues in comparable documents that areoften neglected. Unlike many early strategies, ouralgorithms allow for 1-N, N-1, N-N and long-distance null alignments.In the future, we aim to conduct both intrinsicand extrinsic performance comparisons betweenour algorithms and earlier approaches. We alsoaim to employ our algorithms in the creation ofnew paragraph and sentence-aligned corpora to bemade available to the public.References[Bach et al.2011] Nguyen Bach, Qin Gao, Stephan Vo-gel, and Alex Waibel. 2011. Tris: A statistical sen-tence simplifier with log-linear models and margin-based discriminative training. InJCNLP.[Barzilay and Elhadad2003] Regina Barzilay andNoemie Elhadad. 2003. Sentence alignment formonolingual comparable corpora. InProceedingsof the 2003 EMNLP, pages 25–32.[Biran et al.2011] Or Biran, Samuel Brody, and NoémieElhadad. 2011. Putting it simply: a context-awareapproach to lexical simplification. InProceedings ofthe 49th ACL, pages 496–501.[Bott and Saggion2011] Stefan Bott and Horacio Sag-gion. 2011. An unsupervised alignment algorithmfor text simplification corpus construction. InPro-ceedings of the 2011 MTTG, pages 20–26.[Chandrasekar et al.1996] Raman Chandrasekar, Chris-tine Doran, and Bangalore Srinivas. 1996. Moti-vations and methods for text simplification. InPro-ceedings of the 16th COLING, pages 1041–1044.[Coster and Kauchak2011] William Coster and DavidKauchak. 2011. Simple english wikipedia: A newtext simplification task. InProceedings of the 49thACL, pages 665–669.[Deléger and Zweigenbaum2009] Louise Deléger andPierre Zweigenbaum. 2009. Extracting lay para-phrases of specialized expressions from monolin-gual comparable medical corpora. InProceedingsof the 2nd Workshop on Building and Using Compa-rable Corpora: from Parallel to Non-parallel Cor-pora, pages 2–10.[Feblowitz and Kauchak2013] Dan Feblowitz andDavid Kauchak. 2013. Sentence simplification astree transduction. InProceedings of the 2nd Work-shop on Predicting and Improving Text Readabilityfor Target Reader Populations, pages 1–10.[Horn et al.2014] Colby Horn, Cathryn Manduca, andDavid Kauchak. 2014. Learning a lexical simplifierusing wikipedia. InProceedings of the 52nd ACL,pages 458–463.[Kauchak and Barzilay2006] David Kauchak andRegina Barzilay. 2006. Paraphrasing for automaticevaluation. InProceedings of the 2006 NAACL,pages 455–462.[Paetzold and Specia2013] Gustavo H. Paetzold andLucia Specia. 2013. Text simplification as treetransduction. InProceedings of the 9th STIL, pages116–125.[Siddharthan2006] Advaith Siddharthan. 2006. Syn-tactic simplification and text cohesion.Research onLanguage and Computation, 4(1):77–109, March.[Siddharthan2014] Advaith Siddharthan. 2014. A sur-vey of research on text simplification.InternationalJournal of Applied Linguistics, pages 259–298.[Smith et al.2010] Jason R Smith, Chris Quirk, andKristina Toutanova. 2010. Extracting parallel sen-tences from comparable corpora using documentlevel alignment. InProceedings of the 2010 HLT,pages 403–411.[Woodsend and Lapata2011] Kristian Woodsend andMirella Lapata. 2011. Learning to simplify sen-tences with quasi-synchronous grammar and integerprogramming. InProceedings of the 2011 EMNLP,pages 409–420.[Xu et al.2015] Wei Xu, Chris Callison-Burch, andCourtney Napoles. 2015. Problems in current textsimplification research: New data can help.Trans-actions of the Association for Computational Lin-guistics, 3:283–297.[Zhu et al.2010] Zhemin Zhu, Delphine Bernhard, andIryna Gurevych. 2010. A monolingual tree-basedtranslation model for sentence simplification.Com-putational Linguistics, (August):1353–1361.

Vicinity-driven paragraph and sentence alignment for comparable corpora

http://spiral.imperial.ac.uk/bitstream/10044/1/63805/2/1612.04113v1.pdf

Vicinity-driven paragraph and sentence alignment for comparable corpora

Abstract

Similar works

Full text

Available Versions

Spiral - Imperial College Digital Repository