Cafagana, Michele

De Mattei, Lorenzo

Dell’Orletta, Felice

Gatt, Albert

Nissim, Malvina

English

University of Groningen

   University of GroningenCHANGE-IT@ EVALITA 2020: Change Headlines, Adapt News, GEnerateDe Mattei, Lorenzo; Cafagana, Michele; Dell’Orletta, Felice; Nissim, Malvina; Gatt, AlbertPublished in:Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.Final Workshop (EVALITA 2020), Online. CEUR. orgIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2020Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):De Mattei, L., Cafagana, M., Dell’Orletta, F., Nissim, M., & Gatt, A. (2020). CHANGE-IT@ EVALITA 2020:Change Headlines, Adapt News, GEnerate. In V. Basile, D. Croce, M. Di Maro, & L. C. Passaro (Eds.),Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools forItalian. Final Workshop (EVALITA 2020), Online. CEUR. org (Vol. 2765). (CEUR Workshop Proceedings).European Language Resources Association (ELRA).CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 19-11-2022CHANGE-IT @ EVALITA 2020:Change Headlines, Adapt News, GEnerateLorenzo De MatteiUniversity of PisaCLCG, University of GroningenItaliaNLP Lab, ILC-CNRPisa, Italylorenzo.demattei@di.unipi.itMichele CafagnaAptus.AI, Pisa, ItalyUniversity of Malta, Maltamichele@aptus.aiFelice Dell’OrlettaItaliaNLP Lab, ILC-CNRPisa, Italyfelice.dellorletta@ilc.cnr.itMalvina NissimCLCG, University of GroningenThe Netherlandsm.nissim@rug.nlAlbert GattUniversity of MaltaMaltaalbert.gatt@um.edu.mtAbstractWe propose a generation task for Italian –more specifically, a style transfer task forheadlines of Italian newspapers. This isthe first shared task on generation includedin the EVALITA evaluation framework.Indeed, one of the reasons to have this taskis to stimulate more research on generationwithin the Italian community. With thisaim in mind, we release to the participat-ing teams not only training data, but also abaseline sequence to sequence model thatperforms the task in order to help everyoneget started, even when not accustomed toNatural Language Generation (NLG) ap-proaches. Contextually, we explore thecomplex issue of automatic evaluation ofgenerated text, which is receiving particu-lar attention in the NLG community.1 Task and MotivationWe propose a generation task for Italian in the con-text of the EVALITA 2020 campaign (Basile et al.,2020). More specifically, we design a style trans-fer task for headlines of Italian newspapers.We believe it is the first time that a sharedtask on generation is offered in the context ofEVALITA. Indeed, one of the reasons to havethis task is to stimulate more research on gener-ation within the Italian community. With this goalin mind, we release to the potential participatingCopyright ©2020 for this paper by its authors. Use per-mitted under Creative Commons License Attribution 4.0 In-ternational (CC BY 4.0).teams not only training data, but also a baselinesequence to sequence model that performs the taskin order to help everyone get started, even whennot accustomed to generation models, yet. Thisbaseline model casts the style transfer problem asan extreme summarisation task, just showing howversatile the problem is in terms of possible ap-proaches. Contextually, this task will help to fur-ther explore the complex issue of evaluation ofgenerated text, which is receiving particular at-tention in the Natural Language Generation in-ternational community (Gatt and Krahmer, 2018;van der Lee et al., 2019).Task The task is cast as a “headline translation”problem, and it is as follows. Given a collection ofheadlines from two Italian newspapers at oppositeends of the political spectrum, call them G and R,change all G-headlines to headlines into style R,and all R-headlines to headlines in style G.In the context of this task we need to take care oftwo crucial aspects: data and evaluation. Detailson data are provided in Section 2, and on evalua-tion in Section 3.2 DataWe have collected news coming from two of themost important Italian newspapers situated at op-posite ends of the political spectrum, namely laRepubblica (left) and Il Giornale (right), totallingapproximately 152,000 article-headline pairs, withthe two newspapers equally represented. Althoughthe task only concerns headline change, the teamswill receive both the headlines as well as their re-spective full articles.Leveraging on an alignment procedure de-scribed below (see Cafagna et al. (2019) for fur-cosine score newspaper alignment0.96 (strict) rep Estroverso o nevrotico? Lo dice la foto scelta per il profilo socialen:[Extrovert or neurotic? The photo chosen for the social profile says so]gio L’immagine del profilo usata nei social network rivela la nostra personalitàen:[The profile picture used in social networks reveals our personality]0.5 (strict) rep Egitto, governo si dimette a sorpresaen:[Egypt, government resigns surprisingly]gio Egitto, il governo si dimetteen:[Egypt, government resigns]0.185 (loose) rep Elezioni presidenziali Francia, la Chiesa non si schiera né per Macron né per Le Penen:[Presidential elections France, the Church does not take sides either for Macron or for Le Pen]gio Il primo voto con l’incubo Isis ma il terrorismo esce sconfittoen:[The first vote with the Isis nightmare but terrorism comes out defeated]Table 1: Example of alignments between La Repubblica and Il Giornale, extracted with different simi-larity scores. The second and the third examples would fall into the strict and the loose sets, respectively,according to the thresholds used to split the alignments. The first two headline pairs are well aligned,while the third pair has a very loose alignment.ther details), we account for potential topic biasesin the two newspapers, and we split the data setinto strongly, weakly and not-aligned news. Thisinformation is useful in the creation of the datasetsthat we need to train our three evaluation classi-fiers (see Section 3). Additionally, it could help tobetter disentangle newspaper-specific style.Alignment We compute the tf-idf vectors of allthe articles of both newspapers and create subsetsof relevant news filtering by date, i.e. consider-ing only news which were published in approx-imately the same, short, temporal range for thetwo sources. On the tf-idf vectors we then com-pute cosine similarities for all news in the resultingsubset, rank them, and retain only the alignmentsthat are above a certain threshold. The thresholdis chosen taking into consideration a trade-off be-tween number of documents and quality of align-ment. We choose two different thresholds: one isstricter (≥ 0.5) and we use it to select best align-ments (strict alignments); the other one is looser(≥ 0.185, and < 0.5) — we define these latter asweak alignments. We consider the rest as basicallynot aligned.Data splits We split the dataset into stronglyaligned news, which are selected using the stricterthreshold (∼20K aligned pairs, set A∗ in Fig-ure 1a), and weakly aligned and non-aligned news(∼100K article-headline pairs equally distributedamong the two newspapers, set R in Figure 1a).The strictly aligned data is further split as shownin Figure 1a; this yields a total of four sets over thewhole dataset (A1, A2, A3, and R). A2 is left asideand used as test set for the final style transfer task.The remaining three sets are used for training theevaluation classifiers and the system for the targettask. These are shown in Figure 1b. Note that allsets also always contain the headlines’ respectivefull articles, though these are not necessarily used.Format The data is distributed in the form ofone CSV file with the following fields:id, headline, article, label [R,G]3 EvaluationHuman evaluation is generally viewed as themost desirable method to assess generated text(Novikova et al., 2018; van der Lee et al., 2019).However, human evaluation is not always a viableoption, due to resources, but also due to the factthat humans might not be capable of reliably as-sessing the task at hand. Related to the currentchallenge, De Mattei et al. (2020a) have shownthat people find it difficult to identify subtle stylis-tic differences between texts.Automatic, reliable metrics should thereforealso be sought (Novikova et al., 2017). For ourtask, we propose a fully automatic strategy basedon a series of classifiers to assess style strength andcontent preservation. For style, we train a singleclassifier (main). For content, we train two classi-fiers that perform two ‘sanity checks’: one ensuresthat the two headlines (original and transformed)are still compatible (HH classifier); the other en-sures that the headline is still compatible with theoriginal article (AH classifier). See also Figure 1b.In what follows we describe these classifiers in(a) Overall data splitsEVALUATIONtrain & testmain R+A3+A1HH A1 + random pairsAH R+A3+A1TASKtrain R+A3test A2(b) Training/test setsFigure 1: Data splits and their use in the different training setsmore detail. When discussing baseline results, wewill show how the contribution of each classifieris crucial towards a comprehensive evaluation.Main classifier The main classifier uses a pre-trained BERT (Devlin et al., 2019) encoder with alinear classifier on top fine-tuned with a batch sizeof 256 and sequences truncated at 32 tokens for 6epochs with learning rate 1e-05. Given a headline,this classifier can distinguish the two sources withan f-score of approximately 80% (see Table 2).Since style transfer is deemed successful if theoriginal style is lost in favour of the target style,we use this classifier to assess how many times astyle transfer system manages to reverse the mainclassifier’s decisions.HH classifier This classifier checks compatibil-ity between the original and the generated head-line. We use the same architecture as for the mainclassifier with a slightly different configuration:max. sequence length of 64 tokens, batch sizeof 128 for 2 epochs (early-stopped), with learn-ing rate 1e-05. Being trained on strictly aligneddata as positive instances (A1), with a correspond-ing amount of random pairs as negative instances,it should learn whether two headlines describe thesame content or not. Performance on gold data is.96 (Table 2).AH classifier This classifier performs yet an-other content-related check. It takes a headlineand its corresponding article, and tells whetherthe headline is appropriate for the article. Theclassifier is trained on article-headline pairs fromboth the strongly aligned and the weakly and non-aligned instances (R+A3+A1, Figure 1b). At testtime, the generated headline is checked for com-patibility against the source article. We use thesame base model as for the main and HH classi-fiers with batch size of 8, same learning rate and6 epochs. Performance on gold data is >.97 (Ta-ble 2).prec rec f-scoremainrep 0.77 0.83 0.80gio 0.84 0.78 0.81HHmatch 0.98 0.95 0.96no match 0.95 0.98 0.96AHmatch 0.96 0.99 0.98no match 0.99 0.96 0.97Table 2: Performance of the evaluation classifierson gold data.Overall compliancy We calculate a compliancyscore which assesses the proportion of times thefollowing three outcomes are successful (i) theHH classifier predicts ‘match’; (ii) the AH clas-sifier predicts ‘match’; (iii) the main classifier’sdecision is reversed. As upperbound, we find thecompatibility score for gold at 74.3% for transferfrom La Repubblica to Il Giornale (rep2gio), and78.1% for the opposite direction (gio2rep).4 Baseline SystemWe developed a baseline system using a summari-sation approach, where headlines are viewed asan extreme case of summarisation and generatedfrom the article. We exploit article-headline gener-ators trained on opposite sources to do the transfer,as done in (De Mattei et al., 2020b). The advan-tage of this approach is that in principle it doesn’trequire parallel data for training.Specifically, we use two pointer-generator net-works (See et al., 2017), which include a point-ing mechanism able to copy words from theIl Giornale→ La RepubblicaE in Sicilia è scattata l’allerta rossa −→ Migranti, la Protezione civile continua di-menticata[en: And in Sicily it’s now red alert] [en: Migrants, the Civil Protection Depart-ment goes on forgotten]Nozze gay, toghe contro i sindaci: ”Letrascrizioni sono illegittime”−→ Il Consiglio di Stato boccia le nozze gayall’estero[en: Gay marriages, gowns against mayors:“Transcriptions are not valid”][en: The State Council rejects gay mar-riages abroad]La Repubblica→ Il GiornaleCastelnuovo, lo sdegno di cittadini e asso-ciazioni: ”Attacco all’integrazione che fun-ziona”−→ I migranti non sono più rifugiati[en: Castelnuovo, the indignation of citizensand associations: “Attack to the integrationthat works”][en: Migrants are not refugees anymore]Da Renzi a Di Maio, ecco il redditodichiarato dai politici italiani. Fedeli il mi-nistro con l’imponibile più alto−→ Grillo e Giggino italiani conquistanol’elenco dei redditi italiani[en: From Renzi to Di Maio: here it’s theincome declared by the Italian politicians.Fedeli is the minister with the highest tax-able income][en: Grillo and Giggino Italians conquer thelist of Italian incomes]Table 3: Examples of headlines generated by the baseline system.source as well as pick them from a fixed vocab-ulary, thereby allowing better handling of out-of-vocabulary words.One model is trained on the la Repubblica por-tion of the training set, the other on Il Giornale.In a style transfer setting we use these models asfollows: Given a headline from Il Giornale, forexample, the model trained on la Repubblica canbe run over the corresponding article from Il Gior-nale to generate a headline in the style of la Re-pubblica, and vice versa.The results of the baseline system, measured asperformance of each classifier as well as the over-all compliancy score, are reported in Table 4.5 OutlookThis shared task proposal was intended to stim-ulate research in NLG, with a specific focus onHH AH Main compl.rep2gio .649 .876 .799 .449gio2rep .639 .871 .435 .240avg .644 .874 .616 .345Table 4: Baseline performance on test data.style transfer and automatic evaluation, in the Ital-ian community. Over ten teams expressed their in-terest in participating in the shared task officially,but eventually there were no submitted runs. Wedo hope that the materials developed in the con-text of this challenge will nevertheless be of useto promote research in a field that is still under-researched in the Italian NLP landscape. Allmaterials are available: https://github.com/michelecafagna26/CHANGE-IT.ReferencesValerio Basile, Danilo Croce, Maria Di Maro, and Lu-cia C. Passaro. 2020. Evalita 2020: Overviewof the 7th evaluation campaign of natural languageprocessing and speech tools for italian. In ValerioBasile, Danilo Croce, Maria Di Maro, and Lucia C.Passaro, editors, Proceedings of Seventh Evalua-tion Campaign of Natural Language Processing andSpeech Tools for Italian. Final Workshop (EVALITA2020), Online. CEUR.org.Michele Cafagna, Lorenzo De Mattei, and MalvinaNissim. 2019. Embeddings shifts as proxies fordifferent word use in italian newspapers. In Pro-ceedings of the Sixth Italian Conference on Compu-tational Linguistics (CLiC-it 2019), Bari, Italy.Lorenzo De Mattei, Michele Cafagna, FeliceDell’Orletta, and Malvina Nissim. 2020a. In-visible to People but not to Machines: Evaluationof Style-aware Headline Generation in Absenceof Reliable Human Judgment. In Proceedings ofthe Twelfth International Conference on LanguageResources and Evaluation (LREC 2020), Mar-seille, France, May. European Language ResourcesAssociation (ELRA).Lorenzo De Mattei, Michele Cafagna, FeliceDell’Orletta, and Malvina Nissim. 2020b. In-visible to People but not to Machines: Evaluationof Style-aware Headline Generation in Absenceof Reliable Human Judgment. In Proceedings ofthe Twelfth International Conference on LanguageResources and Evaluation (LREC 2020), Mar-seille, France, May. European Language ResourcesAssociation (ELRA).Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of NAACL, pages 4171–4186.Albert Gatt and Emiel Krahmer. 2018. Survey of thestate of the art in natural language generation: Coretasks, applications and evaluation. Journal of Artifi-cial Intelligence Research, 61:65–170.Jekaterina Novikova, Ondřej Dušek, Amanda Cer-cas Curry, and Verena Rieser. 2017. Why we neednew evaluation metrics for NLG. In Proceedings ofthe 2017 Conference on Empirical Methods in Natu-ral Language Processing, pages 2241–2252, Copen-hagen, Denmark, September. Association for Com-putational Linguistics.Jekaterina Novikova, Ondřej Dušek, and Verena Rieser.2018. RankME: Reliable human ratings for natu-ral language generation. In Proceedings of the 2018Conference of the North American Chapter of theAssociation for Computational Linguistics: HumanLanguage Technologies, Volume 2 (Short Papers),pages 72–78, New Orleans, Louisiana, June. Asso-ciation for Computational Linguistics.Abigail See, Peter J Liu, and Christopher D Manning.2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1073–1083.Chris van der Lee, Albert Gatt, Emiel van Miltenburg,Sander Wubben, and Emiel Krahmer. 2019. Bestpractices for the human evaluation of automaticallygenerated text. In Proceedings of the 12th Interna-tional Conference on Natural Language Generation,pages 355–368, Tokyo, Japan, October–November.Association for Computational Linguistics.

CHANGE-IT@ EVALITA 2020: Change Headlines, Adapt News, GEnerate

https://pure.rug.nl/ws/files/155886639/paper169.pdf

CHANGE-IT@ EVALITA 2020: Change Headlines, Adapt News, GEnerate

Abstract

Similar works

Full text

Available Versions

University of Groningen