Evaluating Artificial Players for the Language Game “La Ghigliottina” (Ghigliottin-AI) task is one of the tasks organized in the context of the 2020 EVALITA edition, a periodic evaluation campaign of Natural Language Processing (NLP) and speech tools for the Italian language. Ghigliottin-AI participants are asked to build an artificial player able to solve “La Ghigliottina”, namely the final game of an Italian TV show called “L'Eredità”. The game involves a single player who is given a set of five words unrelated to each other, but related with a sixth word that represents the solution to the game. Fourteen teams registered to Ghigliottin-AI. Nevertheless, only two teams submitted their run. In order to evaluate the submitted systems, we rely on an API base methodology, via a Remote Evaluation Server (RES). In this report we describe the Ghigliottin-AI task, the data, the evaluation and we discuss results

Basile P.

Lovetere M.

Monti J.

Pascucci A.

Sangati F.

Siciliani L.

English

Evaluating Artificial Players for the Language Game “La Ghigliottina” (Ghigliottin-AI) task is one of the tasks organized in the context of the 2020 EVALITA edition, a periodic evaluation campaign of Natural Language Processing (NLP) and speech tools for the Italian language. Ghigliottin-AI participants are asked to build an artificial player able to solve “La Ghigliottina”, namely the final game of an Italian TV show called “L'Eredità”. The game involves a single player who is given a set of five words unrelated to each other, but related with a sixth word that represents the solution to the game. Fourteen teams registered to Ghigliottin-AI. Nevertheless, only two teams submitted their run. In order to evaluate the submitted systems, we rely on an API base methodology, via a Remote Evaluation Server (RES). In this report we describe the Ghigliottin-AI task, the data, the evaluation and we discuss results

Archivio istituzionale della ricerca - Università di Bari

Ghigliottin-AI @ EVALITA2020: Evaluating Artificial Playersfor the Language Game “La Ghigliottina”Pierpaolo BasileDept. of Computer ScienceUniversity of Bari, Italypierpaolo.basile@uniba.itMarco LovetereGhigliottiniamomarlove@gmail.comJohanna Monti and Antonio PascucciUNIOR NLP Research Group“L’Orientale” University of Naples, Italy{jmonti, apascucci}@unior.itFederico SangatiUNIOR NLP Research Group“L’Orientale” University of Naples, ItalyOIST Graduate University, Japanfederico.sangati@gmail.comLucia SicilianiDept. of Computer ScienceUniversity of Bari, Italylucia.siciliani@uniba.itAbstractEnglish. Evaluating Artificial Playersfor the Language Game “La Ghigliot-tina” (Ghigliottin-AI) task is one of thetasks organized in the context of the 2020EVALITA edition, a periodic evaluationcampaign of Natural Language Process-ing (NLP) and speech tools for the Ital-ian language. Ghigliottin-AI participantsare asked to build an artificial player ableto solve “La Ghigliottina”, namely the fi-nal game of an Italian TV show called“L’Eredità”. The game involves a singleplayer who is given a set of five wordsunrelated to each other, but related witha sixth word that represents the solutionto the game. Fourteen teams registeredto Ghigliottin-AI. Nevertheless, only twoteams submitted their run. In order to eval-uate the submitted systems, we rely onan API base methodology, via a RemoteEvaluation Server (RES). In this report wedescribe the Ghigliottin-AI task, the data,the evaluation and we discuss results.Copyright ©2020 for this paper by its authors. Use per-mitted under Creative Commons License Attribution 4.0 In-ternational (CC BY 4.0).1 Background and MotivationLanguage games draw their challenge and excite-ment from the richness and ambiguity of naturallanguage, and therefore have attracted the atten-tion of researchers in the fields of Artificial In-telligence and Natural Language Processing. Forinstance, IBM Watson is a system which suc-cessfully challenged human champions of “Jeop-ardy!”, a game in which contestants are presentedwith clues in the form of answers, and must phrasetheir responses in the form of a question (Ferrucciet al., 2010; Molino et al., 2015). Another popularlanguage game is solving crossword puzzles. Thefirst experience reported in the literature is Proverb(Littman et al., 2002), that exploits large librariesof clues and solutions to past crossword puzzles.WebCrow is the first solver for Italian crosswords(Ernandes et al., 2008).Following the first edition of the NLP4FUNtask (Basile et al., 2018), proposed at EVALITA2018, we propose a new edition of the task whoseaim is to design a solver for “The Guillotine”(La Ghigliottina, in Italian) game. It is inspiredby the final game of an Italian TV show called“L’Eredità”. The game, broadcast by Italian na-tional TV, involves a single player, who is given aset of five words - the clues - each linked in someway to a specific word that represents the uniquesolution of the game. Words are unrelated to eachother, but each of them has a hidden associationwith the solution. Once the clues are given, theplayer has one minute to find the solution. For ex-ample, given the five clues: pie, bad, Adam, core,eye the solution is apple, because: apple-pie is akind of pie; bad apple is a way of referring to atrouble maker; Adam’s apple is the prominent partof men’s throat; apple core is the centre of the ap-ple; apple of someone’s eye is way of referringto someone’s beloved person. This report is or-ganized as follows: in Section 2 we describe theGhigliottin-AI task. In Section 3 we present thedataset. The task evaluation is in Section 4. Re-sults achieved by participants are shown in Section5. Conclusions are in Section 6.2 Task DescriptionEvaluating Artificial Players for the LanguageGame “La Ghigliottina” (Ghigliottin-AI) is oneof the fourteen EVALITA 2020 tasks (Basile etal., 2020). Ghigliottin-AI participants are askedto build an artificial player able to solve “LaGhigliottina”. They can take advantage of solu-tions adopted by previous systems (Semeraro etal., 2009; Basile et al., 2016; Sangati et al., 2018)and the availability of open repositories on theweb.3 DatasetWe provided a set of 300 games with their solu-tion taken from the last editions of the TV gameas training data. The training data was released inJSON format as shown in Figure 1. In this exam-ple, the first JSON shows the clues “posto” (lit-erally place), “artificiale”(artificial), “lavaggio”(washing), “allenare” (literally to train) and “gal-lina” (chicken) and the solution “cervello” (brain):non avere il cervello a posto (to be nutty), cervelloartificiale (artificial brain), lavaggio del cervello(brainwashing), allenare il cervello (stretch thebrain) and cervello da gallina (hare-brained). Inthe second JSON we find “essere” (to be), “com-parsa” (appearance), “x men”, “ronaldo” and“mondiale” (global) and the solution “fenomeno”(phenomenon): essere un fenomeno (be a phe-nomenon), comparsa di un fenomeno (apperanceof a phenomenon), Fenomeno is one of the X-men,Fenomeno was Ronaldo’s nickname and fenomenomondiale (worldwide phenomenon).The test set consists in 350 games instances,provided by a Remote Evaluation Server (RES)[{"w1": "posto","w2": "artificiale","w3": "lavaggio","w4": "allenare","w5": "gallina","solution": "cervello"},{"w1": "essere","w2": "comparsa","w3": "x men","w4": "ronaldo","w5": "mondiale","solution": "fenomeno"},...]Figure 1: JSON format of the training set.Ghigliottiniamo1 at random intervals of time asa request with a single game challenge to regis-tered systems. The RES allowed the systems toreply with a single solution to the game. Ghigliot-tiniamo2 currently enables both humans and artifi-cial systems to submit solutions to the TV game inreal-time.4 Task evaluationIn order to evaluate the AI systems, we rely onan API based methodology. During the evalua-tion period, at random intervals of time (over aperiod of 7 days), the RES submitted 350 gamechallenges to the registered systems. The systemshad to reply back to the RES with a single solutionto the game.As evaluation measure, we adopt the standardaccuracy score:solved gamestotal games(1)As in the TV game, where players have oneminute to provide the solution, the RES will dis-card system solutions received after 60 secondsfrom the submitted challenge.1https://quiztime.net2https://play.google.com/store/apps/details?id=io.quiztime.game5 ResultsFourteen teams registered to the Ghigliottin-AItask. However, only two teams participated to thefinal test: GUL.LE.VER (De Francesco, 2020) andIl Mago della Ghigliottina (Sangati et al., 2020).GUiLlotine gLovE resolVER (GUL.LE.VER) isbased on the Glove (Pennington et al., 2014)vector representation of the words on the basisof a large collected dataset, containing the Ital-ian Wiktionary, Wikiquote, Wikipedia (only ti-tles), the Italian Collocations Dictionary and otherresources scraped on the web containing Italianmultiword expressions, proverbs and songs titles.The Glove algorithm was chosen for its intrinsicpower in capturing the co-occurrence correlationbetween two words that are not synonyms, due tothe co-occurrence matrix that the algorithm buildsbefore the training. The solution is searched in thevector space near the clues, obtaining a list of solu-tion candidates. This list is descending reorderedusing a hybrid function composed by two parts:one part is based on the Pointwise Mutual Infor-mation; the other one is based on the weightedsum of the cosine similarity between the candi-date solutions and the clues, in which the weightis the normalized IDF of the single clue in the cor-pus (solutions that are correlated with the rarestclues are more important than others). Il Magodella Ghigliottina is the same system submittedwith the name of UNIOR4NLP in the NLP4FUNtask in 2018 without any changes. The systemis based on the observation that most cases cluesand solution are connected because they form amultiword expression. In addition, clues are al-most always nouns, verbs or adjectives, while so-lutions are nouns or adjectives. The system isbased on a number of freely available corpora,such as: Paisà3; itWaC4; Wiki-IT-Titles down-loaded via WikiExtractor5; 1955 proverbs fromWikiquote6 and 371 from an online collection7downloaded on the 24th April 2018. Further lexi-cal resources were developed from “Il Nuovo vo-cabolario di base della lingua italiana” and from3https://www.corpusitaliano.it/4https://wacky.sslmit.unibo.it/doku.php?id=corpora\#italian5http://attardi.github.io/wikiextractor.6https://it.wikiquote.org/wiki/Proverbi_italiani7http://web.tiscali.it/proverbiitalianithe “De Mauro online dictionary”. Technical de-tails about Il Mago della Ghigliottina are avail-able in (Sangati et al., 2018), submitted for theNLP4FUN task.Table 1 shows the results of the two systems.System Correct Total Acc.GUL.LE.VER 94 350 0.269Il Mago dellaGhigliottina240 350 0.686Combined (upperbound)257 350 0.734Table 1: ResultsBoth systems were able to provide a solutionto all 350 games within a minute. The recordedtime of the two systems ranges between 0.316 and9.988 seconds. It is important to keep in mind thatin addition to the response time, the recorded timeincludes the latency of the network and the timerequired for the instance to wake-up if it is set togo to sleep when idle. Il Mago della Ghigliot-tina is the system with the highest accuracy (aboutthree solutions out of four correct), followed byGUL.LE.VER which on average is able to solveone game out of four.We have computed the upper bound of the accu-racy of the two systems on the test set when usedin combination. The resulting accuracy is 73.4%,about 5 percentage points above the best perform-ing system. This means that the two systems havesome complementary and could be used in combi-nation with some aggregating strategy.6 ConclusionsIn this report we presented Ghigliottin-AI, one ofthe EVALITA 2020 task. Despite fourteen teamssubscribed to the task, just two of them submittedtheir system, namely GUL.LE.VER and Il magodella Ghigliottina. This latter achieved the bestperformances in terms of accuracy (68.6%), whileGUL.LE.VER obtained 26.9% of accuracy.Systems have been evaluated through an APImethodology conducted by the Remote EvaluationServer (RES) (Ghigliottiniamo). To our knowl-edge, this is the first time that an API based sys-tem has been used on a NLP evaluation task.We believe this methodology has a strong advan-tage compared to a manual evaluation, as systemscan be tested more systematically, fairly and con-tinuously in time. We strongly hope that moretasks will adopt this evaluation strategy in the fu-ture. The Ghigliottiniamo system currently en-ables both humans and artificial systems to submitsolutions to the Ghigliottina when a new game isbroadcasted on TV. This will allow us in the futureto compare their results more systematically. Thesystem remains open for new artificial systems tojoin the live competition8.ReferencesPierpaolo Basile, Marco de Gemmis, Pasquale Lops,and Giovanni Semeraro. 2016. Solving a complexlanguage game by using knowledge-based word as-sociations discovery. IEEE Transactions on Compu-tational Intelligence and AI in Games, 8(1):13–26.Pierpaolo Basile, Marco de Gemmis, Lucia Siciliani,and Giovanni Semeraro. 2018. Overview of theevalita 2018 solving language games (nlp4fun) task.In Tommaso Caselli, Nicole Novielli, Viviana Patti,and Paolo Rosso, editors, Proceedings of the 6thevaluation campaign of Natural Language Process-ing and Speech tools for Italian (EVALITA’18),Turin, Italy. CEUR.org.Valerio Basile, Danilo Croce, Maria Di Maro, and Lu-cia C. Passaro. 2020. Evalita 2020: Overviewof the 7th evaluation campaign of natural languageprocessing and speech tools for italian. In ValerioBasile, Danilo Croce, Maria Di Maro, and Lucia C.Passaro, editors, Proceedings of Seventh Evalua-tion Campaign of Natural Language Processing andSpeech Tools for Italian. Final Workshop (EVALITA2020), Online. CEUR.org.Nazareno De Francesco. 2020. Gul.le.ver, a glovebased artificial player to solve the language game“la ghigliottina”. In Proceedings of Seventh Evalua-tion Campaign of Natural Language Processing andSpeech Tools for Italian. Final Workshop (EVALITA2020).Marco Ernandes, Giovanni Angelini, and Marco Gori.2008. A web-based agent challenges human expertson crosswords. AI Magazine, 29(1):77.David Ferrucci, Eric Brown, Jennifer Chu-Carroll,James Fan, David Gondek, Aditya A Kalyanpur,Adam Lally, J William Murdock, Eric Nyberg, JohnPrager, et al. 2010. Building watson: An overviewof the deepqa project. AI magazine, 31(3):59–79.Michael L Littman, Greg A Keim, and Noam Shazeer.2002. A probabilistic approach to solving crosswordpuzzles. Artificial Intelligence, 134(1-2):23–55.Piero Molino, Pasquale Lops, Giovanni Semeraro,Marco de Gemmis, and Pierpaolo Basile. 2015.Playing with knowledge: A virtual player for “who8https://quiztime.netwants to be a millionaire?” that leverages ques-tion answering techniques. Artificial Intelligence,222:157–181.Jeffrey Pennington, Richard Socher, and Christopher DManning. 2014. Glove: Global vectors for wordrepresentation. In Proceedings of the 2014 confer-ence on empirical methods in natural language pro-cessing (EMNLP), pages 1532–1543.Federico Sangati, Antonio Pascucci, and JohannaMonti. 2018. Exploiting multiword expressions tosolve “la ghigliottina”. In Sixth Evaluation Cam-paign of Natural Language Processing and SpeechTools for Italian. Final Workshop (EVALITA 2018),volume 2263, pages 258–263. Accademia Univer-sity Press.Federico Sangati, Antonio Pascucci, and Jo-hanna Monti. 2020. “il mago dellaghigliottina”@ghigliottin-ai when linguisticsmeets artificial intelligence. In Proceedings ofSeventh Evaluation Campaign of Natural LanguageProcessing and Speech Tools for Italian. FinalWorkshop (EVALITA 2020).Giovanni Semeraro, Pasquale Lops, Pierpaolo Basile,and Marco De Gemmis. 2009. On the tip of mythought: Playing the guillotine game. In Proceed-ings of the 21st International Jont Conference on Ar-tifical Intelligence, IJCAI’09, pages 1543–1548, SanFrancisco, CA, USA. Morgan Kaufmann PublishersInc.

Ghigliottin-AI @ EVALITA2020: Evaluating artificial players for the language game “La Ghigliottina”

https://ricerca.uniba.it/bitstream/11586/348740/1/paper155.pdf

Ghigliottin-AI @ EVALITA2020: Evaluating artificial players for the language game “La Ghigliottina”

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università di Bari