This paper describes a user evaluation of the text output from the M-PIRO (Multilingual Personalised Information Objects) system, which dynamically generates descriptions of exhibits for a virtual museum. We show that subjects performed significantly better in a factual recall test when the descriptions included more sophisticated text structuring modules. The subjects also judged the structured texts to be more interesting and readable, and felt that they had learned more from them. 1

Isard, Amy

Karasimos, Athanasios

Athanasios Karasimos

Amy Isard

CiteSeerX

Multilingual evaluation of a natural language generation system

This paper describes a user evaluation of the text output from the M-PIRO (Multilingual Personalised Information Objects) system, which dynamically generates descriptions of exhibits for a virtual museum. We show that subjects performed significantly better in a factual recall test when the descriptions included more sophisticated text structuring modules. The subjects also judged the structured texts to be more interesting and readable, and felt that they had learned more from them

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Multi-lingual Evaluation of a Natural Language GenerationSystemCitation for published version:Karasimos, A & Isard, A 2004, Multi-lingual Evaluation of a Natural Language Generation System. in LREC2004 Fourth International Conference on Language Resources and Evaluation. European LanguageResources Association (ELRA), pp. 829-832.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:LREC 2004 Fourth International Conference on Language Resources and EvaluationGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019Multi-lingual Evaluation of a Natural Language Generation SystemAthanasios Karasimos∗, Amy Isard†∗Theoretical and Applied Linguistics, University of Edinburgh40 George Square, Edinburgh, UK†School of Informatics, University of Edinburgh2 Buccleuch Place, Edinburgh, UK{alderk@yahoo.co.uk, amy.isard@ed.ac.uk}AbstractThis paper describes a user evaluation of the text output from the M-PIRO (Multilingual Personalised Information Objects) system,which dynamically generates descriptions of exhibits for a virtual museum. We show that subjects performed significantly better in afactual recall test when the descriptions included more sophisticated text structuring modules. The subjects also judged the structuredtexts to be more interesting and readable, and felt that they had learned more from them.1. IntroductionMany natural language generation systems use textstructuring components to perform enhancements such asaggregation or comparisons, but very few formal evalua-tions have been done to confirm that users actually appre-ciate them and benefit from them in terms of understandingand information retention. This study attempts to make afirst step toward providing such an evaluation for the M-PIRO (Multilingual Personalised Information Objects) sys-tem (Isard et al., 2003), which dynamically generates de-scriptions of exhibits for a virtual museum.First, a brief description of the generation system willbe given, focusing on the modules which implement aggre-gation and comparisons, and then the evaluation procedureand results will be discussed.2. The M-PIRO System and TextEnhancementsThe M-PIRO generation system was designed to beused as part of a virtual museum, either on the web, or ina virtual or augmented reality installation. It generates de-scriptions, which are tailored to the individual user, in oneof three languages: English, Greek and Italian. Its architec-ture was based on the earlier ILEX (Intelligent LabellingExplorer) system (O’Donnell et al., 2001) which producedlabels in English for exhibits in a museum jewelery gallery.Both systems use a typical generation architecture com-posed of four stages: content selection, text planning, mi-croplanning and surface realisation (Reiter and Dale, 2000).The information about museum exhibits is stored in ahierarchical database which provides specifications for thebasic classes of the entities present in the domain (exhibit,historical period), as well as the entities themselves (ex-hibit1, Apollo, the Hellenistic period). As a user navigatesthrough the museum, the system keeps a record of the ex-hibits viewed and avoids repeating information where pos-sible. The system also has different settings for three usertypes: adult, child and expert, and facts in the database areassigned scores according to how important and interestingthey are considered to be for each type by the expert curatorinvolved in inputting the information.2.1. AggregationThe Content Selection module packages informationinto verb-based, clause-sized propositions. Texts composedexclusively of sentences based on such single-fact proposi-tions are very likely to contain repetitions and redundan-cies, and are almost certain to be considered boring andunnatural by human readers. To overcome this problem,the M-PIRO system (in common with many such systems)makes use of aggregation algorithms which combine se-mantically related propositions in order to produce a moreconcise and coherent text (Melengoglou, 2002).For example, given the three sentences:This exhibit is a lekythos. It was painted with theblack figure technique. It originates from Attica.the aggregation module could produceThis exhibit is a lekythos; it was painted with theblack figure technique and it originates from At-tica.The underlying decisions on, for example, when a semi-colon is appropriate, and which propositions can actually becombined into an aggregated sentence, depend on a rangeof factors including the type of proposition and the usertype (children will typically be presented with fewer aggre-gated structures).2.2. ComparisonsThe comparison module takes advantage of the hier-archical nature of the domain information in the M-PIROdatabase by grouping entities in the database in terms ofcommon attributes (Lisowska, 2002; Melengoglou, 2002).For example, as a user navigates through the virtual mu-seum, they may come across several exhibits of the sametype, in which case the system can generate an introductionsuch asThis exhibit is another lekythos. Like the previouslekythos, it originates from Attica. 829It is also possible to create negative comparisons such asUnlike the previous vessels, which were createdduring the classical period, this amphora wascreated during the archaic period.In these cases the system is again using fairly sophisticatedtechniques to identify which features the previous exhibitshave in common, and which are contrastable with the newdescription.3. Method3.1. Experimental DesignOur hypotheses were that texts which contain aggrega-tion and comparisons as described above would help read-ers to retain more information and perform better on a fac-tual recall text, and that readers would rate these texts asmore interesting and pleasant to read. The structure of theexperiment was informed by an evaluation carried out onthe ILEX system in which learning outcomes of subjectswho used the dynamic hypertext version of the system werecompared to those who used a static hypertext version (Coxet al., 1999)The evaluation was carried out on both English andGreek texts. Two sequences of six exhibits each were cho-sen for the test suite; one concerned ancient Greek coins,and the other ancient Greek vessels. For each sequence twoversions were prepared, the only difference between thembeing that one used the aggregation and comparison mod-ules and the other did not. All the texts were created withthe adult user type setting.The subjects tested were 20 adult Greek speakers and20 adult English speakers who did not have expert knowl-edge of Ancient Greek archaeological objects. For eachlanguage the subjects were assigned at random to one oftwo equally sized groups: Group A who read the texts aboutvessels in the structured version and coins in the plain ver-sion, and Group B who read the coins in structured versionand the vessels in the plain one. After each set of texts, thesubject was asked to complete a multiple-choice test andafter they had completed the whole experiment, a usabilityquestionnaire was given to them.A pilot study (see Karasimos (2003)) had shown thatsubjects considered the texts which described ancientGreek vessels to be significantly more difficult to under-stand than those which described coins, and this was sub-sequently borne out by the multiple choice scores of themain experiment and the usability questionnaire (see sec-tion 4.2.). Therefore it was decided to present the coinstexts first in all cases. We considered that the possibility ofan ordering effect was less serious than the problem of sub-jects failing to concentrate because of tiredness after read-ing the “more difficult” set of texts.3.2. Example TextsThe examples below, taken from the texts presented tothe subjects, illustrate the differences between the plain andstructured texts. At this point, the subject has already readdescriptions of several ancient Greek vessels.The following is is the first paragraph of the plain textdescription of a stamnos, where no reference is made to anyprevious objects, and each piece of information is conveyedin a separate sentence.This exhibit is a stamnos. It was created duringthe classical period. It dates from circa 420 B.C.It has a picture of Dionysus (centre) being gar-landed by maenads in a state of ecstasy. Onemaenad (left) is filling a skyphos with wine, an-other (right) is playing a drum. This stamnos waspainted by the painter of Dinos. It was decoratedwith the red figure technique. It is made of clay.The next example is the equivalent structured text,where several sentences are aggregated, and comparisonsare made with previous objects.This exhibit is a stamnos. Unlike the previousvessels, which were created during the archaicperiod, this stamnos was created during the clas-sical period. It shows Dionysus (centre) beinggarlanded by maenads in a state of ecstasy. Onemaenad (left) is filling a skyphos with wine, an-other (right) is playing a drum. This stamnos wasdecorated by the painter of Dinos with the red fig-ure technique and is made of clay.Below is a similar example in Greek, taken from firstthe plain text and then the structured text descriptions of atetradrachm (a type of coin).Autì to èkjema eÐnai èna tetrdraqmo.Dhmiourg jhke kat th dirkeia thc ellhnis-tik c periìdou. QronologeÐtai ston 2o ai¸nap.Q. ApeikonÐzei mia aspÐda kai sth mèsh mi-a protom , ìpwc sunhjizìtan sta makedoniknomÐsmata. Autì to tetrdraqmo èqei fti-aqteÐ apì as mi.Autì to èkjema eÐnai llo èna tetrdraq-mo, pou dhmiourg jhke kat th dirkeia thcellhnistik c periìdou. QronologeÐtai ston2o ai¸na p.Q. ApeikonÐzei mia aspÐda kaisth mèsh mia protom , ìpwc sunhjizìtan stamakedonik nomÐsmata. 'Opwc ta prohgoÔmenanomÐsmata, autì to tetrdraqmo èqei ftiaqteÐapì autì as mi.3.3. QuestionnairesThe multiple choice test consisted of 15 questions, andthe highest possible score was 17. Some of the multiplechoice questions concerned just one exhibit, for example:8. Which picture does the stamnos exhibit show?a. A marriage feastb. A man preparing to throw the javelinc. Dionysus surrounded by maenadsd. A young man sitting down and writing with astylusand others required the subjects to consider the texts whichthey had read as a whole: 83014. What is the characteristic which the fewest ofthese exhibits have in common?a. the creation periodb. the painting techniquec. original locationd. museum locationThe usability questionnaire asked for subjective opin-ions about the quality and interest of the texts. The subjectswere asked to assign numerical scores to quantify their in-terest in and enjoyment of the texts and to estimate howmuch they learned from them. They were also asked togive a score for how difficult they found the questions. Inaddition they were asked to state which of the two sets oftexts they preferred from a point of view of fluency.The full set of texts and questions can be found inKarasimos (2003).4. ResultsThe quantitative scores from the multiple choice testwill be discussed first and then the results of the qualita-tive usability questionnaire.4.1. Multiple Choice ScoresStructured PlainE A mean (stdev) 13.7 (1.95) 11 (3.24)N B mean (stdev) 12.4 (2.80) 9.4 (2.80)A+B mean (stdev) 13.05 (2.44) 10.4 (3.12)G A mean (stdev) 12.2 (2.05) 11.1 (1.73)R B mean (stdev) 12.9 (2.56) 10 (2.16)A+B mean (stdev) 12.55 (2.28) 10.55 (1.99)All mean (stdev) 12.8 (2.34) 10.5 (2.58)Table 1: Summary of Results (Group A read structured ves-sels texts, Group B read structured coins texts)A summary of the results from the multiple choice ques-tionnaires is presented in table 1. The average scores werehigher for the structured texts in all cases.Graphs of the scores for individual subjects appear infigure 1 (English) and figure 2 (Greek). Only one subjectperformed better on the texts they read in the plain versionand many performed much better on the structured version.Our hypothesis was that subjects would learn more fromreading structured texts than from reading plain ones. Weperformed a separate two-way repeated measures ANOVAtest on the multiple choice test results for each languagewith the text type (simple or structured) as within-subjectfactor. The subjects’ group (whether they saw the coins orvessels texts in the structured version) was included as abetween-subjects factor.When the English subjects were considered alone, thetext type factor was significant (F(1,18) = 39.44, p < .001)and there was no significant interaction with the group fac-tor.The text type factor was also significant for the Greeksubjects (F(1,18) = 48.32, p < .001). In this case therewas a small but significant interaction with the group factor02468101214161 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Subject idScorePlain Text Structured TextFigure 1: English Subjects02468101214161 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Subject idScorePlain Text Structured TextFigure 2: Greek Subjects(F(1,18) = 9.79, p < .01). In both groups, the subjects per-formed better with the structured text, but there was a muchgreater variability in the group who read the coins texts inthe structured version. A discussion of some reasons fordifferences between the results for the two language groupsis in section 5.We also performed two-way repeated measuresANOVA test with the text type as within-subject factor andtwo between-subjects factors: group and language. Thetext type factor was again significant (F(1,36) = 82.90,p < .001), and there was no significant interaction witheither the group or the language factor.These results therefore support our hypothesis that thesubjects learned more from the texts which they read in astructured version.4.2. Usability QuestionnairesThe results from the usability questionnaire (see table 2show that the subjects considered the structured texts to bemore fluent and interesting than the others, and thought thatthey had learned more from them.Most of the usability questionnaire results showedtrends towards a preference for the structured texts but werenot statistically significant, however the when the subjectswere asked to rate how much they had learned from eachtext on a scale of 1 to 5 (see table 3, there was a signifi- 831Structured No Pref Plainmore fluent 75% 12.5% 12.5%more educational 50% 37.5% 12.5%more interesting 35% 40% 25%more enjoyable 15% 75% 10%Table 2: Subjects’ Preferences by Type of Textcant difference between the mean scores (F(1,36) = 18.79,p < .001). This shows that the subjects were aware thatthey had retained more information from the structuredtexts, even though at this point they had not yet been giventheir scores from the multiple choice test.Structured Plainmean 3.65 3.12stdev .98 1.20Table 3: Subjects’ Estimation of LearningFor a more detailed discussion of the experimental de-sign, pilot studies, and results, see (Karasimos, 2003).5. Discussion and ConclusionsThis study shows that the use of text structuring tech-niques in the M-PIRO system not only results in descrip-tions which are more pleasant to read, but also has a sig-nificant positive effect on the quality of the texts from aneducational point of view. This is an important first step inthe evaluation of a generation system.One aspect of the study which was not fully exploredwas the difference in prior experience of the subject mat-ter between the English and Greek subjects. The namesof the exhibits would have been familiar to a native Greekspeaker, but most would have been completely new to anative speaker of English (e.g. “kylix”, “lekythos”). In ad-dition, the Greek speakers had greater background knowl-edge of the historical periods from which the exhibits date.This seems to have had both positive and negative effectson performance, for example causing subjects to make as-sumptions about the provenance of coins based on priorknowledge rather than relying entirely on facts which theyhad read during the experiment. However, we consider thatthis did not have an effect on the results of the experiment,as we were testing the difference in results obtained fromreading more or less structured texts, rather than absolutescores.Several subjects performed very well on the multiplechoice test, with near perfect scores from both sets of texts,so there was a possible ceiling effect with the results, asthere was no scope for them to do any better on one setthan the other. It would therefore be useful in a future studyto attempt to add some more difficult questions to make thiseffect less likely.In the future, we would like to compare texts producedby the natural language generation system with those writ-ten by a human curator. The handwritten texts would notcontain the comparisons which our system generates, butwe would expect subjects to consider them more fluent, andwe would like to study the interaction between these twoeffects to see whether the experience of visitors to a virtualmuseum is enhanced by the text structuring of which oursystem is capable.6. ReferencesRichard Cox, Mick O’Donnell, and Jon Oberlander. 1999.Dynamic versus static hypermedia in museum education:an evaluation of ILEX, the intelligent labelling explorer.In Proceedings of the Artificial Intelligence in Educationconference (AI-ED99), Le Mans, July.Amy Isard, Jon Oberlander, Ion Androutsopoulos, andColin Matheson. 2003. Speaking the users’ languages.IEEE Intelligent Systems, 18(1):40–45. Special Issue onAdvances in Natural Language Processing.Athanasios Karasimos. 2003. Evaluation of the M-PIROtext output. Master’s thesis, University of Edinburgh.Agnes Lisowska. 2002. The design and implementationof an architecture for using comparisons in the M-PIROdomain. Master’s thesis, University of Edinburgh.Alexander Melengoglou. 2002. Multilingual aggregationin the M-PIRO system. Master’s thesis, University ofEdinburgh.Mick O’Donnell, Chris Mellish, Jon Oberlander, and Alis-tair Knott. 2001. ILEX: An architecture for a dynamichypertext generation system. Natural Language Engi-neering, 7:225–250.Ehud Reiter and Robert Dale. 2000. Building Natural Lan-guage Generation Systems. Cambridge University Press. 832

Multi-lingual Evaluation of a Natural Language Generation System

http://www.lrec-conf.org/proceedings/lrec2004/pdf/134.pdf

Multi-lingual Evaluation of a Natural Language Generation System

Abstract

Similar works

Full text

Available Versions

CiteSeerX

Edinburgh Research Explorer