In this paper we investigate how much data is required to train an algorithm for attribute selection, a subtask of Referring Expressions Generation (REG). To enable comparison between different-sized training sets, a systematic training method was developed. The results show that depending on the complexity of the domain, training on 10 to 20 items may already lead to a good performance

Koolen, Ruud

Krahmer, Emiel

Theune, Mariët

Wubben, Sander

English

Theune, M.

Koolen, R.M.F.

Krahmer, E.J.

Wubben, S.

NARCIS 

Does size matter - How much data is required to train a REG algorithm?

Theune, Mariet

Does Size Matter – How Much Data is Required to Train a REG Algorithm?

University of Twente Research Information

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:shortpapers, pages 660–664,Portland, Oregon, June 19-24, 2011. c©2011 Association for Computational LinguisticsDoes Size Matter – How Much Data is Required to Train a REG Algorithm?Mari ët TheuneUniversity of TwenteP.O. Box 2177500 AE EnschedeThe Netherlandsm.theune@utwente.nlRuud KoolenTilburg UniversityP.O. Box 901355000 LE TilburgThe Netherlandsr.m.f.koolen@uvt.nlEmiel KrahmerTilburg UniversityP.O. Box 901355000 LE TilburgThe Netherlandse.j.krahmer@uvt.nlSander WubbenTilburg UniversityP.O. Box 901355000 LE TilburgThe Netherlandss.wubben@uvt.nlAbstractIn this paper we investigate how much datais required to train an algorithm for attributeselection, a subtask of Referring ExpressionsGeneration (REG). To enable comparison be-tween different-sized training sets, a system-atic training method was developed. The re-sults show that depending on the complexityof the domain, training on 10 to 20 items mayalready lead to a good performance.1 IntroductionThere are many ways in which we can refer to ob-jects and people in the real world. A chair, for ex-ample, can be referred to as red, large, or seen fromthe front, while men may be singled out in termsof their pogonotrophy (facial hairstyle), clothing andmany other attributes. This poses a problem for al-gorithms that automatically generate referring ex-pressions: how to determine which attributes to use?One solution is to assume that some attributesare preferred over others, and this is indeed whatmany Referring Expressions Generation (REG) al-gorithms do. A classic example is the IncrementalAlgorithm (IA), which postulates the existence ofa complete ranking of relevant attributes (Dale andReiter, 1995). The IA essentially iterates throughthis list of preferred attributes, selecting an attributefor inclusion in a referring expression if it helps sin-gling out the target from the other objects in thescene (the distractors). Crucially, Dale and Reiter donot specify how the ranking of attributes should bedetermined. They refer to psycholinguistic researchsuggesting that, in general, absolute attributes (suchas color) are preferred over relative ones (such assize), but stress that constructing a preference orderis essentially an empirical question, which will dif-fer from one domain to another.Many other REG algorithms similarly rely onpreferences. The graph-based based REG algorithm(Krahmer et al., 2003), for example, models prefer-ences in terms of costs, with cheaper properties be-ing more preferred. Various ways to compute costsare possible; they can be defined, for instance, interms of log probabilities, which makes frequentlyencountered properties cheap, and infrequent onesmore expensive. Krahmer et al. (2008) argue thata less fine-grained cost function might generalizebetter, and propose to use frequency informationto, somewhat ad hoc, define three costs: 0 (free),1 (cheap) and 2 (expensive). This approach wasshown to work well: the graph-based algorithm wasthe best performing system in the most recent REGChallenge (Gatt et al., 2009).Many other attribute selection algorithms alsorely on training data to determine preferences in oneform or another (Fabbrizio et al., 2008; Gervás etal., 2008; Kelleher, 2007; Spanger et al., 2008; Vi-ethen and Dale, 2010). Unfortunately, suitable datais hard to come by. It has been argued that determin-ing which properties to include in a referring expres-sion requires a “semantically transparent” corpus(van Deemter et al., 2006): a corpus that containsthe actual properties of all domain objects as wellas the properties that were selected for inclusion ina given reference to the target. Obviously, text cor-pora tend not to meet this requirement, which is why660semantically transparent corpora are often collectedusing human participants who are asked to producereferring expressions for targets in controlled visualscenes for a given domain. Since this is a time con-suming exercise, it will not be surprising that suchcorpora are thin on the ground (and are often onlyavailable for English). An important question there-fore is how many human-produced references areneeded to achieve a certain level of performance. Dowe really need hundreds of instances, or can we al-ready make informed decisions about preferences ona few or even one training instance?In this paper, we address this question by sys-tematically training the graph-based REG algorithmon a number of “semantically transparent” data setsof various sizes and evaluating on a held-out testset. The graph-based algorithm seems a good can-didate for this exercise, in view of its performancein the REG challenges. For the sake of compari-son, we also follow the evaluation methodology ofthe REG challenges, training and testing on two do-mains (a furniture and a people domain), and usingtwo automatic metrics (Dice and accuracy) to mea-sure human-likeness. One hurdle needs to be takenbeforehand. Krahmer et al. (2008) manually as-signed one of three costs to properties, loosely basedon corpus frequencies. For our current evaluationexperiments, this would hamper comparison acrossdata sets, because it is difficult to do it in a mannerthat is both consistent and meaningful. Therefore wefirst experiment with a more systematic way of as-signing a limited number of frequency-based coststo properties usingk-means clustering.2 Experiment I: k-means clustering costsIn this section we describe our experiment withk-means clustering to derive property costs from En-glish and Dutch corpus data. For this experiment welooked at both English and Dutch, to make sure thechosen method does not only work well for English.2.1 MaterialsOur English training and test data were taken fromthe TUNA corpus (Gatt et al., 2007). This semanti-cally transparent corpus contains referring expres-sions in two domains (furniture and people), col-lected in one of two conditions: in the -LOC con-dition, participants were discouraged from mention-ing the location of the target in the visual scene,whereas in the +LOC condition they could mentionany properties they wanted. The TUNA corpus wasused for comparative evaluation in the REG Chal-lenges (2007-2009). For training in our current ex-periment, we used the -LOC data from the trainingset of the REG Challenge 2009 (Gatt et al., 2009):165 furniture descriptions and 136 people descrip-tions. For testing, we used the -LOC data from theTUNA 2009 development set: 38 furniture descrip-tions and 38 people descriptions.Dutch data were taken from the D-TUNA corpus(Koolen and Krahmer, 2010). This corpus uses thesame visual scenes and annotation scheme as theTUNA corpus, but with Dutch instead of Englishdescriptions. D-TUNA does not include locations asobject properties at all, hence our restriction to -LOCdata for English (to make the Dutch and English datamore comparable). As Dutch test data, we used 40furniture items and 40 people items, randomly se-lected from the textual descriptions in the D-TUNAcorpus. The remaining furniture and people descrip-tions (160 items each) were used for training.2.2 MethodWe first determined the frequency with which eachproperty was mentioned in our training data, relativeto the number of target objects with this property.Then we created different cost functions (mappingproperties to costs) by means ofk-means clustering,using the Weka toolkit. Thek-means clustering al-gorithm assignsn points in a vector space tok clus-ters (S1 to Sk) by assigning each point to the clus-ter with the nearest centroid. The total intra-clustervarianceV is minimized by the functionV =k∑i=1∑xj∈Si(xj − µi)2whereµi is the centroid of all the pointsxj ∈ Si.In our case, the pointsn are properties, the vectorspace is one-dimensional (frequency being the onlydimension) andµi is the average frequency of theproperties inSi. The cluster-based costs are definedas follows:∀xj ∈ Si, cost(xj) = i− 1661whereS1 is the cluster with the most frequentproperties,S2 is the cluster with the next most fre-quent properties, and so on. Using this approach,properties from clusterS1 get cost 0 and thus can beadded “for free” to a description. Free properties arealways included, provided they help distinguish thetarget. This may lead to overspecified descriptions,mimicking the human tendency to mention redun-dant properties (Dale and Reiter, 1995).We ran the clustering algorithm on our Englishand Dutch training data for up to six clusters (k = 2to k = 6). Then we evaluated the performance ofthe resulting cost functions on the test data fromthe same language, using Dice (overlap between at-tribute sets) and Accuracy (perfect match betweensets) as evaluation metrics. For comparison, we alsoevaluated the best scoring cost functions from Theu-ne et al. (2010) on our test data. These “Free-Naı̈ve”(FN) functions were created using the manual ap-proach sketched in the introduction.The order in which the graph-based algorithmtries to add attributes to a description is explicitlycontrolled to ensure that “free” distinguishing prop-erties are included (Viethen et al., 2008). In ourtests, we used an order of decreasing frequency; i.e.,always examining more frequent properties first.12.3 ResultsFor the cluster-based cost functions, the best perfor-mance was achieved withk = 2, for both domainsand both languages. Interestingly, this is the coarsestpossiblek-means function: with only two costs (0and 1) it is even less fine-grained than the FN func-tions advocated by Krahmer et al. (2008). The re-sults for thek-means costs withk = 2 and the FNcosts of Theune et al. (2010) are shown in Table 1.No significant differences were found, which sug-gests thatk-means clustering, withk = 2, can beused as a more systematic alternative for the manualassignment of frequency-based costs. We thereforeapplied this method in the next experiment.3 Experiment II: varying training set sizeTo find out how much training data is requiredto achieve an acceptable attribute selection perfor-1We used slightly different property orders than Theune etal. (2010), leading to minor differences in our FN results.Furniture PeopleLanguage Costs Dice Acc. Dice Acc.English k-means 0.810 0.50 0.733 0.29FN 0.829 0.55 0.733 0.29Dutch k-means 0.929 0.68 0.812 0.33FN 0.929 0.68 0.812 0.33Table 1: Results fork-means costs withk = 2 and theFN costs of Theune et al. (2010) on Dutch and English.mance, in the second experiment we derived costfunctions and property orders from different sizedtraining sets, and evaluated them on our test data.For this experiment, we only used English data.3.1 MaterialsAs training sets, we used randomly selected subsetsof the full English training set from Experiment I,with set sizes of 1, 5, 10, 20 and 30 items. Be-cause the accidental composition of a training setmay strongly influence the results, we created 5 dif-ferent sets of each size. The training sets were builtup in a cumulative fashion: we started with five setsof size 1, then added 4 items to each of them to cre-ate five sets of size 5, etc. This resulted in five seriesof increasingly sized training sets. As test data, weused the same English test set as in Experiment I.3.2 MethodWe derived cost functions (usingk-means clusteringwith k = 2) and orders from each of the trainingsets, following the method described in Section 2.2.In doing so, we had to deal with missing data: not allproperties were present in all data sets.2 For the costfunctions, we simply assigned the highest cost (1)to the missing properties. For the order, we listedproperties with the same frequency (0 for missingproperties) in alphabetical order. This was done forthe sake of comparability between training sets.3.3 ResultsTo determine significance, we calculated the meansof the scores of the five training sets for each setsize, so that we could compare them with the scoresof the entire set. We applied repeated measures of2This problem mostly affected the smaller training sets. Byset size 10 only a few properties were missing, while by set size20, all properties were present in all sets.662variance (ANOVA) to the Dice and Accuracy scores,usingset size (1, 5, 10, 20, 30, entire set) as a withinvariable. The mean results for each training set sizeare shown in Table 2.3 The general pattern is thatthe scores increase with the size of the training set,but the increase gets smaller as the set sizes becomelarger.Furniture PeopleSet size Dice Acc. Dice Acc.1 0.693 0.25 0.560 0.135 0.756 0.34 0.620 0.1510 0.777 0.40 0.686 0.2020 0.788 0.41 0.719 0.2530 0.782 0.41 0.718 0.27Entire set 0.810 0.50 0.733 0.29Table 2: Mean results for the different set sizes.In the furniture domain, we found a main effectof set size (Dice: F(5,185) = 7.209,p < .001; Ac-curacy: F(5,185) = 6.140,p < .001). To see whichset sizes performed significantly different as com-pared to the entire set, we conducted Tukey’s HSDpost hoc comparisons. For Dice, the scores of setsize 10 (p = .141), set size 20 (p = .353), and setsize 30 (p = .197) did not significantly differ fromthe scores of the entire set of 165 items. The Accu-racy scores in the furniture domain show a slightlydifferent pattern: the scores of the entire training setwere still significantly higher than those of set size30 (p < .05). This better performance when trainedon the entire set may be caused by the fact that notall of the five training sets that were used for set sizes1, 5, 10, 20 and 30 performed equally well.In the people domain we also found a main effectof set size (Dice: F(5,185) = 21.359,p < .001; Accu-racy: F(5,185) = 8.074,p < .001). Post hoc pairwisecomparisons showed that the scores of set size 20(Dice: p = .416; Accuracy:p = .146) and set size30 (Dice: p = .238; Accuracy: p = .324) did notsignificantly differ from those of the full set of 136items.3For comparison: in the REG Challenge 2008, (which in-volved a different test set, but the same type of data), the bestsystems obtained overall Dice and accuracy scores of around0.80 and 0.55 respectively (Gatt et al., 2008). These scores maywell represent the performance ceiling for speaker and contextindependent algorithms on this task.4 DiscussionExperiment II has shown that when using small datasets to train an attribute selection algorithm, resultscan be achieved that are not significantly differentfrom those obtained using a much larger trainingset. Domain complexity appears to be a factor inhow much training data is needed: using Dice as anevaluation metric, training sets of 10 sufficed in thesimple furniture domain, while in the more complexpeople domain it took a set size of 20 to achieve re-sults that do not significantly differ from those ob-tained using the full training set.The accidental composition of the training setsmay strongly influence the attribute selection per-formance. In the furniture domain, we found cleardifferences between the results of specific trainingsets, with “bad sets” pulling the overall performancedown. This affected Accuracy but not Dice, possiblybecause the latter is a less strict metric.Whether the encouraging results found for thegraph-based algorithm generalize to other REG ap-proaches is still an open question. We also needto investigate how the use of small training sets af-fects effectiveness and efficiency of target identifica-tion by human subjects; as shown by Belz and Gatt(2008), task-performance measures do not necessar-ily correlate with similarity measures such as Dice.Finally, it will be interesting to repeat Experiment IIwith Dutch data. The D-TUNA data are cleaner thanthe TUNA data (Theune et al., 2010), so the risk of“bad” training data will be smaller, which may leadto more consistent results across training sets.5 ConclusionOur experiment has shown that with 20 or less train-ing instances, acceptable attribute selection resultscan be achieved; that is, results that do not signif-icantly differ from those obtained using the entiretraining set. This is good news, because collectingsuch small amounts of training data should not taketoo much time and effort, making it relatively easyto do REG for new domains and languages.AcknowledgmentsKrahmer and Koolen received financial support fromThe Netherlands Organization for Scientific Re-search (Vici grant 27770007).663ReferencesAnja Belz and Albert Gatt. 2008. Intrinsic vs. extrinsicevaluation measures for referring expression genera-tion. In Proceedings of ACL-08: HLT, Short Papers,pages 197–200.Robert Dale and Ehud Reiter. 1995. Computational in-terpretation of the Gricean maxims in the generation ofreferring expressions.Cognitive Science, 19(2):233–263.Giuseppe Di Fabbrizio, Amanda Stent, and SrinivasBangalore. 2008. Trainable speaker-based refer-ring expression generation. InTwelfth Conference onComputational Natural Language Learning (CoNLL-2008), pages 151–158.Albert Gatt, Ielka van der Sluis, and Kees van Deemter.2007. Evaluating algorithms for the generation of re-ferring expressions using a balanced corpus. InPro-ceedings of the 11th European Workshop on NaturalLanguage Generation (ENLG 2007), pages 49–56.Albert Gatt, Anja Belz, and Eric Kow. 2008. TheTUNA Challenge 2008: Overview and evaluation re-sults. InProceedings of the 5th International NaturalLanguage Generation Conference (INLG 2008), pages198–206.Albert Gatt, Anja Belz, and Eric Kow. 2009. The TUNA-REG Challenge 2009: Overview and evaluation re-sults. InProceedings of the 12th European Workshopon Natural Language Generation (ENLG 2009), pages174–182.Pablo Gerv́as, Raquel Herv́as, and Carlos Ĺeon. 2008.NIL-UCM: Most-frequent-value-first attribute selec-tion and best-scoring-choice realization. InProceed-ings of the 5th International Natural Language Gener-ation Conference (INLG 2008), pages 215–218.John Kelleher. 2007. DIT - frequency based incremen-tal attribute selection for GRE. InProceedings of theMT Summit XI Workshop Using Corpora for NaturalLanguage Generation: Language Generation and Ma-chine Translation (UCNLG+MT), pages 90–92.Ruud Koolen and Emiel Krahmer. 2010. The D-TUNAcorpus: A Dutch dataset for the evaluation of refer-ring expression generation algorithms. InProceedingsof the 7th international conference on Language Re-sources and Evaluation (LREC 2010).Emiel Krahmer, Sebastiaan van Erk, and André Verleg.2003. Graph-based generation of referring expres-sions.Computational Linguistics, 29(1):53–72.Emiel Krahmer, Marïet Theune, Jette Viethen, and IrisHendrickx. 2008. GRAPH: The costs of redundancyin referring expressions. InProceedings of the 5th In-ternational Natural Language Generation Conference(INLG 2008), pages 227–229.Philipp Spanger, Takehiro Kurosawa, and TakenobuTokunaga. 2008. On “redundancy” in selecting at-tributes for generating referring expressions. InCOL-ING 2008: Companion volume: Posters, pages 115–118.Mariët Theune, Ruud Koolen, and Emiel Krahmer. 2010.Cross-linguistic attribute selection for REG: Compar-ing Dutch and English. InProceedings of the 6th In-ternational Natural Language Generation Conference(INLG 2010), pages 174–182.Kees van Deemter, Ielka van der Sluis, and Albert Gatt.2006. Building a semantically transparent corpus forthe generation of referring expressions. InProceed-ings of the 4th International Natural Language Gener-ation Conference (INLG 2006), pages 130–132.Jette Viethen and Robert Dale. 2010. Speaker-dependentvariation in content selection for referring expressiongeneration. InProceedings of the 8th AustralasianLanguage Technology Workshop, pages 81–89.Jette Viethen, Robert Dale, Emiel Krahmer, Mariët Theu-ne, and Pascal Touset. 2008. Controlling redundancyin referring expressions. InProceedings of the SixthInternational Conference on Language Resources andEvaluation (LREC 2008), pages 239–246.664

Tilburg University Repository

Does Size Matter – How Much Data is Required to Train a REG Algorithm?

Abstract

Similar works

Full text

Available Versions

NARCIS

NARCIS

University of Twente Research Information

Tilburg University Repository

Tilburg University Repository