This paper describes the NECA MNLG; a fully implemented Multimodal Natural Language Generation module. The MNLG is deployed as part of the NECA system which generates dialogues between animated agents. The generation module supports the seamless integration of full grammar rules, templates and canned text. The generator takes input which allows for the specification of syntactic, semantic and pragmatic constraints on the output

Piwek, Paul

English

arXiv

Crossref

A flexible pragmatics-driven language generator for animated agents

and other research outputs A Flexible pragmatics-driven language generator for animated agent

Paul Piwek

CiteSeerX

A Flexible Pragmatics-driven Language Generator for Animated Agents

Open Research Online

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsA Flexible pragmatics-driven language generator foranimated agentsConference or Workshop ItemHow to cite:Piwek, Paul (2003). A Flexible pragmatics-driven language generator for animated agents. In: 10th Conference of theEuropean Chapter of the Association for Computational Linguistics, 12-17 Apr 2003, Budapest, Hungary.For guidance on citations see FAQs.c© [not recorded]Version: [not recorded]Link(s) to article on publisher’s website:http://mcs.open.ac.uk/pp2464/piwek-163.pdfCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukA Flexible Pragmatics-driven Language Generator for Animated AgentsPaul PiwekITRI – Information Technology Research InstituteUniversity of BrightonPaul.Piwek@itri.bton.ac.ukAbstractThis paper describes the NECA MNLG;a fully implemented Multimodal Natu-ral Language Generation module. TheMNLG is deployed as part of the NECAsystem which generates dialogues be-tween animated agents. The genera-tion module supports the seamless inte-gration of full grammar rules, templatesand canned text. The generator takes in-put which allows for the specification ofsyntactic, semantic and pragmatic con-straints on the output.1 IntroductionThis paper introduces the NECA MNLG; a Multi-modal Natural Language Generator. It has beendeveloped in the context of the NECA system.1The NECA system generates dialogue scripts foranimated characters. A first demonstrator in thecar sales domain (ESHOWROOM) has been imple-mented. It allows a user to browse a database ofcars, select a car, select two characters and theirattributes, and subsequently view an automaticallygenerated film of a dialogue about the selected car.The demonstrator takes the following input:• A database with facts about the selected car (maximumspeed, horse power, etc.).• A database which correlates facts with value judge-ments.1NECA stands for ‘Net Environment for Embodied Emo-tional Conversational Agents’ and is an EU-IST project.• Information about the characters: 1. Personality traitssuch as extroversion and agreeableness. 2. Personalpreferences concerning cars (e.g., a preference for safecars). 3. Role of the character (seller or customer).This input is processed in a pipeline that consistsof the following modules in this order:• A DIALOGUE PLANNER, which produces an abstractdescription of the dialogue (the dialogue plan).• A MULTI-MODAL NATURAL LANGUAGE GENERA-TOR which specifies linguistic and non-linguistic real-izations for the dialogue acts in the dialogue plan.• A SPEECH SYNTHESIS MODULE, which adds infor-mation for Speech.• A GESTURE ASSIGNMENT MODULE, which controlsthe temporal coordination of gestures and speech.• A PLAYER, which plays the animated characters andthe corresponding speech sound files.Each step in the pipeline adds more concrete in-formation to the dialogue plan/script until finallya player can render it. A single XML compliantrepresentation language, called RRL, has been de-veloped for representing the Dialogue Script at itsvarious stages of completion (Piwek et al., 2002).In this paper, we describe the requirements forthe NECA MNLG, how these have been translatedinto design solutions and finally some of aspectsof the implementation.2 RequirementsThe requirements in this section derive primarlyfrom the use case of the NECA system. We do,however, try to indicate in what respects these re-quirements transcend this specific application andare desirable for generation systems in general.REQUIREMENT 1: The linguistic resources of the gen-erator should support seamless integration of canned text,templates and full grammar rules.In the NECA system, the dialogue planner createsa dialogue plan consisting of (1) a descriptionof the participants, (2) a characterization of thecommon ground at the outset of the dialogue interms of Discourse Representation Theory (Kampand Reyle, 1993) and (3) a set of dialogue actsand their temporal ordering. For each dialogueact, the type, speaker, set of addressees, semanticcontent, what it is a reaction to (i.e., its rhetoricalrelation to other dialogue acts), and emotionsof the speaker can be specified. The amount ofinformation which the dialogue planner actuallyprovides for each of these attributes varies,however, per dialogue act: for some dialogue acts,a full semantic content can be provided –in theform of a Discourse Representation Structure–whereas for other acts, no semantic content isavailable at all. Typically, the dialogue plannercan provide detailed semantics for utteranceswhose content is covered by the domain model(e.g., the car domain) whereas this is not possiblefor utterances which play an important role in theconversation but are not part of the domain model(e.g., greetings). This state of affairs is sharedwith most real-world applications.Since generation by grammar rules is primarilydriven by the input semantics, for certain dialogueacts full grammar rules cannot be used. Thesedialogue acts may be primarily characterized interms of their, possibly domain specific, dialogueact type (greeting, refusal, etc.). Thus, we needa generator which can cope with both types ofinput, and map them to the appropriate output.Input with little or no semantic content can typ-ically be dealt with through templates or cannedtext, whereas input with fully specified semanticcontent can be dealt with through proper grammarrules. Summarizing, we need a generator whichcan cope with (linguistic) resources that containan arbritary combination of grammar rules,templates and canned text.REQUIREMENT 2: The generator should allow forcombinations of different types of constraints on its the out-put, such as syntactic, semantic and pragmatic constraintsIn the NECA project the aim is to generatebehaviour for animated agents which simulatesaffective situated face-to-face conversationalinteraction. This means that the utterances of theagents have to be adapted not only to the contentof the information which is exchanged but also tomany other properties of the interlocutors, such astheir emotional state, gender, cultural background,etc. The generator should therefore allow for suchparameters to be part of its input.REQUIREMENT 3: The generator should be sufficiently fastto be of use in real-world applicationsThe application in which our generator isbeing used is currently fielded as part of a net-environment. The application will be evaluatedwith users through online questionnaires whichare integrated in the application and analysis oflog files (to answer questions such as ‘Do userstry different settings of the application?’, etc. SeeKrenn et al., 2002). Therefore, the generator willhave to be fast in order for it not to negativelyaffect the user experience of the system.3 Design SolutionsThe NECA MNLG adopts the conventional pipelinearchitecture for generators (Reiter and Dale,2000). Its input is a RRL dialogue plan. Thisis parsed and internally represented as a PROFITtyped feature structure (Erbach, 1995). Subse-quently, the dialogue acts in the plan are realizedin accordance with their temporal order. For eachact, first a deep syntactic structure is generated.The deep structure of referring expressions is dealtwith in a separate module, which takes the com-mon ground of the interlocutors into account. Sub-sequently, lexical realization (agreement, inflec-tion) and punctuation is performed. Finally, turn-taking gestures are added and the output is mappedback into the RRL XML format.Here let us concentrate on our approach to thegeneration of deep syntactic structure and how itsatisfies the first two requirements. The input tothe MNLG is a node (i.e., feature structure) stipu-lating the syntactic type of the output (e.g., sen-tence: <s), semantics and further information onthe current dialogue act in PROFIT:2(<s &sem!drs([c_27],[type(c_27,prestigious),arg1(c_27,x_1)])&currentAct!speaker!(name!john &polite!yes & ...))Thus various types of information are combinedwithin one input node. Generation consists of tak-ing the input node and using it to create a treerepresentation of the output. For this purpose,the MNLG tries to match the input node with themother node of one of the trees in its tree repos-itory. This tree repository contains trees whichcan represent proper grammar rules, templates andcanned text. Matching trees might in turn have in-complete daughter nodes. These are recursivelyexpanded by matching them with the trees in therepository, until all daughters are complete.A daughter node is complete if it is lexicallyrealized (i.e., the attribute form of the node hasa value) or it is of the type <np and the seman-tics is an open variable. In the latter instance, thenode is expanded in a separate step by the refer-ring expressions generation module. This modulefinds the discourse referent in the common groundwhich binds the open variable and constructs a de-scription of the object in question. The descrip-tion is composed of the properties which the ob-ject has according to the common ground, but canalso be empty if the object is highly salient. Themodule is based on the work of Krahmer and The-une (2002). The (empty) description is mappedto a deep syntactic structure using the tree repos-itory. Lexicalization subsequently yields expres-sions such as ‘it’ (empty descriptive content) or,for instance, ‘the red car’.Let us return to the tree repository and illus-trate how templates and rules can be representeduniformly. The representation of a tree is of the2That is, PROLOG with some sugaring for the rep-resentation of feature structures. Feature structures arealso used in the FUF/SURGE generator. It is differentfrom the NECA MNLG in that it takes as input thematictrees with content words. Furthermore, it allows for con-trol annotations in the grammar and uses a special inter-preter for unification, rather than directly PROLOG. Seehttp://www.cs.bgu.ac.il/surge/.form (Node,[Tree1,Tree2,...]), wherethe list of trees can be empty, yielding a tree con-sisting of one node: (Node,[]). The followingis a template for dialogue acts of type greetingwith no semantic content and a polite speaker.(<s &currentAct!(type!greeting &speaker!polite!"yes" &speaker!name!Speaker) &sem!"none",[(<s & form!"hello!",[]),(<fragment &form!"My name is",[]),(<np &sem!concept(Speaker),[])]).This is a template for the text ‘Hello! My name isSPEAKER’. Where SPEAKER is a variable whichis bound to the name of the speaker of the utter-ance. The noun phrase (<np) for this name is gen-erated by the referring expression generation mod-ule. The following is a tree representing a gram-mar rule of the familiar type S → NP VP:(<s &currentAct!type!statement &currentAct!CA &argGap!ArgGap &auxGap!AuxGap &sem!drs(_,[negation(drs(_,[type(E,Type)arg1(E,X)|R]))]),[(<np &currentAct!CA &sem!X,[]),(<vp &argGap!ArgGap &auxGap!AuxGap &negated!<true &sem!drs(_,[type(E,Type)|R]) &currentAct!CA,_)]).Note that this rule applies to an input node whosesemantic content contains a negation. The nega-tion is passed on to the VP subtree via the featurenegated. The attributes argGap and auxGapallow us to capture unbounded dependencies viafeature perlocation. Our use of trees is related tothe Tree Adjoining Grammar approach to genera-tion (e.g., Stone and Doran, 1997).33Their generation algorithm is, however, very differentfrom the one proposed here. Whereas they propose an in-tegrated planning approach, we advocate a very modular sys-The value of the attribute currentAct ispassed on from the mother node to the daughternodes. Thus any pragmatic information (personal-ity, politeness, emotion, etc.) is passed on throughthe tree and can be accessed at a later stage, forinstance, when lexical items are selected.4 ImplementationThe NECA MNLG has been implemented in PRO-LOG. The output is in the form of an RRL XMLdocument. Table 1 provides a sample of the re-sponse times of the compiled code running on aPentium III Mobile 1200 Mhz with Sicstus 3.8.5PROLOG. We timed the complete generation pro-cess from parsing the XML input to producingXML output, including generation of deep syn-tactic structure, referring expressions, turn takinggestures (not discussed in this paper), etc.input # acts = 1 ≤ 10A 19 0.230s 0.741sB 22 0.290s 0.872sC 23 0.290s 0.801sD 31 0.431s 1.372sTable 1: Response Times of the MNLGThe results show generation times for entire di-alogues and according to whether the generatorwas asked to produce exactly one solution or se-lect at random a solution from a set of at most tengenerated solutions (the latter strategy was imple-mented to obtain more variation in the generatoroutput). On average for = 1 the generation timefor an individual dialogue act is almost 1100 of asecond. For ≤ 10 it is 4100 of a second. Thegenerator uses a repository of 138 trees (includ-ing the two examples given above). The repos-itory has been developed for and integrated intothe ESHOWROOM system which is currently be-ing fielded. A start is being made with porting theMNLG to a new domain and documentation is be-ing created to allow our project partners to carryout this task. We hope that our efforts will con-tribute to addressing a challenge expressed in (Re-tem, supporting fast generation. Moreover, by using featuresfor unbounded dependencies we do not require the adjunctionoperation, which is incompatible with our topdown genera-tion approach. We follow Nicolov et al. (1996), who also useTAG, in their commitment to flat semantics. Their generatordoes, however, not take pragmatic constraints into account.iter, 1999): “We hope that future systems such asSTOP will be able to make more use of deep tech-niques, because of advances in linguistics and thedevelopment of reusable wide-coverage NLG com-ponents that are robust, well-documented and wellengineered as software artifacts.”In our view the best way to approach this goalis by providing a framework which allows for theflexible integration of shallow and deep genera-tion, thus making it possible that in the course ofvarious projects, deep analyses can be developedalongside the shallow solutions which are diffi-cult to avoid altogether in software developmentprojects, due to the pressure to deliver a completesystem within a certain span of time.AcknowledgementsThis research is supported by the EU Project NECAIST-2000-28580. For comments and discussionthanks are due the EACL reviewers and my col-leagues in the NECA project.ReferencesGregor Erbach. 1995. PROFIT 1.54 user’s guide. Universityof the Saarland, December 3, 1995.Hans Kamp and Uwe Reyle. 1993. From Discourse toLogic. Kluwer, Dordrecht.Emiel Krahmer and Marie¨t Theune. 2002. Efficient context-sensitive generation of referring expressions. In: KeesVan Deemter and Rodger Kibble (eds.), InformationSharing, CSLI, Stanford.Brigitte Krenn, Erich Gstrein, Barbara Neumayr and Mar-tine Grice. 2002. What can we learn from usersof avatars in net environments?. In: Proc. of theAAMAS workshop “Embodied conversational agents -let’s specify and evaluate them!”, Bologna, Italy.Nicholas Nicolov, Chris Mellish & Graeme Ritchie. 1996.Approximate Generation from Non-Hierarchical Rep-resentattions, Proc. 8th International Workshop onNatural Language Generation, Herstmonceux Castle,UK.Paul Piwek, Brigitte Krenn, Marc Schro¨der, Martine Grice,Stefan Baumann and Hannes Pirker. 2002. RRL: ARich Representation Language for the Description ofAgent Behaviour in NECA. Proc. of the AAMAS work-shop “Embodied conversational agents - let’s specifyand evaluate them!”, Bologna, Italy.Ehud Reiter. 1999. Shallow vs. Deep Techniques for han-dling Linguistic Constraints and Optimisations. Proc.of KI-99 Workshop ”May I speak freely”.Ehud Reiter and Robert Dale. 2000. Building naturallanguage generation systems. Cambridge UniversityPress, Cambridge.Matthew Stone and Christy Doran. 1997. Sentence Plan-ning as Description Using Tree-Adjoining Grammar.Proc. ACL 1997, Madrid, Spain.

A Flexible pragmatics-driven language generator for animated agents

This paper describes the NECA MNLG; a fully implemented Multimodal Natu-ral Language Generation module. The MNLG is deployed as part of the NECA system which generates dialogues be-tween animated agents. The genera-tion module supports the seamless inte-gration of full grammar rules, templates and canned text. The generator takes in-put which allows for the specification of syntactic, semantic and pragmatic con-straints on the output. 

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.342.2893

A Flexible pragmatics-driven language generator for animated agents

Abstract

Similar works

Full text

Available Versions

Crossref

CiteSeerX

Open Research Online

CiteSeerX