Hondorp, Hendri

Nijholt, Anton

English

Nijholt, Antinus

Hondorp, G.H.W.

University of Twente Research Information

Some Notes on Agents in the Twente VirtualTheatre EnvironmentAnton Nijholt and Hendri Hondorp?Centre of Telematics and Information TechnologyUniversity of TwentePO Box 217, 7500 AE Enschede, the Netherlands(anijholt|hendri)@cs.utwente.nl1 IntroductionWe present research on visualisation and interaction in a realistic model of anexisting theatre. Our virtual theatre has been built according to design drawingsof the architects of an existing building. Visitors can enter the theatre and walkaround. Information about today’s performances is available on a board thatis automatically updated. Visitors may also go to the information desk, seepreviews and start a (natural language) dialogue with an agent called ‘Karen’who has access to the theatre database with information about performances.A first version of our environment has been available on WWW since early1998. Another version has been installed in a regional technology activity centre.The information that is provided by the system is up to date, many people useit to get informed about performances and we think that at this moment itis the only virtual environment in the world that is really used by the generalpublic. This success has made it possible to start new research. In this paperthe emphasis is on research results that are now being incorporated in a nextversion of the environment, again accessible for the general audience, and newresearch which has to provide results that will be integrated in the environmentin its next update.2 Agents and Interactions2.1 A Navigation AgentNavigation in virtual worlds is a well known problem. Usually, navigation inputis done with keyboard and mouse. This input allows the user to move and torotate, to jump from one location to an other, to interact with objects and totrigger them. We developed a navigation agent that helps the user to explorethe environment and to interact with objects in this environment by means ofspeech commands. Obviously, we do not want completely separated modalities. It? Acknowledgements: Many people are involved in the research reported here andhave contributed to this paper. I would like to mention Joris Hulstijn, Danny Lieand Boris van Schooten.2should be left to the user to choose between the interacting means or to use both,sequentially or simultaneously. A smooth integration of the pointing devices andspeech in a virtual environment requires has to resolve deictic references thatoccur in the interaction. Moreover, the navigation agent should be able to reason(in a modest way) about the geometry of the world in which it moves.Straightforward typed commands or similar speech commands make it possi-ble for the user to explore the virtual environment. Navigation also requires thatnames have to be associated with the different parts of the building, the objectsand the agents. Users may use different words to designate them, including refer-ences that have to be resolved during reasoning. Our present agent uses keywordspotting. A next version of the system will allow a finite state specification ofthe user’s input. We slowly extend and improve the interaction and navigationintelligence of our present navigation agent. At this moment we are exploringthe possibility of speech recognition for several clients on a central server andthe advantages of making the navigation agent visible for the user.Our agent is able to understand command-like speech or keyboard input.It tries to recognise the name of a location in the visitor’s utterance. Whenthe recognition is successful, the agent guides the visitor to this location. Whenthe visitor’s utterance is about performances the navigation agent makes anattempt to contact Karen, the information and transaction agent. Our nextimplementation of the navigation agent assumes that the navigation agent knowsabout (or should be able to compute) the current position and focus of gaze ofthe user, the objects that are in the eyesight of the visitor, properties objectshave, geometric relations between objects and locations, possible walks towardsobjects and locations, a history of previously visited locations or routes, actionsthe agent is performing (or has performed) and the previous communication withthe visitor. This knowledge should make it possible that the navigation agent isable to interpret a majority of user utterances correctly and act accordingly.2.2 The Information and Transaction AgentAs mentioned before, a second agent called Karen allows a natural languagedialogue with the system about performances, artists, dates, prices, etc. Karenwants to sell tickets. She is fed from a database that contains information aboutperformances in some of our local theatres. Developing intelligence for Karen,in this particular environment, is a main aim of our project. Presently the inputto Karen is keyboard-driven natural language and the output in our for thegeneral audience WWW accessible virtual world is screen and menu based. In aprototype system we allow Karen to use a combination of speech synthesis andinformation presentation on the screen. Based on the user utterance, the contextand the database, the system has to decide on a response action, consisting ofdatabase manipulation and dialogue acts.As mentioned earlier, in our experimental system Karen’s spoken dialoguecontribution is presented by visual speech, that is, a ‘talking face’ on the screen,embedded in the virtual world, mouths the questions and part of the responses.If necessary, information is given in a window on the screen and the user can3click items to get more information. The virtual face that has been designedallows animation of lip and face movements and animation of some simple faceexpressions. For speech-image synchronisation 3D images of visemes are calledwhen corresponding phonemes are spoken. Since our application is web-based, itrequires the solution of technical problems dealing with sending and compressingsound files, commands and synchronising sounds and animations.2.3 Language Skills of the AgentsAt this moment our agents have different language skills. On the one hand wehave Karen and a grammar specification of the input for Karen based on a corpusof WoZ obtained keyboard-based dialogue utterances. On the other hand we havea navigation agent with language skills that are based on the current limitationsof speech input. We see two, hopefully, converging developments. On the onehand, speech recognition moves from keyword spotting, to finite state utterancespecification to (word-graph) based context-free language specification. On theother hand, we hope to see the development of corpus-based grammar inferencemethods that allow the design of an interface between domain-dependent speechrecognition, syntactic analysis and interpretation of user dialogue utterances.We have induced for our navigation agent a probabilistic grammar from acorpus of user utterances that have been obtained from scenarios presented to(potential) visitors. This allows the design of a primitive system and it allowsbootstrapping this system from the original corpus and from corpora obtainedfrom logging the interactions between visitors and the navigation agent.2.4 Future AgentsOur approach to designing a virtual environment for an interest community isbottom-up. We started with two agents and in order to introduce others a Javaimplemented agent platform has been designed. Using this platform other agentshave been introduced, e.g., an information board showing upcoming events. Userscan ask questions to the board and get answers. The board passes the questionsto other agents, in this case questions about performances are passed to Karen.One issue that is receiving much attention currently is the non-verbal behaviourof the agents.3 Software Engineering Issues3.1 Use CasesIn our system we have to deal with different agents and with different dialoguesor interactions between agents and users (visitors of our theatre). We are investi-gating the use of object-oriented and agent-oriented design techniques to rebuildour theatre.One design approach which we found useful is the description of use cases.A use case (Cockburn [1]) is a detailed account of the context and goals of the4different actors involved in a transaction, together with a possible sequence ofactions. They are recommended in object-oriented development to find com-mon data elements to be modelled as objects. Use cases become part of therequirement specification document that serves as a contract between the sys-tem developer and the customer, in this case the service provider. Because theycan be intuitively understood, use cases facilitate communication with user focusgroups and with the customer. Based on several scenarios for the navigation andthe information & transaction agent use cases have been developed (Hulstijn[2]).3.2 Formal MethodsBoth from an ergonomical and a software-engineering viewpoint, the design ofinteraction in virtual environments is complex. Virtual environments feature avariety of interactive objects, agents which may use natural language to commu-nicate, and multiple simultaneous users. All may operate in parallel, and mayinteract with each other concurrently. Next to this, the possibility of using Vir-tual Reality techniques to enhance the experience of virtual worlds offers newways of interaction, such as 3D navigation and visualisation, sound effects, andspeech input and output, possibly used so as to complement each other.We attempt to address both of these issues by means of a formal modellingtechnique based on the process algebra CSP. For that reason, in our virtual the-atre a simplified flow of interaction has been specified, showing all relevant inter-action options for any given point in time. The architecture has been modelledin an agent-oriented way, representing all system- and user-controlled objects,and even the users themselves, as parallel processes. The interaction betweenprocesses is modelled by signals passing through specific channels. Interactionmodalities (video versus audio, text versus graphics, etc.) may also be modelledas separate channels.This modelling technique has some strong points. Firstly, a simplified andformal model enables a clear and unambiguous specification of system architec-ture and dynamics. Secondly, it may be useful as a conceptual model, modellingthe fact that a user experiences interaction with other users and agents in asimilar way than in a completed system, and explicitly showing which optionsare available when and through which modalities. Thirdly, it enables automaticprototyping, such as architecture visualisation and verification of some systemproperties (Schooten [3]).References1. Cockburn, A.: Using goal-based use cases - transitioning from theory to practice. J.of object-oriented programming, 1997.2. Hulstijn, J.: Modelling usability: Development methods for dialogue systems. In:Alexandersson, J. et al. (eds.) IJCAI-99 Workshop on Knowledge and Reasoning inPractical Dialogue Systems, Sweden.3. Schooten, B. van et al: Modelling Interaction in Virtual Environments using Process-algebra. In: Interactions in Virtual Worlds, May 1999, University Twente, 195–212.

Some Notes on Agents in the Twente Virtual Theatre Environment

Modelling usability: Development methods for dialogue systems.

Using goal-based use cases - transitioning from theory to practice.

van et al: Modelling Interaction in Virtual Environments using Processalgebra. In: Interactions in Virtual Worlds,

https://ris.utwente.nl/ws/files/6124591/ukvrsig99.pdf

Some Notes on Agents in the Twente Virtual Theatre Environment

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information