Bui Huu Trung, B.H.T.

Hofs, D.H.W.

van Schooten, B.W.

English

Partially Observable Markov Decision Processes (POMDPs) are attractive for dialogue management because they are made to deal with noise and partial information. This paper addresses the problem of using them in a practical development cycle. We apply factored POMDP models to three applications. We examine our experiences with respect to design choices and issues, and compare performance with hand-crafted policies

University of Twente Research Information

Practical dialogue manager development using POMDPsTrung H. Bui, Boris van Schooten, and Dennis HofsUniversity of TwentePO Box 217 Enschede, The Netherlands{buith, schooten, hofs}@ewi.utwente.nlAbstractPartially Observable Markov Decision Pro-cesses (POMDPs) are attractive for dialoguemanagement because they are made to dealwith noise and partial information. This pa-per addresses the problem of using them in apractical development cycle. We apply fac-tored POMDP models to three applications.We examine our experiences with respect todesign choices and issues, and compare per-formance with hand-crafted policies.1 IntroductionPartially Observable Markov Decision Processes(POMDPs) are attractive for dialogue managementin the cases where the dialogue manager has tomake choices which depend on statistical informa-tion. They can determine optimal strategies in theface of error and partial information. POMDPs cantake advantage of statistical information about be-havior or error to the fullest extent, and take into ac-count extensive hidden information.Using POMDPs for spoken dialogue manage-ment has been examined thoroughly in (Williamsand Young, 2007). Current POMDP-based dialoguemanagers model a complete slot-filling dialogue in-cluding all slots with all values. Large numbers ofslots and values lead to a large state space, whichis not tractable for current POMDP solvers. Usu-ally, this restricts us to toy problems. Recent ef-fort to scale up POMDP-based models is reportedin (Williams and Young, 2007; Bui et al., 2007).It is not yet clear enough how to employ POMDPsin a systematic development cycle. A number ofpractical issues with POMDPs has not really beenaddressed yet. How do you obtain the user modeland the probability distributions? How do you testand debug POMDPs? How do you tweak rewardvalues? How do you evaluate and compare per-formance of the POMDP policy which other ap-proaches? We address these questions by usingthe factored POMDP models (Williams and Young,2007; Bui et al., 2007) as a basis, and applying themto three dialogue management systems.2 MethodologyDesign guidelines. The state space represents theuser’s state and action. It is defined as a set of fea-tures. We should keep it compact. This can bedone by specifying only features which are relevantin selecting the system action and by pruning allunreachable states. For example, when analyzingthe Williams’s 1945-state travel problem (Williamsand Young, 2007), we found that could increasetractability by pruning 1626 states1, leaving only319 reachable states. The system actions are notonly the actions toward the user but also actions forother dialogue manager tasks such as querying thedatabase. Similar to the state space, the observationspace is also defined as a set of observation featuressuch as user’s action with noise (from the ASR) andobserved user’s emotional state.Designing a reward model that leads to a goodpolicy is a very challenging task. The typical param-eters used to design a reward model are task success,the number of turns, and dialogue act appropriate-ness (for example, the system should not confirm avalue if it has not yet been provided by the user). Theprecise numerical values used may have significantimpact on the policy and convergence behaviour.Evaluation setup and toolset. From the literature,the typical approach is first to test the quality ofthe POMDP-based dialogue policy with a simulateduser. The real-user evaluation is considered at the1For example, the states which the user’s goal feature is aband the user’s action feature is cfinal step. An advantage of modeling dialogue as aPOMDP is that we can use the POMDP environmentmodel itself as a simulated user model. The prob-ability distributions of the simulated user (testingmodel) might be varied with the ones of the dialoguemanager (training model). The probability distribu-tions of all the user models used in our three appli-cations are handcrafted. We have developed a soft-ware toolkit to conduct our experiments, which in-cludes a factored POMDP to flat POMDP translator,and an interactive simulator for both the user and thesystem. The POMDP problem is first solved with aPOMDP solver (we used Perseus (Spaan and Vlas-sis, 2005) and ZMDP (Smith and Simmons, 2005)).The generated alpha file is then used to carry outthe performance test with simulated user models.Section 3 shows our test results on three differentproblems. We conducted a large number of dialogueepisodes (≥ 10, 000) to guarantee the statistical sig-nificance.3 EvaluationRitel QA dialogue system. Ritel (Galibert et al.,2005) is a telephone-based question answering (QA)dialogue system. Dialogue functionality includesconfirmation of key phrases and the type of the an-swer sought, and handling follow-up questions. Inour model, we focus on confirmation, modeling keyphrases and answer type as slots. In the real system,there are thousands of possible key phrases, but an-swer type only has a few possible values. To makeit tractable, we simplified the model to one slot withbetween 3 and 10 values, suitable at least for model-ing answer type fully.The POMDP state space consists of the user goalsand the user actions (S = Gu×Au). The user goalsare the different questions or question types that theuser may ask, Gu = q1, ..., qn. The user actions arecomposed of the questions, plus positive and nega-tive feedback, a ‘bye’ utterance, and a ‘hang-up’ sig-nal, Au = q1, ..., qn, pos, neg, bye, null. The obser-vation set Z is the same as Au. The system actionsconsist of confirming each question, answering it,and the ‘ask’ action, asking an open question to theuser, A = confirmqi , answerqi , ask. When thesystem answers the correct question, the user posesa new question, otherwise the user either repeats orgives negative feedback. The user may hang up inany dialogue turn, with a fixed probability 0.1.We made the reward model as simple as possible:give a reward of 1 for answering the right question,−1 for answering a wrong one, zero otherwise. Wefound that modelling dialogue state was not neces-sary, and it increases state space to intractable levels.This model yields the desired behaviour, though likeWilliams et al., we found that the system starts con-firming even when the user has not yet said anything.This can be remedied by rewarding the ask actionwith a reward slightly more than 0. Note that it isnot necessary to give an explicit penalty for dialoguelength. The problem can be translated as: answeras many questions as possible before the user hangsup. The results of Perseus were not useable, so theexperiments were done with ZMDP only. Conver-gence was good up till nine slot values. We ob-served that, when the ASR error becomes high, 0.7or above, the system actually wants to hear a ques-tion multiple times in a row before answering it. Thepolicy was compared to a hand-crafted policy (fig-ure 1), similar to the actual Ritel policy, which isbased on counting the number of times a particularkeyword was heard. It was optimised to each partic-ular problem by determining the optimal number oftimes a question should be heard before confirma-tion is sufficient, just as the POMDP does.Figure 1: Performance comparison of POMDP andoptimised hand-crafted models for different prob-lem sizes and ASR error rates. The solid line is thePOMDP, the dashed line is the hand-crafted model.For three values, an error more than 0.6 would re-sult in the probability of hearing the wrong questionbeing higher than the right one. For nine values anderror=0.8, no sensible policy could be calculated.ICIS route navigation system. In the ICISproject2, we are developing a multimodal human-2http://www.icis.decis.nl/computer framework for crisis management (Fitri-anie, 2007). A subtask of the system is to assist res-cuers to find a route description to evacuate victimsfrom an unsafe tunnel. This task has been imple-mented as a multimodal route navigation dialoguesystem (Bui et al., 2007).The simplified POMDP for this problem (one slotcase) is represented by S = 〈Gu×Au×Eu×Du〉 =〈{v0, ..., vm} × {v0, ..., vm, yes, no} ×{stress, nostress} × {notstated, stated}〉,A = {ask, confirm-v0, ..., confirm-vm, ok-v0,..., ok-vm, fail}, and Z = 〈OAu × OEu〉 =〈{v0, ..., vm, yes, no} × {stress, nostress}〉.The full flat-POMDP model is composed of(4m2 + 8m + 1) states (including a special endstate), (2m+2) actions, and (2m+4) observations.The transition and observation models are gen-erated from the two time-slices Dynamic DecisionNetwork (Bui et al., 2007). We assume that the ob-served user’s action only depends on the true user’saction (i.e. P (oau|au) = (1 − poa) if oau = au,otherwise P (oau|au) = 1/(m + 1) × poa). Theobserved user’s emotional state is computed in asimilar way. The reward model is defined as fol-lows: if the system confirms when dialogue stateis notstated, the reward is -2, the reward is -5for action fail, the reward is 10 for action ok-xwhere gu = x (x ∈ v0, ..., vm), otherwise the re-ward is -10. The reward for any action taken in theabsorbing end state is 0. The reward for any otheraction is -1.We set different values for parametersm, pe, poa, poe3 and use two POMDP solversPerseus and ZMDP to compute the near-optimalpolicy. Previous research showed that the optimalpolicy depends on the user’s stress level in casepe > 0 and the POMDP policy outperforms hand-crafted policies (Bui et al., 2007). The size of thestate space of POMDP model increases as the squareof slot numbers and computing the optimal policyis not possible when the number of slot values isgreater than 30 because the POMDP parameter filesize rapidly increases (for example with m = 30,the size is bigger than 200MB). Therefore, thePOMDP solver got stuck in initializing the problem.3pe is the probability of the user’s action error being inducedby stress. poa and poe are the probabilities of the observeduser’s action and observed user’s stress errors.An alternative solution is to use DDN-POMDP (Buiet al., 2007) or summary POMDP (Williams andYoung, 2007). However, when the number of slotvalues is greater than 100, the belief update task isnot tractable. Therefore, a further research on thePOMDP problem representation is necessary. Apractical issue is that ZMDP is more suitable forthe more complex problem (10 ≤ m ≤ 30). Thisis because ZMDP is able to handle a larger statespace by more effective use of sparsity (Smith andSimmons, 2005). On the other hand, Perseus solvessmall problems very well. The reason for this wasnot theoretically indicated in the Perseus paper, butthey found the same result when testing with thestandard POMDP problems from the literature.Virtual Guide application. The Virtual Guide isa character in a Virtual Reality model of the MusicCentre in Enschede (Hofs et al., 2003). The charac-ter can help users find their way in the building. Itencompasses a multimodal dialogue system that al-lows users to refer to locations and objects with spo-ken or written language or by pointing at a locationon a map. The system uses clarification questionsand implicit confirmations. The user can continue adialogue with follow-up questions.It is currently impossible to create a tractablePOMDP model for the system. In our simplifiedmodels the user can only ask for the route betweentwo objects, and the world is limited to three or eightobjects. Moreover we made a closed model wherefollow-up questions are not allowed. We have fit theproblem into a POMDP dialogue model of Williams.A reward is given when the system gives the correctroute and the user provided both locations.Evaluations were performed for four models. Foreach of them we compared the solutions of Perseusand ZMDP and an adapted hand-crafted system. Thesolvers were run for ten minutes, when convergencewas usually slowing down, although it had not al-ways reached a desirable level. We then ran dia-logues with an automatic user simulation based onthe user model of the POMDP.The first model stops after giving any answer, hasobservation error 0.2, and three locations. We var-ied the observation error of the simulator. The re-sults in figure 2 show that with increasing errors thePOMDP solutions produced higher returns than thehand-crafted system.Figure 2: Average returns for simulation with differ-ent observation errorsFor the second model, we increased the observa-tion error to 0.6. The Perseus solution contained astate from which the dialogue never ended. ZMDPdid not converge acceptably. Therefore its solutionperformed worse than the hand-crafted system.The third model has observation error 0.2 again,but the dialogue only stops after giving a correct an-swer. The average returns for Perseus, ZMDP andthe hand-crafted system were 8.08, 6.84 and 6.69(higher than the first model, because of a reward foran extra system action).In the last model we increased the number of lo-cations to eight, resulting in 729 POMDP states in-stead of about 80. Perseus was not able to load thisproblem. The average returns obtained with ZMDPand the hand-crafted system were 5.08 and 4.04.4 ConclusionsAlthough our experiments indicate that POMDP-based dialogue systems can perform better thanhand-crafted ones, we identified several problemswith modelling them. One of the major problemsremains tractability. It is not possible to obtain use-ful solutions for any but strongly simplified models,which may bear little relation to the original prob-lem. For example, when reducing the number ofslot values, the strategy of trying them one by onecan be employed, something that may not have beenfeasible for the original number of values. Anotherexample was the need to simplify an open model,where the end of a dialogue is determined by theuser, to a closed model.The definition of a good reward model is anotherhard problem. While the reward model models psy-chological factors such as user satisfaction, whichcannot easily be quantified precisely, the POMDPsproved very sensitive to small changes in the rewardmodel, in particular the relative magnitude of differ-ent types of reward. In practice we had to experi-ment with different reward values.The POMDP policies sometimes came up withsurprising strategies. For example, some policiesdecided to confirm multiple times in a row, some-thing which our original hand-crafted models didnot. We could significantly improve performance ofthe hand-crafted policies by adapting them accord-ing to the strategies found by the POMDP policies.This shows how POMDPs could be used to improvehand-crafted systems.Acknowledgements. This work is part of the ICISand IMIX programs. ICIS is sponsored by the Dutchgovernment under contract BSIK 03024. IMIX isfunded by the Netherlands Organization for Scien-tific Research (NWO).ReferencesT.H. Bui, M. Poel, A. Nijholt, and J. Zwiers. 2007. Atractable ddn-pomdp approach to affective dialoguemodeling for general probabilistic frame-based dia-logue systems. In Proc. of the 5th Workshop onKnowledge and Reasoning in Practical Dialogue Sys-tems, pages 34–37.S. Fitrianie. 2007. A multimodal human-computer inter-action framework for research into crisis management.In Proc. of the Intelligent Human Computer Systemsfor Crisis Response and Management, pages 149–158.O. Galibert, G. Illouz, and S. Rosset. 2005. Ritel: anopen-domain, human-computer dialog system. In In-terspeech 2005, pages 909–912.D. Hofs, R. op den Akker, and A. Nijholt. 2003. Ageneric architecture and dialogue model for multi-modal interaction. In Proc. of the 1st Nordic Sympo-sium on Multimodal Communication, pages 79–92.Trey Smith and Reid Simmons. 2005. Point-basedpomdp algorithms: Improved analysis and implemen-tation. In Proc. of the Conference on Uncertainty inArtificial Intelligence, pages 542–549.M.T.J. Spaan and N. Vlassis. 2005. Perseus: Random-ized point-based value iteration for pomdps. Journalof Artificial Intelligence Research, 24:195–220.J.D. Williams and S. Young. 2007. Partially observablemarkov decision processes for spoken dialogue sys-tems. Computer Speech and Language, 21:393–422.

Practical Dialogue Manager Development using POMDPs

https://ris.utwente.nl/ws/files/5371719/pomdp-development.pdf

Practical Dialogue Manager Development using POMDPs

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information