The software suite we will demonstrate at AAAI ’06 was designed around planning with factored Markov decision processes (MDPs). It is a user-friendly suite that facilitates domain elicitation, preference elicitation, planning, and MDP policy display. The demo will concentrate on user interactions for domain experts and those for whom plans are made

Cornett, Austin

Dekhtyar, Alex

Goldsmith, Judy

Lengacher, Casey

Mathias, Krol Kevin

Williams, Derek

English

DigitalCommons@CalPoly

Factored MDP Elicitation and Plan DisplayKrol Kevin Mathias, Casey Lengacher, Derek Williams, Austin CornettAlex Dekhtyar and Judy Goldsmith{kevin9,clleng0,dwwill0,amcorn3}@uky.edu, {dekhtyar,goldsmit}@cs.uky.eduDepartment of Computer ScienceUniversity of Kentucky, Lexington, KYAbstractThe software suite we will demonstrate at AAAI ’06 was de-signed around planning with factored Markov decision pro-cesses (MDPs). It is a user-friendly suite that facilitates do-main elicitation, preference elicitation, planning, and MDPpolicy display. The demo will concentrate on user interac-tions for domain experts and those for whom plans are made.IntroductionThere are two aspects of our software suite that are, as sepa-rate components, truly novel. The first is the plan displayerthat allows a user to walk through possible state/action se-quences using the plan’s actions or others. We know of noother displayer of factored Markov decision process policiesthat are designed with a nontechnical user in mind, yet coverand convey the multitude of possible futures inherent in sucha plan.The second truly novel contribution is the new “shape”for Bayesian network fragment elicitation, called the bowtie(Mathias et al. 2006). These fragments explicitly representan action’s success as a stochastic function of its predictors,and explicitly represent the effects of the action’s success.The other major contribution, which is the main thrust ofthe demo, is the integration of multiple functionalities forboth user and domain expert. The user can specify pref-erences, walk through plans, and move with ease betweenthese modes. Furthermore, if she has domain-expert permis-sions, she can edit the preference or planning domains. Allof the elicitation functions, both preference and domain, areavailable qualitatively, although the models are all convertedto quantitative representations. The die-hard Bayesian can,of course, input her model as a quantitative Bayesian net-work.We have developed a software suite to accompanystochastic planners (Figure 1). The software suite containselicitation tools, databases, and display tools.The Domain Instance Elicitation tool (DIET) helps adomain expert describe the possible preference attributesin their domain. DIET elicits preference variables and“archetypes,” which are hand-built templates of preferencesmade up of different attributes and their values, that areCopyright c© 2006, American Association for Artificial Intelli-gence (www.aaai.org). All rights reserved.used by the Preference Elicitation tool (POET). A user pro-vides her preferences to POET by adjusting values for theattributes of her selected archetype. The Bayes Net Buildertool (BNB) helps a domain expert build Bayesian networkdomain models. The Probability Elicitation tool (PET) as-sists a domain expert in inputting conditional probabilitiesfor nodes in the Bayesian network. The BNB and PETtogether provide a quantitative method for domain elicita-tion. The High Level Elicitor (HELL) (Mathias et al. 2006),developed for our welfare-to-work application, provides aqualitative method for modelling an action as a “bowtie frag-ment” (see Figure 2) instead of a dynamic Bayesian network(DBN) representation of factored MDPs (Boutilier, Dean, &Hanks 1999). A factored-MDP planner uses the informationprovided by the elicitation tools to generate a plan. The planis provided as input to PlanIt, the plan displayer (Williams2006). PlanIt allows a user to step through the plan, as de-scribed in the next section.PLANNERPOET     HELLPET        BNB      DIETELICITATION TOOLS DISPLAY TOOLSPLANITFigure 1: The planning suite.Overview of the DemoIn our demonstration we will provide AAAI/IAAI partici-pants with advice on attending sessions at the conference.In particular, we consider the following MDP model. Theactions are attending a particular parallel session or taking awalk instead. The attendee’s utility is expressed in terms oftopics covered by the talks she hears, and the times of day.We assume that a talk covers topics K1, . . . ,Kj stochas-tically. The probability of a session covering a topic is com-puted according to the topics listed for the talks in that ses-sion.The state variables represent the sessions (Si), the satura-tion for each topic (Kj), and the time (Tk). At time T = t,only finitely many actions are enabled; those actions rely onTt−1 having been incremented by the plan. (Any action attime t has as one of its effects that Tt changes value from 0to 1.) There is a variable, “coverage”, that depends on thenumber of topic variables that reach saturation.The attendee expresses interest levels (capacities) in top-ics. The attendees also express different habitual prefer-ences. Thus, an attendee chooses an “archetype” prefer-ence description, and then edits the suggested preferences asneeded. For instance, the archetype SOCIAL might specify apreference for taking walks during all the morning sessions,and on coverage being high. The archetype MODELLINGmight put high preferences on topics such as preferences,knowledge elicitation, and Bayesian networks.The planning domain assumes a fixed probability that theuser will leave any session before saturation occurs (i.e.,the action of attending that session fails); assumes a priorithat she will attend nearby talks with higher probability thantalks that are farther away.The planner uses factored MDPs. A Markov decision pro-cess is a 4-tuple M = 〈S,A, t, r〉 where S is a finite set ofstates, A a finite set of actions, t : S × A× S → [0, 1], andr : S → R. The function t(s, a, s′) specifies the probabilityof ending up in state s′, given that you take action a in states. The function r(s) gives the (positive or negative) rewardfor being in state s. A factored MDP has a state space ex-pressed as the product of finitely many variables, so a statecorresponds to a setting for each variable.Each action in the planning domain is modelled as abowtie. For example, the action “attending a session” hasa probability of success associated with it. The probabilityof success is determined by the session characteristics andthe attendee’s state. If the action is successful, there is astochastic increase in coverage of topics. On failure of theaction, there is no change in the coverage of topics.Figure 2: The bowtie model for action “Take a nap”.The bowtie model consists of a set of input nodes linkedto a central action success node. The success node, in re-turn, is linked to the set of output nodes. The bowtie modelis similar to the QPN model(Wellman 1990) except that inthe bowtie model, a weight is associated with each of theinfluence factors.The user will walk through our preference elicitation pro-cess by choosing a preference archetype and then editing thesuggested preferences as needed in POET. If the user is sat-isfied with her preferences, she can request a plan. Oncethe plan is computed, the user can step through her plan inPlanIt. At each step, she can see a representation of her cur-rent state, a set of boxes representing different outcomes,and the most likely outcome. The boxes are sized accordingto probability and colored according to her specified utility.She can click on a box to continue the walkthrough, or canchoose a different, unrecommended action to explore.If she is not happy with the suggested plan, she has manyoptions: to update her preferences and request a replan, orto assume the role of domain expert and edit either the pref-erence domain or the planning domain.If she chooses to edit the planning domain, for instance byadding a “take a nap” action (Figure 2), we have availablethe BNB and PET tools for entering or editing dependenciesand conditional probabilities for an action. She can also optfor building a qualitative Bayesian network fragments usingthe HELL tool. This allows her to specify predictors of anaction’s success and their relevance (“weight”) and the at-tributes that are likely to change as a result of the action’ssuccess or failure, and the direction of that change (positiveor negative—it is assumed that all domains are ranked). Forinstance, the success of “take a nap” is likely affected by theattendee’s level of tiredness, and by her overwhelmedness,as measured by the “coverage” variable. If she sleeps, theninformation from the talks may be transferred from short-term to long-term memory, thus probabilistically increasingany non-zero Ki values. HELL uses this qualitative elicitedinformation to compute a consistent quantitative dynamicBayesian network.In summary, ours is the only integrated tool for domainelicitation, Bayesian network management, planning withfactored MDPs, preference elicitation, and plan display thatis designed to be accessible to noncomputer-scientist usersand domain experts. The domain elicitation has as an op-tion of a qualitative-to-quantitative process based on a highlyintuitive and accessible model of action effects called thebowtie model. The plan displayer makes use of an inno-vative method of walking a user through a plan with manypossible outcomes at each step.ReferencesBoutilier, C.; Dean, T.; and Hanks, S. 1999. Decision-theoretic planning: Structural assumptions and computa-tional leverage. J. Artif. Intell. Res. (JAIR) 11:1–94.Mathias, K. K.; Isenhour, C.; Dekhtyar, A.; Goldsmith, J.;and Goldstein, B. 2006. Eliciting and combining influ-ence diagrams: Tying many bowties together. TechnicalReport 453-06, University of Kentucky Dept. of ComputerScience.Wellman, M. P. 1990. Fundamental concepts of qualitativeprobabilistic networks. Artif. Intell. 44(3):257–303.Williams, D. 2006. Interactive preference elicitation andplan visualization. Master’s thesis, University of Kentucky.

Factored MDP Elicitation and Plan Display

https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1145&amp;context=csse_fac

Factored MDP Elicitation and Plan Display

Abstract

Similar works

Full text

Available Versions

DigitalCommons@CalPoly