KUMAR, Akshat

ZILBERSTEIN, Shlomo

Institutional Knowledge at Singapore Management University

Singapore Management UniversityInstitutional Knowledge at Singapore Management UniversityResearch Collection School Of Information Systems School of Information Systems5-2011Message-Passing Algorithms for Large StructuredDecentralized POMDPsAkshat KUMARSingapore Management University, akshatkumarR@smu.edu.sgShlomo ZILBERSTEINUniversity of Massachusetts AmherstFollow this and additional works at: https://ink.library.smu.edu.sg/sis_researchPart of the Artificial Intelligence and Robotics Commons, and the Operations Research, SystemsEngineering and Industrial Engineering CommonsThis Conference Proceeding Article is brought to you for free and open access by the School of Information Systems at Institutional Knowledge atSingapore Management University. It has been accepted for inclusion in Research Collection School Of Information Systems by an authorizedadministrator of Institutional Knowledge at Singapore Management University. For more information, please email libIR@smu.edu.sg.CitationKUMAR, Akshat and ZILBERSTEIN, Shlomo. Message-Passing Algorithms for Large Structured Decentralized POMDPs. (2011).AAMAS '11: The 10th International Conference on Autonomous Agents and Multiagent Systems, Taipei, Taiwan, May 2-6. 1087-1088.Research Collection School Of Information Systems.Available at: https://ink.library.smu.edu.sg/sis_research/2207Message-Passing Algorithms for Large StructuredDecentralized POMDPs(Extended Abstract)Akshat KumarDepartment of Computer ScienceUniversity of Massachusetts, Amherstakshat@cs.umass.eduShlomo ZIlbersteinDepartment of Computer ScienceUniversity of Massachusetts, Amherstshlomo@cs.umass.eduABSTRACTDecentralized POMDPs provide a rigorous framework formulti-agent decision-theoretic planning. However, their highcomplexity has limited scalability. In this work, we presenta promising new class of algorithms based on probabilis-tic inference for infinite-horizon ND-POMDPs—a restrictedDec-POMDP model. We first transform the policy opti-mization problem to that of likelihood maximization in amixture of dynamic Bayes nets (DBNs). We then developthe Expectation-Maximization (EM) algorithm for maximiz-ing the likelihood in this representation. The EM algorithmfor ND-POMDPs lends itself naturally to a simple message-passing paradigm guided by the agent interaction graph. Itis thus highly scalable w.r.t. the number of agents, can beeasily parallelized, and produces good quality solutions.Categories and Subject DescriptorsI.2 [Artificial Intelligence]: Multiagent systemsGeneral TermsAlgorithms, TheoryKeywordsAgent Reasoning: Planning (single and multiagent)1. INTRODUCTIONDecentralized partially observable MDPs (Dec-POMDPs)have emerged in recent years as an important frameworkfor sequential multi-agent planning under uncertainty [2].Their expressive power allows them to capture situationswhen agents must act based on different partial informationabout the environment and about each other to maximizea global objective function. Many problems such as multi-robot coordination [1], broadcast channel protocols [2] andtarget tracking by a team of sensor agents [7] can be modeledas a Dec-POMDP. However, their NEXP-Complexity evenfor two agents has limited their scalability.To counter such scalability issues, an emerging paradigmis to consider restricted forms of interaction among agentsCite as: Message-Passing Algorithms for Large Structured Decentral-ized POMDPs (Extended Abstract), Akshat Kumar and Shlomo Zilberstein,Proc. of 10th Int. Conf. on Autonomous Agents and Multia-gent Systems (AAMAS 2011), Tumer, Yolum, Sonenberg and Stone(eds.), May, 2–6, 2011, Taipei, Taiwan, pp. XXX-XXX.Copyright c© 2011, International Foundation for Autonomous Agents andMultiagent Systems (www.ifaamas.org). All rights reserved.that arise frequently in practice [1, 7]. In particular, we tar-get the Network-Distributed POMDP (ND-POMDP) modelthat is inspired by the realistic problem of coordinating tar-get tracking sensors [6, 7]. The key assumptions in thismodel are that of conditional transition independence andconditional observation independence along with factoredimmediate rewards. We aim to solve infinite-horizon ND-POMDPs using stochastic, finite-state controllers to repre-sent policies. To the best of our knowledge, our work is thefirst approach to tackle infinite-horizon ND-POMDPs, andthe first to solve such problems with 20 agents. We presenta promising new class of algorithms, which combines decen-tralized planning with probabilistic inference. Our work isbased on recently developed techniques for planning underuncertainty using probabilistic inference [8, 5].The expectation-maximization algorithm we develop forND-POMDPs lends itself naturally to a simple message pass-ing implementation based on the agent interaction graph.In each iteration of EM, an agent only needs to exchangemessages with its immediate neighbors. The complexity ofcomputing and propagating such messages is linear in thenumber of links in the agent interaction graph. Thus EMis highly scalable w.r.t. the number of agents allowing us tosolve a 20-agent problem. Furthermore, using the DBN rep-resentation, we efficiently exploit the highly factored stateand action spaces of the ND-POMDP model, allowing us tosolve large problems which are highly intractable when us-ing a flat representation. To test the scalability of the EM,we also design new benchmarks that are much larger thanthe existing ND-POMDP instances. Empirically, EM pro-vides good solution quality when compared against randomcontrollers and a loose upper bound.2. THE ND-POMDP MODELThe ND-POMDP model is motivated by target trackingapplications such as the one illustrated in Fig. 1. This ex-ample includes a sensor network with 5 camera sensors (oragents). For details, we refer to [3]. In our work, sensorsalso have an internal state, which indicates battery level.Each action consumed some power. Sensors could rechargeat some cost and save battery power by being idle.2.1 Policy evaluation in ND-POMDPsWe present two new results regarding policy evaluationin infinite-horizon ND-POMDPs. The stationary policy ofeach agent is represented using a fixed size, stochastic finite-state controller (FSC). An FSC for agent i is described by a!" #"$"%"&"(a) Camera sensor network!" #" $"%" &"(b) Interaction graphFigure 1: Targets T1 and T2 follow dotted trajectories.tuple 〈Q, pi, λ, ν〉. Q denotes a set of controller nodes q. pi :Q×Si → ∆Ai denotes the stochastic action selection policy,i.e., piaiqsi = P (ai|q, si). λ : Q × Yi → ∆Q represents thestochastic node transition model, i.e., λq′qyi =P (q′|q, yi). ν :Q→ ∆Q denotes the initial distribution over the controllernodes, i.e., νq = P (q).Theorem 1. The value of starting the joint controller inthe configuration q in the joint-state s is factored and addi-tive along the links l, that is, V (q, s) =∑l∑alpialqlsl ·{Rl(su, sl, al) + γ∑s′l,s′u,ylpupl∑q′lλq′lqlylVl(q′l, s′l, s′u)}We can further use the fact that the external state-spaceSu is factored; in the sensor network example each factorcorresponds to a location of a target.Theorem 2. Let the external state-space Su be factoredas St1 × . . .× Stm with each state-factor having its own in-dependent transition function. Let the immediate reward Rland the transition and observation probabilities of all theagents on a link l involve at most the state factors Stl ⊆ Su,then the policy value along a link l satisfies:Vl(ql, sl, su) = Vl(ql, sl, stl) s.t. su ∈ Su , stl ∈ Stl3. EM ALGORITHM FOR ND-POMDPSAlgorithm 1 shows the message-passing implementationof EM. Messages are exchanged locally among immediateneighbors in the interaction graph. The function fij([aqs]j)is defined for each edge (i, j) of the interaction graph andboth the agents i and j of this edge. The argument of thisfunction, [aqs]j , represents a specific action a, controllernode q and internal state s of the agent j. The functionis given by the following probabilistic inference in the DBNmixture corresponding to the edge (i, j):f(a, q, s) =∞∑T=0P (T )T∑t=0Pt(rˆ=1, a, q, s|L, T ; θ). (1)where rˆ is the auxiliary reward variable as introduced in [5].This inference can be implemented using a message-passingparadigm as in [8, 5], which makes EM highly scalable withthe number of agents. EM also offers a great potential forparallelization. All the messages in EM for each link canbe computed in parallel leading to a significant speedupwhen using massively parallel computing platforms, such asGoogle’s MapReduce. This further highlights the scalabilityof EM for large multiagent planning benchmarks.We experimented on several sensor network benchmarksfrom [7, 3]. In addition, we also used a 20-agent benchmarkAlgorithm 1: Message-Passing for ND-POMDPsInitialize parameters pi[aqs]i randomly for each agent i1for iter = 1 until MaxIter do2for Agent i = 1 until n do3for each agent j ∈ Ne(i) do4Compute fij([aqs]j) for each [aqs]j5Send message µi→j = fij to agent j6end7end8for Agent i = 1 until n do9Receive all messages µj→i from j ∈ Ne(i)10Set pi?aqs =1Cqs∑j∈Ne(i) µj→i([aqs])11end12Set pi[aqs]i ← pi?[aqs]i for each agent i13end14from [4]. For all these problem, EM converged quickly, of-ten within 200 iterations. When compared against randomcontrollers, EM provided significantly better solution qual-ity. Against a loosely computed upper bound, EM providea solution within 45% of the bound.4. CONCLUSIONWe developed a new approach for solving infinite-horizonND-POMDPs using probabilistic inference in a mixture ofdynamic Bayes nets. We then derived the EM algorithm foriteratively improving the policy. The resulting algorithm canbe easily implemented using local message passing amongthe agents. Each message can be computed efficiently andinvolves only the parameters of agents connected to a singleinteraction link, making this message passing scheme par-ticularly scalable w.r.t. the number of agents and links inthe interaction graph. Another practical advantage of EMis that it naturally lends itself to parallelization; our exper-iments on a multi-core machine showed linear speedup.AcknowledgmentsSupport for this work was provided in part by the NSF GrantNo. IIS-0812149 and AFOSR Grant No. FA9550-08-1-0181.5. REFERENCES[1] R. Becker, S. Zilberstein, V. Lesser, and C. V. Goldman.Solving transition independent decentralized markovdecision processes. JAIR, 22:423–455, 2004.[2] D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein.The complexity of decentralized control of Markov decisionprocesses. J. MOR, 27:819–840, 2002.[3] A. Kumar and S. Zilberstein. Constraint-based dynamicprogramming for decentralized pomdps with structuredinteractions. In AAMAS, pages 561–568, 2009.[4] A. Kumar and S. Zilberstein. Event-detecting multi-agentmdps: complexity and constant-factor approximation. InIJCAI, pages 201–207, 2009.[5] A. Kumar and S. Zilberstein. Anytime planning fordecentralized POMDPs using expectation maximization. InUncertainty in Artificial Intelligence, pages 294–301, 2010.[6] V. Lesser, M. Tambe, and C. L. Ortiz, editors. DistributedSensor Networks: A Multiagent Perspective. KluwerAcademic Publishers, Norwell, MA, USA, 2003.[7] R. Nair, P. Varakantham, M. Tambe, and M. Yokoo.Networked distributed POMDPs: A synthesis of distributedconstraint optimization and POMDPs. In AAAI, pages133–139, 2005.[8] M. Toussaint, S. Harmeling, and A. Storkey. Probabilisticinference for solving (PO)MDPs. Technical ReportEDIINF-RR-0934, University of Edinburgh, 2006.

Message-Passing Algorithms for Large Structured Decentralized POMDPs

https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=3207&amp;context=sis_research

Message-Passing Algorithms for Large Structured Decentralized POMDPs

Abstract

Similar works

Full text

Available Versions

Institutional Knowledge at Singapore Management University