Oliehoek, F.A.

Spaan, M.T.J.

Whiteson, S.

English

Dec-POMDPs are a powerful framework for planning in multiagent systems, but are provably intractable to solve. This paper proposes a factored forward-sweep policy computation method that tackles the stages of the problem one by one, exploiting weakly coupled structure at each of these stages. An empirical evaluation shows that the loss in solution quality due to these approximations is small and that the proposed method achieves unprecedented scalability, solving Dec-POMDPs with hundreds of agents

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Approximate Solutions for Factored Dec-POMDPs with Many AgentsOliehoek, F.A.; Whiteson, S.; Spaan, M.T.J.Publication date2013Document VersionFinal published versionPublished inBNAICLink to publicationCitation for published version (APA):Oliehoek, F. A., Whiteson, S., & Spaan, M. T. J. (2013). Approximate Solutions for FactoredDec-POMDPs with Many Agents. BNAIC, 25, 340-341.http://bnaic2013.tudelft.nl/proceedings/papers/paper_31.pdfGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:09 Mar 2023Approximate Solutions for Factored Dec-POMDPswith Many Agents1Frans A. Oliehoek a Shimon Whiteson b Matthijs T.J. Spaan ca Maastricht University. P.O.Box 616, 6200 MD Maastrichtb University of Amsterdam. Science Park 904, 1098 XH Amsterdamc Delft University of Technology. Mekelweg 4, 2628 CD DelftAbstractDec-POMDPs are a powerful framework for planning in multiagent systems, but are provably intractableto solve. This paper proposes a factored forward-sweep policy computation method that tackles the stagesof the problem one by one, exploiting weakly coupled structure at each of these stages. An empiricalevaluation shows that the loss in solution quality due to these approximations is small and that the proposedmethod achieves unprecedented scalability, solving Dec-POMDPs with hundreds of agents.1 IntroductionCooperative multiagent systems are of significant scientific interest, not only because they can tackle dis-tributed problems, but also because they facilitate the decomposition of problems too complex to be tackledby a single agent. As a result, a fundamental question in artificial intelligence is: how should teams of agentsact so as to most effectively achieve common goals? When uncertainty and many agents are involved, thisquestion is challenging, and has not yet been answered in a satisfactory way. Even single-agent decisonmaking is complicated by incomplete knowledge of the environment’s state (e.g., due to noisy sensors).However, multiagent settings exacerbate this problem, as agents have access only to their own sensors.This paper focuses on the finite-horizon decentralized partially observable Markov decision process(Dec-POMDP) [1], a model that can represent such problems under uncertainty. We consider factored Dec-POMDPs, which—by explicitly representing the structure present in the problem—open the door to methodsthat exploit this structure and thus scale to many more agents. While there have been positive results forspecific subclasses that restrict the type of structure, e.g., [7], so far only moderate scalability has beenachieved for general Dec-POMDPs. Since even finding an ε-approximate solution is NEXP-complete, anymethod that is guaranteed to be computationally efficient cannot guarantee an absolute bound on the error.Therefore, in this paper, we abandon optimality guarantees and aim for scalability in the number of agents.2 ApproachThis paper proposes an approximate planning method for factored Dec-POMDPs. These are problems suchas FireFightingGraph, illustrated in Fig. 1(a), where the state consists of multiple variables and each agentcan only affect a subset of these variables directly. Additionally each agent only gets local observations thatdepend on a local subset of variables. The goal is to maximize the sum of local rewards over time.1Full version appeared in the Proc. of the Twelfth International Conference on Autonomous Agents and Multiagent Systems [6].h4 h1h2h3a1h1'h2'h3'h4'o1a2 o2a3 o3t t+1R1Expected reward for house 11 2 3 4(a) The FireFightingGraph benchmark.a1h1h2h3h4a2a3a1h1h2h3h4o1o2o3a2a3a1h1h2h3h4o1o2o3a2a3t=0 t=1 t=2Q3Q21 2 32 3 4(b) Decompostion into smaller source problems. (c) Scalability w.r.t. agents.We propose an approach based on forward-sweep policy computation (FSPC), which approximates theDec-POMDP by a series of one-shot Bayesian games (BGs), one for each stage [3]. Our method, called fac-tored FSPC (FFSPC), exploits weakly coupled structure at each stage by replacing the BGs with collabora-tive graphical BGs (CGBGs) [4]. The main algorithmic contribution of the paper is a set of approximationsnecessary to make FFSPC feasible for problems with many agents:1. First, we approximate the interaction structure between agents by constructing CGBGs with a prede-fined factorization. E.g., as illustrated by the rectangles in the left part of Fig. 1(b).2. Second, instead of following the common practice of solving the underlying (PO)MDP, we employ anew class of heuristics based on transfer planning that directly approximate the factored Dec-POMDPvalue function. This works by defining source problems for each component and transferring the valuefunctions to the original target problem (also shown in Fig. 1(b)).3. Third, we use approximate inference [2] to efficiently construct the CGBGs.4. Finally, we approximately solve the CGBGs by applying Max-Sum to their agent and type indepen-dence factor graphs, an approach that has shown state-of-the-art results [5].An extensive empirical evaluation that shows that FFSPC is highly scalable with respect to the number ofagents (see Fig. 1(c)), while attaining (near-) optimal values. In particular, our method shows scalabilityto hundreds of agents which is unprecedented for the class of factored Dec-POMDPs that we consider;previous approaches applicable to these problems have not been demonstrated beyond 20 agents. Moreover,a detailed analysis of our approximation indicates that there is no significant decrease in value due to sparsefactorization and approximate CGBG construction, and that the transfer planning heuristics significantlyoutperform two baselines. While this works successfully scales to large number of agents, future workshould concentrate on combining this scalability with the ability to tackle longer planning horizons [3].References[1] Daniel S. Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralizedcontrol of Markov decision processes. Math. of OR, 27(4):819–840, 2002.[2] Kevin P. Murphy and Yair Weiss. The factored frontier algorithm for approximate inference in DBNs. In UAI, 2001.[3] Frans A. Oliehoek, Matthijs T. J. Spaan, Christopher Amato, and Shimon Whiteson. Incremental clustering andexpansion for faster optimal planning in decentralized POMDPs. JAIR, 46:449–509, 2013.[4] Frans A. Oliehoek, Matthijs T. J. Spaan, Shimon Whiteson, and Nikos Vlassis. Exploiting locality of interaction infactored Dec-POMDPs. In AAMAS, 2008.[5] Frans A. Oliehoek, Shimon Whiteson, and Matthijs T. J. Spaan. Exploiting structure in cooperative Bayesian games.In UAI, 2012.[6] Frans A. Oliehoek, Shimon Whiteson, and Matthijs T. J. Spaan. Approximate solutions for factored Dec-POMDPswith many agents. In AAMAS, pages 563–570, 2013.[7] Stefan J. Witwicki and Edmund H. Durfee. Influence-based policy abstraction for weakly-coupled Dec-POMDPs.In ICAPS, 2010.

Approximate Solutions for Factored Dec-POMDPs with Many Agents

https://pure.uva.nl/ws/files/1539488/167015_paper_31.pdf

Approximate Solutions for Factored Dec-POMDPs with Many Agents

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications