An Abstract Framework for Non-Cooperative Multi-Agent Planning

[EN] In non-cooperative multi-agent planning environments, it is essential to have a system that enables the agents¿ strategic behavior. It is also important to consider all planning phases, i.e., goal allocation, strategic planning, and plan execution, in order to solve a complete problem. Currently, we have no evidence of the existence of any framework that brings together all these phases for non-cooperative multi-agent planning environments. In this work, an exhaustive study is made to identify existing approaches for the different phases as well as frameworks and different applicable techniques in each phase. Thus, an abstract framework that covers all the necessary phases to solve these types of problems is proposed. In addition, we provide a concrete instantiation of the abstract framework using different techniques to promote all the advantages that the framework can offer. A case study is also carried out to show an illustrative example of how to solve a non-cooperative multi-agent planning problem with the presented framework. This work aims to establish a base on which to implement all the necessary phases using the appropriate technologies in each of them and to solve complex problems in different domains of application for non-cooperative multi-agent planning settings.This work was partially funded by MINECO/FEDER RTI2018-095390-B-C31 project of the Spanish government. Jaume Jordan and Vicent Botti are funded by Universitat Politecnica de Valencia (UPV) PAID-06-18 project. Jaume Jordan is also funded by grant APOSTD/2018/010 of Generalitat Valenciana Fondo Social Europeo.Jordán, J.; Bajo, J.; Botti, V.; Julian Inglada, VJ. (2019). An Abstract Framework for Non-Cooperative Multi-Agent Planning. Applied Sciences. 9(23):1-18. https://doi.org/10.3390/app9235180S118923De Weerdt, M., & Clement, B. (2009). Introduction to planning in multiagent systems. Multiagent and Grid Systems, 5(4), 345-355. doi:10.3233/mgs-2009-0133Dunne, P. E., Kraus, S., Manisterski, E., & Wooldridge, M. (2010). Solving coalitional resource games. Artificial Intelligence, 174(1), 20-50. doi:10.1016/j.artint.2009.09.005Torreño, A., Onaindia, E., Komenda, A., & Štolba, M. (2018). Cooperative Multi-Agent Planning. ACM Computing Surveys, 50(6), 1-32. doi:10.1145/3128584Fikes, R. E., & Nilsson, N. J. (1971). Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3-4), 189-208. doi:10.1016/0004-3702(71)90010-5Hoffmann, J., & Nebel, B. (2001). The FF Planning System: Fast Plan Generation Through Heuristic Search. Journal of Artificial Intelligence Research, 14, 253-302. doi:10.1613/jair.855Dukeman, A., & Adams, J. A. (2017). Hybrid mission planning with coalition formation. Autonomous Agents and Multi-Agent Systems, 31(6), 1424-1466. doi:10.1007/s10458-017-9367-7Hadad, M., Kraus, S., Ben-Arroyo Hartman, I., & Rosenfeld, A. (2013). Group planning with time constraints. Annals of Mathematics and Artificial Intelligence, 69(3), 243-291. doi:10.1007/s10472-013-9363-9Guo, Y., Pan, Q., Sun, Q., Zhao, C., Wang, D., & Feng, M. (2019). Cooperative Game-based Multi-Agent Path Planning with Obstacle Avoidance*. 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE). doi:10.1109/isie.2019.8781205v. Neumann, J. (1928). Zur Theorie der Gesellschaftsspiele. Mathematische Annalen, 100(1), 295-320. doi:10.1007/bf01448847Mookherjee, D., & Sopher, B. (1994). Learning Behavior in an Experimental Matching Pennies Game. Games and Economic Behavior, 7(1), 62-91. doi:10.1006/game.1994.1037Ochs, J. (1995). Games with Unique, Mixed Strategy Equilibria: An Experimental Study. Games and Economic Behavior, 10(1), 202-217. doi:10.1006/game.1995.1030Applegate, C., Elsaesser, C., & Sanborn, J. (1990). An architecture for adversarial planning. IEEE Transactions on Systems, Man, and Cybernetics, 20(1), 186-194. doi:10.1109/21.47820Sailer, F., Buro, M., & Lanctot, M. (2007). Adversarial Planning Through Strategy Simulation. 2007 IEEE Symposium on Computational Intelligence and Games. doi:10.1109/cig.2007.368082Willmott, S., Richardson, J., Bundy, A., & Levine, J. (2001). Applying adversarial planning techniques to Go. Theoretical Computer Science, 252(1-2), 45-82. doi:10.1016/s0304-3975(00)00076-1Nau, D. S., Au, T. C., Ilghami, O., Kuter, U., Murdock, J. W., Wu, D., & Yaman, F. (2003). SHOP2: An HTN Planning System. Journal of Artificial Intelligence Research, 20, 379-404. doi:10.1613/jair.1141Knuth, D. E., & Moore, R. W. (1975). An analysis of alpha-beta pruning. Artificial Intelligence, 6(4), 293-326. doi:10.1016/0004-3702(75)90019-3Vickrey, W. (1961). COUNTERSPECULATION, AUCTIONS, AND COMPETITIVE SEALED TENDERS. The Journal of Finance, 16(1), 8-37. doi:10.1111/j.1540-6261.1961.tb02789.xClarke, E. H. (1971). Multipart pricing of public goods. Public Choice, 11(1), 17-33. doi:10.1007/bf01726210Groves, T. (1973). Incentives in Teams. Econometrica, 41(4), 617. doi:10.2307/1914085Savaux, J., Vion, J., Piechowiak, S., Mandiau, R., Matsui, T., Hirayama, K., … Silaghi, M. (2016). DisCSPs with Privacy Recast as Planning Problems for Self-Interested Agents. 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI). doi:10.1109/wi.2016.0057Buzing, P., Mors, A. ter, Valk, J., & Witteveen, C. (2006). Coordinating Self-interested Planning Agents. Autonomous Agents and Multi-Agent Systems, 12(2), 199-218. doi:10.1007/s10458-005-6104-4Ter Mors, A., & Witteveen, C. (s. f.). Coordinating Non Cooperative Planning Agents: Complexity Results. IEEE/WIC/ACM International Conference on Intelligent Agent Technology. doi:10.1109/iat.2005.60Hrnčíř, J., Rovatsos, M., & Jakob, M. (2015). Ridesharing on Timetabled Transport Services: A Multiagent Planning Approach. Journal of Intelligent Transportation Systems, 19(1), 89-105. doi:10.1080/15472450.2014.941759Galuszka, A., & Swierniak, A. (2009). Planning in Multi-agent Environment Using Strips Representation and Non-cooperative Equilibrium Strategy. Journal of Intelligent and Robotic Systems, 58(3-4), 239-251. doi:10.1007/s10846-009-9364-4Rosenthal, R. W. (1973). A class of games possessing pure-strategy Nash equilibria. International Journal of Game Theory, 2(1), 65-67. doi:10.1007/bf01737559Jordán, J., Torreño, A., de Weerdt, M., & Onaindia, E. (2017). A better-response strategy for self-interested planning agents. Applied Intelligence, 48(4), 1020-1040. doi:10.1007/s10489-017-1046-5Veloso, M., Muñoz-Avila, H., & Bergmann, R. (1996). Case-based planning: selected methods and systems. AI Communications, 9(3), 128-137. doi:10.3233/aic-1996-9305VOORNEVELD, M., BORM, P., VAN MEGEN, F., TIJS, S., & FACCHINI, G. (1999). CONGESTION GAMES AND POTENTIALS RECONSIDERED. International Game Theory Review, 01(03n04), 283-299. doi:10.1142/s0219198999000219Han-Lim Choi, Brunet, L., & How, J. P. (2009). Consensus-Based Decentralized Auctions for Robust Task Allocation. IEEE Transactions on Robotics, 25(4), 912-926. doi:10.1109/tro.2009.2022423Monderer, D., & Shapley, L. S. (1996). Potential Games. Games and Economic Behavior, 14(1), 124-143. doi:10.1006/game.1996.0044Friedman, J. W., & Mezzetti, C. (2001). Learning in Games by Random Sampling. Journal of Economic Theory, 98(1), 55-84. doi:10.1006/jeth.2000.2694Aamodt, A., & Plaza, E. (1994). Case-Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches. AI Communications, 7(1), 39-59. doi:10.3233/aic-1994-7104Bertsekas, D. P. (1988). The auction algorithm: A distributed relaxation method for the assignment problem. Annals of Operations Research, 14(1), 105-123. doi:10.1007/bf02186476Bertsekas, D. P., & Castanon, D. A. (1989). The auction algorithm for the transportation problem. Annals of Operations Research, 20(1), 67-96. doi:10.1007/bf0221692

Inferring the Goals of Communicating Agents from Actions and Instructions

When humans cooperate, they frequently coordinate their activity through both verbal communication and non-verbal actions, using this information to infer a shared goal and plan. How can we model this inferential ability? In this paper, we introduce a model of a cooperative team where one agent, the principal, may communicate natural language instructions about their shared plan to another agent, the assistant, using GPT-3 as a likelihood function for instruction utterances. We then show how a third person observer can infer the team's goal via multi-modal Bayesian inverse planning from actions and instructions, computing the posterior distribution over goals under the assumption that agents will act and communicate rationally to achieve them. We evaluate this approach by comparing it with human goal inferences in a multi-agent gridworld, finding that our model's inferences closely correlate with human judgments (R = 0.96). When compared to inference from actions alone, we also find that instructions lead to more rapid and less uncertain goal inference, highlighting the importance of verbal communication for cooperative agents.Comment: 8 pages, 5 figures. Accepted to the ICML 2023 Workshop on Theory of Mind in Communicating Agents. Supplementary Information: https://osf.io/gh758

Non-Cooperative Games for Self-Interested Planning Agents

Multi-Agent Planning (MAP) is a topic of growing interest that deals with the problem of automated planning in domains where multiple agents plan and act together in a shared environment. In most cases, agents in MAP are cooperative (altruistic) and work together towards a collaborative solution. However, when rational self-interested agents are involved in a MAP task, the ultimate objective is to find a joint plan that accomplishes the agents' local tasks while satisfying their private interests. Among the MAP scenarios that involve self-interested agents, non-cooperative MAP refers to problems where non-strictly competitive agents feature common and conflicting interests. In this setting, conflicts arise when self-interested agents put their plans together and the resulting combination renders some of the plans non-executable, which implies a utility loss for the affected agents. Each participant wishes to execute its plan as it was conceived, but congestion issues and conflicts among the actions of the different plans compel agents to find a coordinated stable solution. Non-cooperative MAP tasks are tackled through non-cooperative games, which aim at finding a stable (equilibrium) joint plan that ensures the agents' plans are executable (by addressing planning conflicts) while accounting for their private interests as much as possible. Although this paradigm reflects many real-life problems, there is a lack of computational approaches to non-cooperative MAP in the literature. This PhD thesis pursues the application of non-cooperative games to solve non-cooperative MAP tasks that feature rational self-interested agents. Each agent calculates a plan that attains its individual planning task, and subsequently, the participants try to execute their plans in a shared environment. We tackle non-cooperative MAP from a twofold perspective. On the one hand, we focus on agents' satisfaction by studying desirable properties of stable solutions, such as optimality and fairness. On the other hand, we look for a combination of MAP and game-theoretic techniques capable of efficiently computing stable joint plans while minimizing the computational complexity of this combined task. Additionally, we consider planning conflicts and congestion issues in the agents' utility functions, which results in a more realistic approach. To the best of our knowledge, this PhD thesis opens up a new research line in non-cooperative MAP and establishes the basic principles to attain the problem of synthesizing stable joint plans for self-interested planning agents through the combination of game theory and automated planning.La Planificación Multi-Agente (PMA) es un tema de creciente interés que trata el problema de la planificación automática en dominios donde múltiples agentes planifican y actúan en un entorno compartido. En la mayoría de casos, los agentes en PMA son cooperativos (altruistas) y trabajan juntos para obtener una solución colaborativa. Sin embargo, cuando los agentes involucrados en una tarea de PMA son racionales y auto-interesados, el objetivo último es obtener un plan conjunto que resuelva las tareas locales de los agentes y satisfaga sus intereses privados. De entre los distintos escenarios de PMA que involucran agentes auto-interesados, la PMA no cooperativa se centra en problemas que presentan un conjunto de agentes no estrictamente competitivos con intereses comunes y conflictivos. En este contexto, pueden surgir conflictos cuando los agentes ponen en común sus planes y la combinación resultante provoca que algunos de estos planes no sean ejecutables, lo que implica una pérdida de utilidad para los agentes afectados. Cada participante desea ejecutar su plan tal como fue concebido, pero las congestiones y conflictos que pueden surgir entre las acciones de los diferentes planes fuerzan a los agentes a obtener una solución estable y coordinada. Las tareas de PMA no cooperativa se abordan a través de juegos no cooperativos, cuyo objetivo es hallar un plan conjunto estable (equilibrio) que asegure que los planes de los agentes sean ejecutables (resolviendo los conflictos de planificación) al tiempo que los agentes satisfacen sus intereses privados en la medida de lo posible. Aunque este paradigma refleja muchos problemas de la vida real, existen pocos enfoques computacionales para PMA no cooperativa en la literatura. Esta tesis doctoral estudia el uso de juegos no cooperativos para resolver tareas de PMA no cooperativa con agentes racionales auto-interesados. Cada agente calcula un plan para su tarea de planificación y posteriormente, los participantes intentan ejecutar sus planes en un entorno compartido. Abordamos la PMA no cooperativa desde una doble perspectiva. Por una parte, nos centramos en la satisfacción de los agentes estudiando las propiedades deseables de soluciones estables, tales como la optimalidad y la justicia. Por otra parte, buscamos una combinación de PMA y técnicas de teoría de juegos capaz de calcular planes conjuntos estables de forma eficiente al tiempo que se minimiza la complejidad computacional de esta tarea combinada. Además, consideramos los conflictos de planificación y congestiones en las funciones de utilidad de los agentes, lo que resulta en un enfoque más realista. Bajo nuestro punto de vista, esta tesis doctoral abre una nueva línea de investigación en PMA no cooperativa y establece los principios básicos para resolver el problema de la generación de planes conjuntos estables para agentes de planificación auto-interesados mediante la combinación de teoría de juegos y planificación automática.La Planificació Multi-Agent (PMA) és un tema de creixent interès que tracta el problema de la planificació automàtica en dominis on múltiples agents planifiquen i actuen en un entorn compartit. En la majoria de casos, els agents en PMA són cooperatius (altruistes) i treballen junts per obtenir una solució col·laborativa. No obstant això, quan els agents involucrats en una tasca de PMA són racionals i auto-interessats, l'objectiu últim és obtenir un pla conjunt que resolgui les tasques locals dels agents i satisfaci els seus interessos privats. D'entre els diferents escenaris de PMA que involucren agents auto-interessats, la PMA no cooperativa se centra en problemes que presenten un conjunt d'agents no estrictament competitius amb interessos comuns i conflictius. En aquest context, poden sorgir conflictes quan els agents posen en comú els seus plans i la combinació resultant provoca que alguns d'aquests plans no siguin executables, el que implica una pèrdua d'utilitat per als agents afectats. Cada participant vol executar el seu pla tal com va ser concebut, però les congestions i conflictes que poden sorgir entre les accions dels diferents plans forcen els agents a obtenir una solució estable i coordinada. Les tasques de PMA no cooperativa s'aborden a través de jocs no cooperatius, en els quals l'objectiu és trobar un pla conjunt estable (equilibri) que asseguri que els plans dels agents siguin executables (resolent els conflictes de planificació) alhora que els agents satisfan els seus interessos privats en la mesura del possible. Encara que aquest paradigma reflecteix molts problemes de la vida real, hi ha pocs enfocaments computacionals per PMA no cooperativa en la literatura. Aquesta tesi doctoral estudia l'ús de jocs no cooperatius per resoldre tasques de PMA no cooperativa amb agents racionals auto-interessats. Cada agent calcula un pla per a la seva tasca de planificació i posteriorment, els participants intenten executar els seus plans en un entorn compartit. Abordem la PMA no cooperativa des d'una doble perspectiva. D'una banda, ens centrem en la satisfacció dels agents estudiant les propietats desitjables de solucions estables, com ara la optimalitat i la justícia. D'altra banda, busquem una combinació de PMA i tècniques de teoria de jocs capaç de calcular plans conjunts estables de forma eficient alhora que es minimitza la complexitat computacional d'aquesta tasca combinada. A més, considerem els conflictes de planificació i congestions en les funcions d'utilitat dels agents, el que resulta en un enfocament més realista. Des del nostre punt de vista, aquesta tesi doctoral obre una nova línia d'investigació en PMA no cooperativa i estableix els principis bàsics per resoldre el problema de la generació de plans conjunts estables per a agents de planificació auto-interessats mitjançant la combinació de teoria de jocs i planificació automàtica.Jordán Prunera, JM. (2017). Non-Cooperative Games for Self-Interested Planning Agents [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/90417TESI

iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning

Navigating safely and efficiently in dense and heterogeneous traffic scenarios is challenging for autonomous vehicles (AVs) due to their inability to infer the behaviors or intentions of nearby drivers. In this work, we introduce a distributed multi-agent reinforcement learning (MARL) algorithm that can predict trajectories and intents in dense and heterogeneous traffic scenarios. Our approach for intent-aware planning, iPLAN, allows agents to infer nearby drivers' intents solely from their local observations. We model two distinct incentives for agents' strategies: Behavioral Incentive for high-level decision-making based on their driving behavior or personality and Instant Incentive for motion planning for collision avoidance based on the current traffic state. Our approach enables agents to infer their opponents' behavior incentives and integrate this inferred information into their decision-making and motion-planning processes. We perform experiments on two simulation environments, Non-Cooperative Navigation and Heterogeneous Highway. In Heterogeneous Highway, results show that, compared with centralized training decentralized execution (CTDE) MARL baselines such as QMIX and MAPPO, our method yields a 4.3% and 38.4% higher episodic reward in mild and chaotic traffic, with 48.1% higher success rate and 80.6% longer survival time in chaotic traffic. We also compare with a decentralized training decentralized execution (DTDE) baseline IPPO and demonstrate a higher episodic reward of 12.7% and 6.3% in mild traffic and chaotic traffic, 25.3% higher success rate, and 13.7% longer survival time

Cooperative Epistemic Multi-Agent Planning for Implicit Coordination

Epistemic planning can be used for decision making in multi-agent situations with distributed knowledge and capabilities. Recently, Dynamic Epistemic Logic (DEL) has been shown to provide a very natural and expressive framework for epistemic planning. We extend the DEL-based epistemic planning framework to include perspective shifts, allowing us to define new notions of sequential and conditional planning with implicit coordination. With these, it is possible to solve planning tasks with joint goals in a decentralized manner without the agents having to negotiate about and commit to a joint policy at plan time. First we define the central planning notions and sketch the implementation of a planning system built on those notions. Afterwards we provide some case studies in order to evaluate the planner empirically and to show that the concept is useful for multi-agent systems in practice.Comment: In Proceedings M4M9 2017, arXiv:1703.0173