22 research outputs found
TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient
Multi-Agent Policy Gradient (MAPG) has made significant progress in recent
years. However, centralized critics in state-of-the-art MAPG methods still face
the centralized-decentralized mismatch (CDM) issue, which means sub-optimal
actions by some agents will affect other agent's policy learning. While using
individual critics for policy updates can avoid this issue, they severely limit
cooperation among agents. To address this issue, we propose an agent topology
framework, which decides whether other agents should be considered in policy
gradient and achieves compromise between facilitating cooperation and
alleviating the CDM issue. The agent topology allows agents to use coalition
utility as learning objective instead of global utility by centralized critics
or local utility by individual critics. To constitute the agent topology,
various models are studied. We propose Topology-based multi-Agent Policy
gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove
the policy improvement theorem for stochastic TAPE and give a theoretical
explanation for the improved cooperation among agents. Experiment results on
several benchmarks show the agent topology is able to facilitate agent
cooperation and alleviate CDM issue respectively to improve performance of
TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm
are devised to show the efficacy of the agent topology
TAPE: leveraging agent topology for cooperative multi-agent policy gradient
Multi-Agent Policy Gradient (MAPG) has made significant progress in recent years. However, centralized critics in state-of-the-art MAPG methods still face the centralized-decentralized mismatch (CDM) issue, which means sub-optimal actions by some agents will affect other agent's policy learning. While using individual critics for policy updates can avoid this issue, they severely limit cooperation among agents. To address this issue, we propose an agent topology framework, which decides whether other agents should be considered in policy gradient and achieves compromise between facilitating cooperation and alleviating the CDM issue. The agent topology allows agents to use coalition utility as learning objective instead of global utility by centralized critics or local utility by individual critics. To constitute the agent topology, various models are studied. We propose Topology-based multi-Agent Policy gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove the policy improvement theorem for stochastic TAPE and give a theoretical explanation for the improved cooperation among agents. Experiment results on several benchmarks show the agent topology is able to facilitate agent cooperation and alleviate CDM issue respectively to improve performance of TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm are devised to show the efficacy of the agent topology