Multi-agent reinforcement learning in mixed-motivesettings allows for the study of complex dynamics ofagent interactions. Embodied agents in partially ob-servable environments with the ability to communicatecan share information, agree on strategies, or even lieto each other.In order to study this, we propose a sim-ple environment where we can impose varying levels ofcooperation, communication and competition as pre-requisites to reach an optimal outcome. Welcome tothe jungle

Alonso, E.

Garcia-Ortiz, M.

Ikram, K.

Mondragon, E.

City Research Online

              City, University of London Institutional RepositoryCitation: Ikram, K., Mondragon, E., Alonso, E. ORCID: 0000-0002-3306-695X and Garcia-Ortiz, M. (2021). HexaJungle: a MARL Simulator to Study the Emergence of Language. Paper presented at the Conference on Computer Vision and Pattern Recognition (CVPR 2021), Embodied AI Workshop, 19-25 Jun 2021, Nashville, Tennessee. This is the accepted version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  https://openaccess.city.ac.uk/id/eprint/26284/Link to published version: Copyright: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.Reuse: Copies of full items can be used for personal research or study, educational, or not-for-profit purposes without prior permission or charge. Provided that the authors, title and full bibliographic details are credited, a hyperlink and/or URL is given for the original metadata page and the content is not changed in any way. City Research OnlineCity Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukHexaJungle: a MARL Simulator to Study the Emergence of LanguageKiran Ikram, Esther Mondragón, Eduardo Alonso, Michael Garcia OrtizArtificial Intelligence Research Centre (CitAI)City, University of LondonLondon EC1V 0HB, UK{ kiran.ikram, e.mondragon, e.alonso, michael.garcia-ortiz } @city.ac.ukAbstractMulti-agent reinforcement learning in mixed-motivesettings allows for the study of complex dynamics ofagent interactions. Embodied agents in partially ob-servable environments with the ability to communicatecan share information, agree on strategies, or even lieto each other.In order to study this, we propose a sim-ple environment where we can impose varying levels ofcooperation, communication and competition as pre-requisites to reach an optimal outcome. Welcome tothe jungle.1. Introduction and MotivationTraining agents to act while accounting for the (po-tentially adversarial) motives of other agents is a fun-damental challenge in AI research. This ties to Theoryof Mind: the intent (communicated or inferred) of oneagent’s influence an others behavior [6]. Learning tocommunicate is imperative to allow agents to share in-tent as well as information, leading to potential agree-ment on collaboration. Most of the prior work at theintersection of MARL[5] and Emergent Communica-tion assumes full cooperation, rarely considering theinclusion selfish motives. In such settings emergence ofcommunication is usually only assessed in relation toan increase in shared global reward value [8],[13][14],-a metric that might not be indicative of emergence ofcommunication [10]. To study emergence of communi-cation between interacting agents, we must also under-stand the conditions under which language evolves [7].Thus we need to be able to test for these conditionsin a consistent manner, and evaluate the effect of theenvironment on the emerged protocol [2].Classical works from Game Theory[1] [11] [4] haveproved that certain incentive structures, and in partic-ular those which fall under the purvey of general sum,do in fact necessitate some form of communication toreach equilibrium . Additionally, [12] show that in themixed-motive setting, the level of required collabora-tion tends to determine the emergence of communica-tion between agents. These works establish a strongrelation between mixed-motive settings and the emer-gence of communication. However, a classical short-coming is their reliance on referential games with sim-plified structured interactions between agents. We takeinspiration from their demonstrations and intuitions topropose a versatile mixed-motive RL environment thatis not only conducive for the emergence of communica-tion [3], but also, allows for fine-grained evaluation ofthe impact of communication on agent behaviour.2. HexaJungleHexaJungle is a suite of environments to test inter-actions and evaluate the emergence of communication.The goal is for two agents to reach a jungle exit. Theoptimal outcome for agents is dependent on predefinedincentive structures that require collaboration or com-munication.The agents live on an hexagonal grid as seen in 1,where they can go from cell to cell by changing theirorientation and moving forward. To emulate realis-tic partial observability, they perceive cells in a 120degree field of view, with a simplified form of occlu-sion. Agents may execute an additional action thatallows them to climb on the shoulders of an otheragent, giving then an unobstructed field of view, andto move towards Boulders (see later). Agents interactwith the cells of the hexagonal grid depending on thetype of elements (detailed therafter, illustrated in 1).Obstacles (dark brown) are non-traversable elements.Trees (dark green) disappear when crossed by an agent,which collects one wood log. Rivers (blue) are not ob-structing the view of the agent, but an agent drownswhen entering a river cell. If both agents enter it at thesame time and have collected enough logs, they build abridge. Boulders (light brown) are obstacles that can1Figure 1. Left:At the top left is the Easy Exit accessible to both. To cross the river(bottom left) agents must collect enoughlogs. Middle:Agents may collaborate to reach the bottom-left RiverExit. In the top-right corner the exit is advantageousto Black, but neither agent has this information at the outset. Right:At the bottom left is an Exit surrounded by boulderswhich is advantageous only to Agent White.Figure 2. Performance: 1. Easy Exit 2. White and Easy Exit 3. River Exit 4. Black and River Exitbe climbed from an adjacent cell, by an agent standingon the shoulders of an other agent. Finally, we pro-pose 4 different kind of Exits, which are essential inshaping the incentive structure. EasyExit terminatesthe episode for the agent entering the cell, providinga low reward. HardExit, instead, provides an aver-age reward. BlackExit provides a reward only to theblack agent, and symmetrically, WhiteExit provides areward only to the white agent (the other agent canexit through these cells, but receive a low reward). Anepisode fully terminates when both agents have exitedor after a time limit.These reward structures allow us to create simpleenvironments for mixed-motive MARL, and thereforetest for the emergence of communication and collabora-tion in an embodied scenario. They allow for physicalinteraction with the world as well as between agents,which can collaborate to gain more information abouttheir surrounding or access areas that they couldn’t ac-cess alone. We illustrate different settings that exhibitthese properties in a sample of environments presentedin 1.3. Experiments and ResultsWe carry out four experiments that reflect varyingincentive structures. Utilizing RL Lib[9] we use PPOwith LSTMS for 800 training episodes.For Experiment(1) we set only an EasyExit, whichmeans that agents neither need to collaborate norcommunicate to get to it.For Experiment(2), we addan additional exit that carries a reward for AgentWhite alone(promoting self-interested behaviour inAgent White), and is surrounded by boulders. Its lo-cation is unknown to either agent until they cooperate.Experiment(3) is meant to test for collaboration. Thereare two exits, EasyExit with low reward for both agentsand RiverExit, with high reward for both agents, iffthey can collaborate to reach the exit together(aftercollecting logs from around the jungle) and build abridge. Experiment(4) adds a higher degree of com-plexity as in addition to the RiverExit, we add an exitthat incentivises Agent Black to behave selfishly.The results as seen in 2 show clearly that it becomesharder for the agents to exit the jungle successfully (a)as the conflict of interest increases and (b) the need forcollaboration increases. What is especially interestingis the results from Experiment(4) which indicate howagents may overcome the limitation posed by a self-ish agent in the group and learn to work together toachieve a common goal.4. ConclusionWe propose a simulator that is not onlywell suited to the emergence of communica-tion, but allows for rigorous evaluation of saidcommunication. Our code can be found at:https://github.com/kiranikram/HexaJungle2References[1] Robert J Aumann. “Subjectivity and correlationin randomized strategies”. In: Journal of mathe-matical Economics 1.1 (1974), pp. 67–96.[2] Alexander I Cowen-Rivers and Jason Narad-owsky. “Emergent Communication with WorldModels”. In: arXiv preprint arXiv:2002.09604(2020).[3] Françoise Forges. “Equilibria with communica-tion in a job market example”. In: The QuarterlyJournal of Economics 105.2 (1990), pp. 375–398.[4] Francoise Forges. “An approach to communica-tion equilibria”. In: Econometrica: Journal of theEconometric Society (1986), pp. 1375–1385.[5] Pablo Hernandez-Leal, Bilal Kartal, andMatthew E Taylor. “A very condensed surveyand critique of multiagent deep reinforcementlearning”. In: Proceedings of the 19th Interna-tional Conference on Autonomous Agents andMultiAgent Systems. 2020, pp. 2146–2148.[6] A Harry Klopf. Brain function and adaptive sys-tems: a heterostatic theory. 133. Air Force Cam-bridge Research Laboratories, Air Force SystemsCommand, United …, 1972.[7] Angeliki Lazaridou and Marco Baroni. “Emer-gent Multi-Agent Communication in theDeep Learning Era”. In: arXiv preprintarXiv:2006.02419 (2020).[8] Angeliki Lazaridou, Anna Potapenko, andOlivier Tieleman. “Multi-agent Communicationmeets Natural Language: Synergies betweenFunctional and Structural Language Learning”.In: arXiv preprint arXiv:2005.07064 (2020).[9] Eric Liang et al. “RLlib: Abstractions for dis-tributed reinforcement learning”. In: Interna-tional Conference on Machine Learning. PMLR.2018, pp. 3053–3062.[10] Ryan Lowe et al. “On the pitfalls of measur-ing emergent communication”. In: arXiv preprintarXiv:1903.05168 (2019).[11] Roger B Myerson. “Multistage games with com-munication”. In: Econometrica: Journal of theEconometric Society (1986), pp. 323–358.[12] Michael Noukhovitch et al. “Emergent Commu-nication under Competition”. In: arXiv preprintarXiv:2101.10276 (2021).[13] Yi Ren et al. “Compositional languages emergein a neural iterated learning model”. In: arXivpreprint arXiv:2002.01365 (2020).[14] Cinjon Resnick et al. “Capacity, bandwidth, andcompositionality in emergent language learning”.In: arXiv preprint arXiv:1910.11424 (2019).3

HexaJungle: a MARL Simulator to Study the Emergence of Language

https://openaccess.city.ac.uk/id/eprint/26284/1/Final%20EmobodiedAI%20Submission.pdf

HexaJungle: a MARL Simulator to Study the Emergence of Language

Abstract

Similar works

Full text

Available Versions

City Research Online