We present a computational framework capable of inferring the existence of groups, built upon social networks of re- ciprocal friendship, in Complex Adaptive Artificial Societies (CAAS). Our modelling framework infers the group identi- ties by following two steps: first, it aims to learn the on- going levels of cooperation among the agents and, second, it applies evolutionary computation, based on the learned cooperation values, to partition the agents into groups. Ex- perimental investigations, based on CAAS of agents who interact with each other by means of the Ultimatum Game, show that a cooperation learning phase, based on Reinforce- ment Learning, can provide highly promising results for min- imising the mismatch between the existing and the inferred groups, for two different society sizes under investigation.peer-reviewe

15th Annual Conference Companion on Genetic and Evolutionary Computation

Grappiolo, Corrado

Togelius, Julian

Yannakakis, Georgios N.

OAR@UM

Using Reinforcement Learning and Artificial Evolutionfor the Detection of Group Identitiesin Complex Adaptive Artificial SocietiesCorrado Grappiolo, Julian TogeliusCenter for Computer Games ResearchIT University of Copenhagen, Denmark{cogr, juto}@itu.dkGeorgios N. YannakakisInstitute of Digital GamesUniversity of Malta, Maltageorgios.yannakakis@um.edu.mtABSTRACTWe present a computational framework capable of inferringthe existence of groups, built upon social networks of re-ciprocal friendship, in Complex Adaptive Artificial Societies(CAAS). Our modelling framework infers the group identi-ties by following two steps: first, it aims to learn the on-going levels of cooperation among the agents and, second,it applies evolutionary computation, based on the learnedcooperation values, to partition the agents into groups. Ex-perimental investigations, based on CAAS of agents whointeract with each other by means of the Ultimatum Game,show that a cooperation learning phase, based on Reinforce-ment Learning, can provide highly promising results for min-imising the mismatch between the existing and the inferredgroups, for two different society sizes under investigation.Categories and Subject DescriptorsI.2.m [Computing Methodologies]: Artificial Intelligence—MiscellaneousGeneral TermsAlgorithmsKeywordsGroup Identity Detection; Artificial Societies; UltimatumGame; Reinforcement Learning; Evolutionary Computation.1. INTRODUCTIONWhen a population of socially-driven individuals are letto interact with each other in absence of any form of cen-tralised control, global patterns arise, such as the emergenceof friendship [13], culture [11], or other forms of group struc-tures. The patterns have an influence on the behavioursof each individuals; these are manifested by the concept ofgroup identity — i.e. the ability of the individuals to iden-tify themselves as part of a group —, meaning that indi-viduals belonging to the same group (in-group) will tend tobe more collaborative, intended as reciprocally altruistic [9],than when they interact with individuals belonging to othergroups (out-group) [4]. A straightforward, direct approachCopyright is held by the author/owner(s).GECCO’13 Companion, July 6–10, 2013, Amsterdam, The Netherlands.ACM 978-1-4503-1964-5/13/07.for the detection of groups is represented by the direct query-ing of each individual. However, this might not always bepossible, making thus the task of group detection/modellingproblematic. Nevertheless, due to the aforementioned re-lationship existing between the collaborative nature of theinteractions and the presence of group identities, alternativeindirect approaches could be adopted. The overall aim ofour research is to provide a computational framework of in-direct inference — hereafter called Group Modelling (GM)framework — capable of detecting the presence of groupstructures, and assign relative group identities, in ComplexAdaptive Society Systems. Whilst our ultimate goal is theapplication of our framework to virtual environments pop-ulated by human-controlled avatars, e.g. collaborative mul-tiplayer games, the focus given in this paper is on ComplexAdaptive Artificial Societies (CAAS) [8].2. EXPERIMENTAL SETUPThe agents forming the CAAS manifest relational-basedsocial preferences when interacting with each other by meansof the Ultimatum (or Bargain) Social Dilemma Game (UG).Our choice to use UG for our research is motivated by itsextensive application in many akin studies, among whichwe remark those which put an emphasis on social prefer-ences [2, 8, 10]. The complex global pattern under investiga-tion is the existence of group structures, based on the notionof reciprocal friendship, perceived by the adaptive agents ofthe society at any time throughout the experiments. Theevolution of friendship is implemented as follows: at regu-lar intervals of interaction episodes, a new friendship linkis added to the society’s social network, F , by applyingthe well-established Baraba´si-Albert algorithm for the cre-ation of Scale-free networks [1, 6]. As soon as a new edgeis added to F , the two involved agents are notified, theirbehaviours are affected (i.e. they start behaving more altru-istically with each other) and their own social perception to-wards each other moves towards friendship. The true groupidentities are obtained by partitioning F by means of thewell-established Clauset-Newman-Moore algorithm [3]. Theprocess ends when all the agents have at least one friend.3. GROUP MODELLING FRAMEWORKOur GM framework is composed of two interconnectedmodules: the first, called Cooperation Learning Module (CL),is in charge of learning the ongoing levels of cooperation ex-isting among the artificial agents; the second module, calledGroup Identity Detection Module (GID), partitions the agentsEpisode (t)Normalised Mismatch Error (nme)0.00.20.40.60.81.00 50 100 150PheromoneTD0α − cMC(a) 20 AgentsEpisode (t)Normalised Mismatch Error (nme)0.00.20.40.60.81.00 100 300 500 700PheromoneTD0α − cMC(b) 50 AgentsFigure 1: 20-Agent Societiesinto group structures, based on the learnt cooperation val-ues, and assigns a group identity to each one of the identifiedstructures.The main focus of this paper is the investigation of appro-priate and efficient techniques to be used for the CL module,being its computation the fundamental feature for the cor-rect computation of the GID module. For the GID module,instead, we rely on a Evolutionary Algorithm (EA) follow-ing the principles introduced in our previous study on staticsocieties [6].The CL module aims to learn the Society’s ongoing collab-oration values by solely processing the offers of the agents,given the assumption that friendship influences altruism orcooperation [7, 10]. We consider three different learningrules for the CL module: (1) Reinforcement Learning’s (RL)α-constant Monte Carlo (α -cMC) update rule [12] for non-stationary environments with Agent-Preference reward func-tion (AP) [7]; (2) Temporal-Difference (TD) update rule [12]with Entropy-Agent-Preference reward function (HAP) [7];(3) pheromone evaporation update rule used in Ant ColonyOptimisation [5].4. RESULTS AND DISCUSSIONIn order to examine which of the CL three modules, com-bined with the evolutionary GID module, would better in-fer the existence of emergent group structures of reciprocalfriendship we have conducted experiments on two scenarios:20 and 50 agent societies. Each scenario was iterated for 30experimental runs; the three GM frameworks were evaluatedagainst the same experimental setup, in order to grasp a bet-ter, more generic understanding of their performance, andto limit random fluctuations introduced by the interactionclassifiers and the stochastic nature of the EA. The perfor-mance measure we consider in this study is the normalisedmismatch error [6] between the true community structuresbuilt upon F and those detected by the GM framework.The results gathered (see Figure 4, the grey vertical bar in-dicates the interaction episode beyond which no new recipro-cal friendship links are perceived by the agents — consolida-tion of friendship) seem to suggest that the three techniquesused for the CL module are equally good. Nonetheless α-cMC with AP reward function appears to perform generallybetter than the other two, especially with respect to the con-solidation of the friendship structures and 50 agent societies,though statistical significance of such better performance isregistered only occasionally throughout the episodes [8].5. FUTUREWORK AND CONCLUSIONSThe rationale behind the success of α-cMC, and in partTD0, is their ability to perform the transition from theraw interactions to the two in/out-group classes. This wasachieved via the AP and HAP reward functions [7]. Thesemeasure altruism as a more general property of the agents,which cannot be successfully estimated by isolating each sin-gle agent interaction [7]. The CL module provides an aug-mented representation of F , by transforming the dichotomyfriend-vs-non friend into a continuous one by means of theupdate rules. This opens to a plethora of improvement pos-sibilities, two of which are proposed here: (1) a new modulecould be embedded after the CL module in order to refinethe collaboration values so that the GID module would fur-ther improve its detection performance [6]; (2) GM could beenriched by a new interactive component, giving the pos-sibility to the agents to self-report, at any time, their ownperceptions of friendship, and thus directly affect the learntcollaboration values. Other future work will consider thefurther investigation of alternative techniques for the CLand GID modules, plus the application of the GM frame-work to collaborative multiplayer games.6. REFERENCES[1] A. Baraba´si and R. Albert. Emergence of Scaling inRandom Networks. Science, 286(5439):509–512, 1999.[2] Y. Chen and S. Li. Group Identity and SocialPreferences. The American Economic Review, pages431–457, 2009.[3] A. Clauset, M. Newman, and C. Moore. FindingCommunity Structure in Very Large Networks.Physical Review E, 70(6):066111, 2004.[4] R. M. Dawes and D. M. Messick. Social Dilemmas.International Journal of Psychology, 2(35):111–116,2000.[5] M. Dorigo and K. Socha. An Introduction to AntColony Optimization. Handbook of ApproximationAlgorithms and Metaheuristics, pages 26–1, 2006.[6] C. Grappiolo, J. Togelius, and G. N. Yannakakis.Artificial Evolution for the Detection of GroupIdentities in Complex Artificial Societies. In Proc. ofIEEE ALife, pages 126–133, 2013.[7] C. Grappiolo and G. N. Yannakakis. TowardsDetecting Group Identities in Complex ArtificialSocieties. In Proc. of SAB, pages 421–430, 2012.[8] C. Grappiolo, J. Togelius, and G. N. Yannakakis.Interaction-based Group Identity Detection viaReinforcement Learning and Artificial Evolution. InProc. of EcoMass, (to appear), 2013.[9] S. J. Lansing. Complex Adaptive Systems. AnnualReview of Anthropology, 32:183–204, 2003.[10] F. Marzo, B. J. Grosz, and A. Pfeffer. Socialpreferences in Relational Contexts. In Proc. ofCollective Intentionality, 2005.[11] H. Situngkir. On Selfish Memes: Culture as ComplexAdaptive System. Journal of Social Complexity,2(1):20–32, 2004.[12] R. Sutton and A. Barto. Reinforcement Learning: AnIntroduction. Cambridge University Press, 1998.[13] E. Wolf. Kinship, Friendship, and Patron-clientRelations in Complex Societies. The SocialAnthropology of Complex Societies, pages 1–22, 1966.View publication stats

Using reinforcement learning and artificial evolution for the detection of group identities in complex adaptive artificial societies

https://www.um.edu.mt/library/oar//bitstream/123456789/23162/3/GrappioloTogeliusYannakakis_GECCO13%20%281%29.pdf

Using reinforcement learning and artificial evolution for the detection of group identities in complex adaptive artificial societies

Abstract

Similar works

Full text

Available Versions

OAR@UM