Havrylov, Serhii

Titov, Ivan

Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator (Jang et al., 2017)) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol

Edinburgh Research Explorer

Emergence of Language with Multi-Agent Games: Learning to Communicate with Sequence of Symbols

     Edinburgh Research Explorer                                      Emergence of Language with Multi-Agent Games: Learning toCommunicate with Sequence of SymbolsCitation for published version:Havrylov, S & Titov, I 2017, Emergence of Language with Multi-Agent Games: Learning to Communicatewith Sequence of Symbols. in 5th International Conference on Learning Representations (ICLR 17,workshop track). 5th International Conference on Learning Representations, Toulon, France, 24/04/17.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:5th International Conference on Learning Representations (ICLR 17, workshop track)General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 11. May. 2020Workshop track - ICLR 2017EMERGENCE OF LANGUAGE WITH MULTI-AGENTGAMES: LEARNING TO COMMUNICATE WITHSEQUENCES OF SYMBOLSSerhii Havrylov, Ivan TitovInstitute for Logic, Language and ComputationUniversity of Amsterdam{s.havrylov,titov}@uva.nlABSTRACTLearning to communicate through interaction, rather than relying on explicit su-pervision, is often considered a prerequisite for developing a general AI. We studya setting where two agents engage in playing a referential game and, from scratch,develop a communication protocol necessary to succeed in this game. We requirethat messages they exchange, both at train and test time, are in the form of alanguage (i.e. sequences of discrete symbols). As the ultimate goal is to ensurethat communication is accomplished in natural language, we perform preliminaryexperiments where we inject prior information about natural language into ourmodel and study properties of the resulting protocol.1 INTRODUCTIONWith the rapid advances in machine learning in recent years, the goal of enabling intelligent agents tocommunicate with each other and with humans is turning from a hot topic of philosophical debatesinto a practical engineering problem. It is believed that supervised learning alone is not going toprovide a solution to this challenge (Mikolov et al., 2015). These, as well as other, considerationshave motivated previous research into set-ups where agents invent a communication protocol whichlets them succeed in a given collaborative task. For an extensive overview of earlier work in thisarea, we refer the reader to Kirby (2002). We continue this line of research and specifically considera setting where the collaborative task is a game. Neural network models have been shown to beable to successfully induce a communication protocol for this setting (Lazaridou et al., 2016; Jorgeet al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016). One important difference with theseprevious approaches is that we assume that messages exchanged between the agents are variable-length sequences of symbols rather than atomic categories (as in previous work). In our experiments,we focus on a referential game. The setting of the game is slightly different from that of Lazaridouet al. (2016). Its description can be found in the appendix 5.1.2 MODEL2.1 AGENTS’ ARCHITECTURESThe sender and the receiver are implemented as LSTM networks (Hochreiter & Schmidhuber,1997). Figure 1 shows the sketch of model architecture, where diamond-shaped, dashed and solid ar-rows represent sampling, copying and deterministic functions respectively. The inputs to the senderare the target image t and the special token <S>, which denotes the start of a message. Given theseinputs, the sender generates next token wi in a sequence by sampling from the categorical distribu-tion Cat(pti), where pti = softmax(Whsi + b). Here, hsi is the hidden state of sender’s LSTM andcan be calculated as hsi = LSTM(hsi−1, wi−1)1. In the first time step we have hs0 = f(t), where f(t)is an affine transformation of image features extracted from a convolutional neural network (CNN).Message mt is obtained by sequentially sampling until the maximum possible length L is reachedor the special token <S> is generated.1We omitted the cell state in the equation for brevity.1Workshop track - ICLR 2017Figure 1: Architectures of sender and receiverThe inputs to the receiver are the generated message mt and a set of images that contain the targetimage t and distracting images {dk}Kk=1. Receiver interpretation of message is represented by thelast hidden state of the LSTM network hrl . The loss function for the whole system can be written as:Lφ,θ(t) =K∑k=1max[0, 1 + g(t)Thrl − g(dk)Thrl ] (1)Here g(·) is yet another affine transformation of image features extracted from the CNN. The energyfunction g(·)Thrl can be used to define the probability distribution over a set of images. Communi-cation between two agents is successful if the target image has the highest probability according tothis distribution.2.2 STRAIGHT-THROUGH GUMBEL-SOFTMAX ESTIMATORGenerating message mt requires sampling from categorical distributions over vocabulary, whichmakes backpropagating the error through the message impossible. It is tempting to formulate thisgame as a reinforcement learning problem. However, the number of possible messages2 is propor-tional to |V |L. Therefore, it seems unlikely that such model can be learned with naı¨ve Monte Carlomethods. Also, in this setup, the receiver Rθ would correspond to the non-stationary environmentin which sender Sφ acts, making the learning problem even more challenging. As a solution, weconsider replacement of one-hot encoded symbols w ∈ V with continuous relaxation w˜ obtainedfrom the Gumbel-softmax distribution (Jang et al., 2016; Maddison et al., 2016). As a result of thisrelaxation the game becomes completely differentiable and can be trained using the backpropaga-tion algorithm. Communicating with real values allows the sender to encode much more informationinto a message comparing to using a discrete one and is unrealistic if our ultimate goal is commu-nication in natural language. To prevent this behaviour, we discretize w˜ back with argmax in theforward pass. Nevertheless, we use continuous relaxation in the backward pass, effectively assuming∂L∂w ≈ ∂L∂w˜ . This biased estimator is known as the straight-through Gumbel-softmax estimator (Janget al., 2016; Bengio et al., 2013). As a result of applying this trick, there is no difference in usingmessages during training and testing stages, which contrasts with previous differentiable frameworksfor learning communication protocols (Foerster et al., 2016; Sukhbaatar et al., 2016).3 EXPERIMENTS3.1 TABULA RASA COMMUNICATIONWe used the Microsoft COCO dataset (Chen et al., 2015) as a source of images. The MSCOCO2014 validation set was used for evaluating the learned language. In our experiments images arerepresented by outputs of the relu7 layer from the pretrained 16-layer VGG convolutional net-work (Simonyan & Zisserman, 2014).2In our experiments |V | = 10000 and L is up to 14.2Workshop track - ICLR 2017Table 1: Comparison of grounded protocol with natural language and artificial language.Model Comm. success (%) Number of updates Omission scoreWith KL regularization 52.51 11600 0.258Without regularization 95.65 27600 0.193Imaginet 52.51 16100 0.287Figure 2: The performance and properties of learned proto-cols.The model details can be found in ap-pendix 5.2. Figure 2 shows the com-munication success rate as a functionof the maximum message length L.Interestingly, the number of updatesthat are required to achieve trainingconvergence decreases when we letthe sender use longer messages. Inother words, using longer sequenceshelps learn a communication protocolfaster. This behaviour is slightly sur-prising as one could expect that it isharder to learn the protocol when thespace of messages is larger. We alsoplot the perplexity of the encoder, itis relatively high and increasing withsentence length. This implies redun-dancy in the encodings: there ex-ist multiple paraphrases that encodethe same semantic content. For addi-tional qualitative analysis of the learned language we refer reader to appendix 5.3.3.2 GROUNDING ARTIFICIAL LANGUAGE IN NATURAL LANGUAGEAs the ultimate goal is to ensure that communication is accomplished with a language that is un-derstandable by humans, we should favour protocols which resemble, in some respects, a natu-ral language. One possible solution is to use the Kullback-Leibler (KL) divergence regularizationDKL (qφ(m|t)‖pω(m)), from the estimated natural language model to the learned protocol. Detailsregarding language model could be found in appendix 5.2. This regularization provides indirectsupervision by encouraging generated messages to have a high probability in natural language butat the same time maintaining high entropy for the communication protocol.To get an estimate of communication success when using natural language we trained receiver withpairs of images and captions. This model is similar to Imaginet (Chrupała et al., 2015). Also,inspired by their analysis, we report the omission score. The score quantifies the change in thetarget image probability after removing the most important word in a sentence. Natural languageshave content words that name objects and encode their qualities. One can expect that a protocolthat distinguishes between content words and function words would have a higher omission score incomparison to a protocol that distributes information evenly across tokens. As Table 1 shows, thegrounded language has the communication success rate similar to natural language. However, it hasa slightly lower omission score. The model without regularization has the lowest omission scorewhich probably means that symbols in the developed protocol have similar nature to characters orsyllables rather than words.4 CONCLUSIONWe have shown that agents, modeled using neural networks, can successfully invent a language thatconsists of sequences of discrete tokens. Despite the common belief that it is hard to train suchmodels, we proposed an efficient learning strategy which relies on the straight-through Gumbel-softmax estimator. We have performed basic analysis of the learned language and learning dynamics.We have also conducted preliminary experiments with the grounding learned language in naturallanguage. Many open questions remain, so further investigation of this topic is needed.3Workshop track - ICLR 2017ACKNOWLEDGMENTSThis project is supported by SAP ICN and ERC Starting Grant BroadSem (678254).REFERENCESYoshua Bengio, Nicholas Le´onard, and Aaron Courville. Estimating or propagating gradientsthrough stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dolla´r, andC Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXivpreprint arXiv:1504.00325, 2015.Grzegorz Chrupała, Akos Ka´da´r, and Afra Alishahi. Learning language through pictures. arXivpreprint arXiv:1506.03694, 2015.Jakob Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communi-cate with deep multi-agent reinforcement learning. In Advances in Neural Information ProcessingSystems, pp. 2137–2145, 2016.Sepp Hochreiter and Ju¨rgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXivpreprint arXiv:1611.01144, 2016.Emilio Jorge, Mikael Ka˚geba¨ck, and Emil Gustavsson. Learning to play guess who? and inventinga grounded language as a consequence. arXiv preprint arXiv:1611.03218, 2016.Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014.Simon Kirby. Natural language from artificial life. Arificial Life, 8:185–215, 2002.Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and theemergence of (natural) language. arXiv preprint arXiv:1612.07182, 2016.Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuousrelaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016.Tomas Mikolov, Armand Joulin, and Marco Baroni. A roadmap towards machine intelligence. arXivpreprint arXiv:1511.08130, 2015.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale imagerecognition. arXiv preprint arXiv:1409.1556, 2014.Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropaga-tion. In Advances in Neural Information Processing Systems, pp. 2244–2252, 2016.5 APPENDIX5.1 REFERENTIAL GAME DESCRIPTION1. There is a collection of images {in}Nn=1 from which target image t is sampled as well as Kdistracting images {dk}Kk=1.2. There are two agents: sender Sφ and receiver Rθ.3. After seeing target image t, the sender has to come up with a message mt, which is rep-resented by a sequence of symbols from the vocabulary V of a size |V |. The maximumpossible length of a sequence is L.4. Given a message mt and a set of images, which consists of distracting images and a targetimage, the goal of the receiver is to identify the target image correctly.4Workshop track - ICLR 20175.2 MODEL SPECIFICATIONWe set the following hyperparameters without tuning: embedding dimensionality is 256, the dimen-sionality of LSTM layer is 512, vocabulary size is 10000, Gumbel-softmax distribution temperatureis 1.0, the number of distracting images is 127, batch size is 128. We used Adam (Kingma & Ba,2014) as an optimizer, with default hyperparameters and a learning rate of 0.001. We implementedlanguage model pω(m) by using an LSTM recurrent neural network. Image captions of randomlyselected (50%) images from the training set were used to estimate parameters of the language model.These images were not used for training the sender and the receiver.5.3 QUALITATIVE ANALYSIS OF THE LEARNED LANGUAGETo better understand the nature of the learned language, we inspected a small subset of sentencesthat were produced by the model with maximum possible message length equal to 5. Figure 3shows some samples from the MSCOCO 2014 validation set that correspond to (5747 * * **) code3. Images in this subset depict mainly animals. On the other hand, it seems that samples infigure 4 do not correspond to any predefined category. This suggests that word order is crucial in thedeveloped language. Particularly, word 5747 on the first position encodes presence of an animal inthe image. Considering Figure 5, we can conclude that adding word 5490 on the second positionreduces possible options just to zebras, giraffes and sometimes horses. When we move token 5490to the end of the message, we end up just with zebras in the images (Figure 6). Figure 7 showsthat message (5747 5747 7125 * *) corresponds to a particular type of bears. This suggeststhat developed language implements some kind of hierarchical coding. This is interesting by itselfbecause the model was not constrained explicitly to use any hierarchical encoding scheme. Figures8, 9, 10 show similar behaviour for images in a food domain.Figure 3: Images that correspond to (5747 * * * *) code.Figure 4: Images that correspond to (* * * 5747 *) code.Figure 5: Images that correspond to (5747 5490 * * *) code.3* means any word from the vocabulary.5Workshop track - ICLR 2017Figure 6: Images that correspond to (5747 * * * 5490) code.Figure 7: Images that correspond to (5747 5747 7125 * *) code.Figure 8: Images that correspond to (5261 * * * *) code.Figure 9: Images that correspond to (5261 2250 * * *) code.Figure 10: Images that correspond to (5261 2250 5211 * *) code.6

English

https://www.research.ed.ac.uk/portal/files/37795495/bb5181d2caf01da2808e9a0a99274b4da70b20d4.pdf

Emergence of Language with Multi-Agent Games: Learning to Communicate with Sequence of Symbols

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Edinburgh Research Explorer