Unsupervised learning of compact and relevant state representations has beenproved very useful at solving complex reinforcement learning tasks Ha and Schmid-huber (2018). In this paper, we propose a recurrent capsule network Hinton et al.(2011) that learns such representations by trying to predict the future observationsin an agent’s trajector

Annabi, L.

Garcia Ortiz, M.

City Research Online

              City, University of London Institutional RepositoryCitation: Annabi, L. and Garcia Ortiz, M. ORCID: 0000-0003-4729-7457 (2018). State representation learning with recurrent capsule networks. Paper presented at the Workshop on Modeling the Physical World: Perception, Learning, and Control, NeurIPS 2018 - 32nd Conference on Neural Information Processing Systems, 07 Dec 2018, Montreal, Canada. This is the accepted version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  http://openaccess.city.ac.uk/id/eprint/22453/Link to published version: Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research OnlineState representation learning with recurrent capsulenetworksLouis Annabi, Michael Garcia OrtizAI LabSoftbank Robotics EuropeParis, Francelouis.annabi@gmail.commgarciaortiz@softbankrobotics.comAbstractUnsupervised learning of compact and relevant state representations has beenproved very useful at solving complex reinforcement learning tasks Ha and Schmid-huber (2018). In this paper, we propose a recurrent capsule network Hinton et al.(2011) that learns such representations by trying to predict the future observationsin an agent’s trajectory.1 IntroductionWe study the mechanisms that could allow task-independent and self-supervised learning for embodiedagents. Specifically, we focus here on the acquisition of perceptive skills, in particular representationfor physical objects.Recently, capsule networks were proposed as a model for representing visual objects. A capsule is aset of artificial neurons that returns both an activation and instantiation parameters. The activationis the estimated probability of the feature being present in the input image, and is similar to theoutputs of neurons in convolutional neural networks. The instantiation parameters are estimations ofdifferent properties of the detected feature, such as position, orientation, lighting or color. However,reaching configurations where each capsule models one visual entity can be tricky without using aclassification loss, here we are looking for an approach that doesn’t rely on external information.Sensorimotor prediction is often proposed to learn representations. It has been extensively used inreinforcement learning literature as an auxiliary task to learn both state representations and forwardmodels. In Ha and Schmidhuber (2018), the authors first learn a latent representation using variationalauto-encoders Kingma and Welling (2013) and then learn a forward model by trying to predict futurelatent states. In Wayne et al. (2018), the authors propose a fully differentiable RL architecture trained,specially, with a reconstruction loss of the inputs and a KL divergence between the estimated andactual latent state distributions.In this work, we explore the use of capsule networks for state representation learning. We reformulatethe principle of capsules in order to use them in the framework of sensorimotor prediction. Thisallows us to build a state representation preserving the benefits explained in Hinton et al. (2011) whileavoiding the use of external data.2 ModelOur predictive model is made of four functional blocks, that are repetitively used at each time step :an encoder, a recurrent cell, a transformation cell and a decoder. We use artificial neural networks for32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.arXiv:1812.11202v4  [cs.LG]  22 Feb 2019Figure 1: Recurrent model used to train capsules. E encodes the observation xt into a capsulerepresentation ct, which is fed to a recurrent cellR that produces the state capsule representation rt.T uses rt together with the motor commands mt to predict the next state ht+1. Finally, the decoderD estimates the next observation ˆxt+1each of these blocks, making the model fully differentiable. Figure 1 shows how those blocks areorganized to form the full model in charge of predicting the future observations.2.1 EncoderFigure 2: Architecture of the capsule network used for encoding the observations. The network iscomposed of two convolutional layers, and one convolutional capsule layer.The encoder outputs a capsule representation ct of shape (k, d+1) where k is the number of capsules,d is the dimension of the instantiation parameter vector returned by each capsule, and the +1 standsfor the additional activation output for each capsule. Figure 2 shows the architecture of the encodernetwork that we used in our experiments.2.2 Custom recurrent cellThe recurrent cell is a custom neural network model that updates the state capsule representation htwith the capsule representation of the observation ct. This network outputs an updated state capsulerepresentation rt. All three variables have the same shape (k, d+ 1). The custom recurrent cell ismaking use of the structure of the capsule output (activation dimension denoted a and instantiationparameters dimensions denoted v) and performs better than traditional recurrent cells (GRU, LSTM)on the sensorimotor prediction task. Here is the detail of the operation performed:rat = max(hat , cat )t = f(hat , cat ) =11 +hatcat2rvt = (1− t) · hvt + t · g(hvt , cvt ;w)Where g is a linear function of parameter w taking as input the concatenation of hvt and cvt . The mainidea here is to modulate the output with an update parameters t depending on the activations of thestate capsule representation and the activations coming from the observation capsule representation.2.3 Transformation cellThe transformation cell applies the transformation induced by the continuous action mt to the staterepresentation rt. We split the instantiation parameters in two categories : the "fixed" parameters2denoted vf that will not be affected by the transformation, and the "variable" parameters denoted vvthat are allowed to change. The transformation cell first computes a matrix representation zt for theaction mt using a multi-layer perceptron. The action representation is a square matrix of size dv < d,where dv is the number of variable parameters. Then the capsules received in input are transformedby multiplying the vector of variable parameters with the action matrix representation. Here is thedetail of the operation performed :hat+1 = rathvft+1 = rvfthvvt+1 = matmul(mlp(mt), rvvt )2.4 DecoderThe decoder is responsible for computing an estimation of the next observation xˆt+1 based on thetransformed state ht+1. It is composed of a deep transposed convolutional neural network shared foreach capsule, outputting an individual reconstruction for each capsule, and a custom merge layer, incharge of computing the observation estimation based on the individual reconstructions.3 Experiment3.1 EnvironmentTo validate our approach, we tested our model in a partially observable environment with first-personobservations. To be able to conduct many experiments, we chose to use the 2D simulator FlatlandCaselles-Dupré et al. (2018), where we simulate an environment similar to Eslami et al. (2018).Having a 2D simulator produces observations in 1D, making it faster to conduct experiments. Ourenvironment is composed of three simple round objects of three different colors, as shown in figure 3a.An agent equipped with vision sensor (raw RGB input) moves by random longitudinal displacement,lateral displacement, and rotation.3.2 ResultsThe model is trained with a mean squared error loss on the estimated observations. Two additionallosses are used to ensure a proper convergence : a sparsity constraint on the activations in the hiddenstate, and a mean squared error between the hidden state in input of the recurrent cell and the outputof the recurrent cell.(a) One of the simulated envi-ronments, with the agent in lightgray and the three objects in pur-ple, orange and green. The posi-tion and size of the three objectsvary along episodes.(b) Next observation prediction at the end of training. This figures shows asample from a trajectory of length 100 on which the model was not trained.The five columns correspond to: 1. The real observation. 2. The predictedobservation. 3. The individual prediction of the first capsule. 4. Theindividual prediction of the second capsule. 5. The individual prediction ofthe third capsule.Figure 3b shows the sensorimotor prediction at the end of training. We can see from this figure thateach capsule represents one object of the environment, which was the desired configuration. We canalso notice that the purple object is properly predicted in frame 84 even though it had left the field3of vision of the agent since frame 77, which indicates that our model is able to properly track theposition of an object through long sequences of actions.Figure 4: Individual capsule representation plotted according to two dimensions of the variableparameters (left and middle), and the two dimensions of the fixed parameters (right). The color mapused correspond to the real relative X (left) and Y (middle) coordinates of the encoded object in theagent’s relative coordinate system, and to the color of the object (right).Moreover, when looking at the learned representations of each object (see figure 4), we observea high correlation between some of the capsule dimensions with the real relative position of thecorresponding object with regard to the agent. This suggests that the agent has build a notion of spacethrough this prediction process, without any information directly related to space in its observations(such as depth).4 Future workFirst, we need to run more experiments on the learned representations. Early results tend to showthat this capsule representation performs well as input of a reinforcement learning algorithm in anavigation task, compared to learning from raw images. To further verify the quality of our model forreinforcement learning, we would need to compare it with other state representation learning methods(like Ha and Schmidhuber (2018)) and on several environments.In the future, we would like to extend this model, for instance by adding routing with a second layerof capsules, and apply it to more complex environments, with more objects and more variabilitybetween objects.ReferencesHugo Caselles-Dupré, Louis Annabi, Oksana Hagen, Michael Garcia-Ortiz, and David Filliat. 2018.Flatland: a Lightweight First-Person 2-d Environment for Reinforcement Learning. (2018).arXiv:cs.LG/1809.00510S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Mor-cos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gre-gor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosen-baum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Ko-ray Kavukcuoglu, and Demis Hassabis. 2018. Neural scene representation and render-ing. Science 360, 6394 (2018), 1204–1210. https://doi.org/10.1126/science.aar6170arXiv:http://science.sciencemag.org/content/360/6394/1204.full.pdfDavid Ha and Jürgen Schmidhuber. 2018. World Models. CoRR abs/1803.10122 (2018).arXiv:1803.10122 http://arxiv.org/abs/1803.10122Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. 2011. Transforming Auto-Encoders. InArtificial Neural Networks and Machine Learning – ICANN 2011, Timo Honkela, WłodzisławDuch, Mark Girolami, and Samuel Kaski (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg,44–51.Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. CoRR abs/1312.6114(2013). arXiv:1312.6114 http://arxiv.org/abs/1312.61144Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska-Barwinska, Jack W. Rae, Piotr Mirowski, Joel Z. Leibo, Adam Santoro, Mevlana Gemici, MalcolmReynolds, Tim Harley, Josh Abramson, Shakir Mohamed, Danilo Jimenez Rezende, David Saxton,Adam Cain, Chloe Hillier, David Silver, Koray Kavukcuoglu, Matthew Botvinick, Demis Hassabis,and Timothy P. Lillicrap. 2018. Unsupervised Predictive Memory in a Goal-Directed Agent. CoRRabs/1803.10760 (2018). arXiv:1803.10760 http://arxiv.org/abs/1803.107605

State representation learning with recurrent capsule networks

http://openaccess.city.ac.uk/id/eprint/22453/1/1812.11202.pdf

State representation learning with recurrent capsule networks

Abstract

Similar works

Full text

Available Versions

City Research Online