Deep Reinforcement learning is a powerful machine learning paradigm that has had significant success across a wide range of control problems. This success often requires long training times to achieve. Observing that many problems share similarities, it is likely that much of the training done could be redundant if knowledge could be efficiently and appropriately shared across tasks. In this paper we demonstrate a novel adversarial domain adaptation approach to transfer state knowledge between domains and tasks on the Atari game suite. We show how this approach can successfully transfer across very different visual domains of the Atari platform. We focus on semantically related games that involve returning a ball with the user controlled agent. Our experiments demonstrate that our method reduces the number of samples required to successfully train an agent to play an Atari game

Carr, Thomas

Chli, Maria

Vogiatzis, George

English

arXiv

Aston Publications Explorer

Domain Adaptation for Reinforcement Learning on the AtariExtended AbstractThomas CarrComputer Science, Aston UniversityBirmingham, United Kingdomcarrtp@aston.ac.ukMaria ChliComputer Science, Aston UniversityBirmingham, United Kingdomm.chli@aston.ac.ukGeorge VogiatzisComputer Science, Aston UniversityBirmingham, United Kingdomg.vogiatzis@aston.ac.ukABSTRACTDeep Reinforcement learning is a powerful machine learning para-digm that has had significant success across a wide range of controlproblems. This success often requires long training times to achieve.Observing that many problems share similarities, it is likely thatmuch of the training done could be redundant if knowledge couldbe efficiently and appropriately shared across tasks. In this paperwe demonstrate a novel adversarial domain adaptation approach totransfer state knowledge between domains and tasks on the Atarigame suite. We show how this approach can successfully transferacross very different visual domains of the Atari platform. We focuson semantically related games that involve returning a ball withthe user controlled agent. Our experiments demonstrate that ourmethod reduces the number of samples required to successfullytrain an agent to play an Atari game.KEYWORDSDeep Learning; Reinforcement Learning; Domain AdaptationACM Reference Format:Thomas Carr, Maria Chli, and George Vogiatzis. 2019. Domain Adaptationfor Reinforcement Learning on the Atari. In Proc. of the 18th InternationalConference on Autonomous Agents and Multiagent Systems (AAMAS 2019),Montreal, Canada, May 13–17, 2019, IFAAMAS, 3 pages.1 INTRODUCTIONDeep Reinforcement Learning (DRL) successfully extends the re-inforcement learning paradigm to complex control problems byalleviating the need for expert hand-crafted features and allowingfor end-to-end learning from the input space, for example fromimages. One way to view learning within DRL is to consider thehidden layers as learning a state representation on top of whicha policy can be learned. This view has held true in the standarddeep learning paradigm, where the learned features are often re-purposed through direct transfer with or without fine-tuning for asubsequent task [11]. Learning the feature space from a sparse re-ward signal is difficult and so deep RL algorithms typically requirea large number of training samples; this is further exacerbated bythe standard RL problem of the exploration-exploitation trade-off.Transfer Learning can help by taking advantage of previoustraining to reduce the effort needed to solve a new task. In thiswork we show how learning an input mapping from source totarget task embedding in an unsupervised manner can be usedProc. of the 18th International Conference on Autonomous Agents and Multiagent Systems(AAMAS 2019), N. Agmon, M. E. Taylor, E. Elkind, M. Veloso (eds.), May 13–17, 2019,Montreal, Canada. © 2019 International Foundation for Autonomous Agents andMultiagent Systems (www.ifaamas.org). All rights reserved.ActionSource	TaskTarget	taskEncoder DecoderEncoderDiscriminatorPolicyEmbeddingsFigure 1: Our agent’s architecture for domain adaptation,combining adversarially discriminative domain adaptationand adversarial autoencoder architectures. A fully-trainedsource task is depicted on top, while on the bottom is thetarget task autoencoder.as an initialization step to improve learning even when the inputdomain, task and action space vary.2 ARCHITECTURE AND METHODWe propose an Adversarial Domain Adaptation approach to theproblem of Knowledge Transfer for reinforcement learning in theAtari domain. In this view the source and target problem share astate space and we wish to learn a mapping from the target obser-vations to this previously-learned source state space. Complicatingthis problem is the lack of an obvious alignment between sourceand target observation pairs.Our proposed Architecture, depicted in Fig. 1, builds on theAdversarial AutoEncoder (AAE) [6]. The AAE combines an au-toencoder (AE) and generative adversarial network (GAN) [4]. TheGAN regularizes the embedding space of the AE as an alternativeapproach to the approach of the variational autoencoder [5] Inour approach we replace the standard Gaussian distribution withsamples from a learned policy embedding.By using samples from the source task and the pre-trained sourcepolicy network, we can generate samples from the embedding ofthe source policy, which we use to represent the state space ofour policy. In this way, real states are sampled from the sourcedomain. We train a generator function from target observations tomatch the distribution of source states, as learned by our sourcepolicy. If the tasks are related and require similar state featurespre-training the generator in this manner should make it easierto learn a new task, as the problem of learning a state descriptor0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00number of steps ×107−20−1001020scoreBaselineOurs0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0number of steps ×1070100200300400scoreBaselineOurs0 1 2 3 4 5 6 7 8number of steps ×107−25.0−22.5−20.0−17.5−15.0−12.5−10.0−7.5−5.0scoreBaselinePongOursBreakoutOursFigure 2: The 100-episode average score plot against total number of steps. From left to right, 1: Source Task: Breakout, TargetTask Pong. 2: Source Task: Pong, Target Task: Breakout. 3: Source Tasks: Pong, Breakout, Target Task: Tennis.has been addressed. To ensure the learned mapping maintains allthe information contained in the target observations an AE is alsotrained in conjunction with the GAN element of the architecture.An alternative view is to see the source embedding as a regulariz-ing force on the auto-encoder embedding. This interpretation alignsthe method with alternative approaches many of which seek tominimize the Kullback-Liebler divergence between aligned sourceand target embeddings [12]. Shared embeddings are a natural wayto conceptualize transfer as we consider related domains to be partof a shared representation space. In this case the GAN elementallows us to perform that minimization without direct correspon-dences. Applying a GAN to align reinforcement learning domainsfor generalization has also recently been applied in the roboticscontext in [10].3 EXPERIMENTS AND RESULTSOur experiments investigate the applicability of Adversarial Do-main Adaptation to the reinforcement learning problem. We usethe Arcade Learning Environment (ALE) [2] to provide a set ofarcade games which are commonly used as benchmarks for DRL.We interface with ALE through the OpenAi Gym platform [3]. Ourexperiments focus on transfer between pairs of games. We beginby selecting pairs of games which we deem to be related in orderto verify the effectiveness of our approach; we leave automaticallyidentifying appropriate source tasks for future work. In this workwe focus on three games: Pong, Breakout and Tennis.To perform transfer we train an agent to solve the source task,in our case using A2C [1, 9]; a type of actor-critic algorithm. Wethen run this trained policy and store samples from the source task;in our case 100000 images. We also run a random policy on thetarget task and store 100000 sample images from this domain aswell. We now sample from these datasets to train our AdversarialAutoEncoder framework. The generator weights are then used toinitialise the weights of a new policy network for the target task.The final layers for the policy and value function are initialisedrandomly, as they were untrained during the pre-training phaseand the action space differs from the source task’s action-space.We can see in Fig. 2 that our approach significantly improveslearning on the target tasks. The baseline we compare against istraining a standard randomly initialised A2C agent on only thetarget task. The comparison clearly shows that our approach hasmuch promise.Since we always start training with randomly initialised policyand value function layers, we cannot expect transfer to provide ajumpstart [8] in rewards. We do however see observe that initialis-ing the state representation can improve over tabula rasa learningby decreasing the number of samples required to learn a solutionand therefore our method does achieve faster learning, which sug-gests our learned representation provides a better initialisation thanlearning tabula rasa.When examining the result of transfer to Tennis from either Pongor Breakout we can see that while both games provide benefit thereis a significant difference between them. Conceptually Pong andTennis are more aligned as they are both versions of the same gamewith a similar scoring mechanic. Breakout however is rotationallyaligned with Tennis, both games play vertically on the screen, pongplays horizontally. Understanding this difference and developinga method to identify the best sources for transfer is left as futurework.4 CONCLUSIONThis paper presents an adversarial method for knowledge transfer inreinforcement learning. We have demonstrated how this approachcan be used for domain adaptation to improve performance onthe difficult task of learning to play Atari games. As evident fromour results, the technique can help learning in the reinforcementlearning case, even when the final layer needs to be relearned asthe action space and task have changed, making this applicationof domain adaptation significantly more difficult than standardapplications of the technique.Transfer Learning is a complex problem with many approachesand applications from reducing the number of samples requiredto solve a problem to improving the usefulness of simulation fortraining of agents. Future work will compare our approach to otherTransfer learning methods such as the Progressive Neural Network[7].REFERENCES[1] 2018. a2c announcement. https://blog.openai.com/baselines-acktr-a2c/. (2018).accessed 09/03/2018.[2] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. 2013. The Arcade LearningEnvironment: An Evaluation Platform for General Agents. Journal of ArtificialIntelligence Research 47 (jun 2013), 253–279.[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, JohnSchulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. (2016).arXiv:arXiv:1606.01540[4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarialnets. In Advances in neural information processing systems. 2672–2680.[5] Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114 (2013).[6] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and BrendanFrey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015).[7] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, JamesKirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Pro-gressive neural networks. arXiv preprint arXiv:1606.04671 (2016).[8] Matthew E Taylor and Peter Stone. 2009. Transfer learning for reinforcementlearning domains: A survey. Journal of Machine Learning Research 10, Jul (2009),1633–1685.[9] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo,Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. 2016.Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 (2016).[10] Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. 2017. Mutual alignmenttransfer learning. arXiv preprint arXiv:1707.07907 (2017).[11] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transfer-able are features in deep neural networks?. In Advances in neural informationprocessing systems. 3320–3328.[12] Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. 2015.Supervised Representation Learning: Transfer Learning with Deep Autoencoders..In IJCAI. 4119–4125.

Domain adaptation for reinforcement learning on the Atari

https://publications.aston.ac.uk/id/eprint/39572/1/DAAAMAS.pdf

Domain adaptation for reinforcement learning on the Atari

Abstract

Similar works

Full text

Available Versions

Aston Publications Explorer