Evolving deep unsupervised convolutional networks for vision-based reinforcement learning

Abstract

Dealing with high-dimensional input spaces, like visual in-put, is a challenging task for reinforcement learning (RL). Neuroevolution (NE), used for continuous RL problems, has to either reduce the problem dimensionality by (1) compress-ing the representation of the neural network controllers or (2) employing a pre-processor (compressor) that transforms the high-dimensional raw inputs into low-dimensional fea-tures. In this paper, we are able to evolve extremely small re-current neural network (RNN) controllers for a task that pre-viously required networks with over a million weights. The high-dimensional visual input, which the controller would normally receive, is first transformed into a compact fea-ture vector through a deep, max-pooling convolutional neu-ral network (MPCNN). Both the MPCNN preprocessor and the RNN controller are evolved successfully to control a car in the TORCS racing simulator using only visual input. This is the first use of deep learning in the context evolutionary RL