Data-efficient learning of feedback policies from image pixels using deep dynamical models

Data-efficient reinforcement learning (RL) in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. We consider a particularly important instance of this challenge, the pixels-to-torques problem, where an RL agent learns a closed-loop control policy ( torques ) from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model for learning a low-dimensional feature embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning is crucial for long-term predictions, which lie at the core of the adaptive nonlinear model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art RL methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces, is lightweight and an important step toward fully autonomous end-to-end learning from pixels to torques

Practical considerations for deep learning

The work in this dissertation was done as a major shift in machine perception and deep learning research was happening. Neural networks have proved to be an important part of machine perception and other domains of artificial intelligence over the last several years. This is due to several advances that have made neural networks more practical for real world applications. The goal of this dissertation is to present several works that track some advances in deep learning including: the move from greedy unsupervised pre-training to end-to-end supervised learning, GPU accelerated training of large neural, and the more recent successes of auto-regressive models for generating high-dimensional data. This dissertation will present four of my works. The first, develops a novel convolutional auto-encoder, and shows it can learn useful features that improve supervised image classification results when data is scarce. The second, uses distributed systems with multiple GPUs to train neural networks. The third, develops a method for using neural networks for object detection in video. The fourth speeds up generation for auto-regressive models of time-series, i.e. Wavenet. Then I will conclude and describe some follow up research I would like to pursue including: work on speeding up generation for auto-regressive models of images, i.e. PixelCNN, and using dilated causal convolutional models for Reinforcement Learning