129,278 research outputs found
Recommended from our members
Understanding Model-Based Reinforcement Learning and its Application in Safe Reinforcement Learning
Model-based reinforcement learning algorithms have been shown to achieve successful results on various continuous control benchmarks, but the understanding of model-based methods is limited. We try to interpret how model-based method works through novel experiments on state-of-the-art algorithms with an emphasis on the model learning part. We evaluate the role of the model learning in policy optimization and propose methods to learn a more accurate model. With a better understanding of model-based reinforcement learning, we then apply model-based methods to solve safe reinforcement learning (RL) problems with near-zero violation of hard constraints throughout training. Drawing an analogy with how humans and animals learn to perform safe actions, we break down the safe RL problem into three stages. First, we train agents in a constraint-free environment to learn a performant policy for reaching high rewards, and simultaneously learn a model of the dynamics. Second, we use model-based methods to plan safe actions and train a safeguarding policy from these actions through imitation. Finally, we propose a factored framework to train an overall policy that mixes the performant policy and the safeguarding policy. This three-step curriculum ensures near-zero violation of safety constraints at all times. As an advantage of model-based method, the sample complexity required at the second and third steps of the process is significantly lower than model-free methods and can enable online safe learning. We demonstrate the effectiveness of our methods in various continuous control problems and analyze the advantages over state-of-the-art approaches
Delta Networks for Optimized Recurrent Network Computation
Many neural networks exhibit stability in their activation patterns over time
in response to inputs from sensors operating under real-world conditions. By
capitalizing on this property of natural signals, we propose a Recurrent Neural
Network (RNN) architecture called a delta network in which each neuron
transmits its value only when the change in its activation exceeds a threshold.
The execution of RNNs as delta networks is attractive because their states must
be stored and fetched at every timestep, unlike in convolutional neural
networks (CNNs). We show that a naive run-time delta network implementation
offers modest improvements on the number of memory accesses and computes, but
optimized training techniques confer higher accuracy at higher speedup. With
these optimizations, we demonstrate a 9X reduction in cost with negligible loss
of accuracy for the TIDIGITS audio digit recognition benchmark. Similarly, on
the large Wall Street Journal speech recognition benchmark even existing
networks can be greatly accelerated as delta networks, and a 5.7x improvement
with negligible loss of accuracy can be obtained through training. Finally, on
an end-to-end CNN trained for steering angle prediction in a driving dataset,
the RNN cost can be reduced by a substantial 100X
- …