11,495 research outputs found
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
We present the first provably convergent two-timescale off-policy
actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is
the introduction of a new critic, the emphasis critic, which is trained via
Gradient Emphasis Learning (GEM), a novel combination of the key ideas of
Gradient Temporal Difference Learning and Emphatic Temporal Difference
Learning. With the help of the emphasis critic and the canonical value function
critic, we show convergence for COF-PAC, where the critics are linear and the
actor can be nonlinear.Comment: ICML 202
Deep Residual Reinforcement Learning
We revisit residual algorithms in both model-free and model-based
reinforcement learning settings. We propose the bidirectional target network
technique to stabilize residual algorithms, yielding a residual version of DDPG
that significantly outperforms vanilla DDPG in the DeepMind Control Suite
benchmark. Moreover, we find the residual algorithm an effective approach to
the distribution mismatch problem in model-based planning. Compared with the
existing TD() method, our residual-based method makes weaker assumptions
about the model and yields a greater performance boost.Comment: AAMAS 202
- …