7 research outputs found
Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
Offline multi-agent reinforcement learning is challenging due to the coupling
effect of both distribution shift issue common in offline setting and the high
dimension issue common in multi-agent setting, making the action
out-of-distribution (OOD) and value overestimation phenomenon excessively
severe. Tomitigate this problem, we propose a novel multi-agent offline RL
algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct
conservative value estimation. Rather than regarding all the agents as a high
dimensional single one and directly applying single agent methods to it, CFCQL
calculates conservative regularization for each agent separately in a
counterfactual way and then linearly combines them to realize an overall
conservative value estimation. We prove that it still enjoys the
underestimation property and the performance guarantee as those single agent
conservative methods do, but the induced regularization and safe policy
improvement bound are independent of the agent number, which is therefore
theoretically superior to the direct treatment referred to above, especially
when the agent number is large. We further conduct experiments on four
environments including both discrete and continuous action settings on both
existing and our man-made datasets, demonstrating that CFCQL outperforms
existing methods on most datasets and even with a remarkable margin on some of
them.Comment: 37th Conference on Neural Information Processing Systems (NeurIPS
2023
DARL: Distance-Aware Uncertainty Estimation for Offline Reinforcement Learning
To facilitate offline reinforcement learning, uncertainty estimation is commonly used to detect out-of-distribution data. By inspecting, we show that current explicit uncertainty estimators such as Monte Carlo Dropout and model ensemble are not competent to provide trustworthy uncertainty estimation in offline reinforcement learning. Accordingly, we propose a non-parametric distance-aware uncertainty estimator which is sensitive to the change in the input space for offline reinforcement learning. Based on our new estimator, adaptive truncated quantile critics are proposed to underestimate the out-of-distribution samples. We show that the proposed distance-aware uncertainty estimator is able to offer better uncertainty estimation compared to previous methods. Experimental results demonstrate that our proposed DARL method is competitive to the state-of-the-art methods in offline evaluation tasks
State Deviation Correction for Offline Reinforcement Learning
Offline reinforcement learning aims to maximize the expected cumulative rewards with a fixed collection of data. The basic principle of current offline reinforcement learning methods is to restrict the policy to the offline dataset action space. However, they ignore the case where the dataset's trajectories fail to cover the state space completely. Especially, when the dataset's size is limited, it is likely that the agent would encounter unseen states during test time. Prior policy-constrained methods are incapable of correcting the state deviation, and may lead the agent to its unexpected regions further. In this paper, we propose the state deviation correction (SDC) method to constrain the policy's induced state distribution by penalizing the out-of-distribution states which might appear during the test period. We first perturb the states sampled from the logged dataset, then simulate noisy next states on the basis of a dynamics model and the policy. We then train the policy to minimize the distances between the noisy next states and the offline dataset. In this manner, we allow the trained policy to guide the agent to its familiar regions. Experimental results demonstrate that our proposed method is competitive with the state-of-the-art methods in a GridWorld setup, offline Mujoco control suite, and a modified offline Mujoco dataset with a finite number of valuable samples