A Policy-Guided Imitation Approach for Offline Reinforcement Learning

Offline reinforcement learning (RL) methods can generally be categorized into two types: RL-based and Imitation-based. RL-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneous off-policy evaluation. Imitation-based methods avoid off-policy evaluation but are too conservative to surpass the dataset. In this study, we propose an alternative approach, inheriting the training stability of imitation-style methods while still allowing logical out-of-distribution generalization. We decompose the conventional reward-maximizing policy in offline RL into a guide-policy and an execute-policy. During training, the guide-poicy and execute-policy are learned using only data from the dataset, in a supervised and decoupled manner. During evaluation, the guide-policy guides the execute-policy by telling where it should go so that the reward can be maximized, serving as the \textit{Prophet}. By doing so, our algorithm allows \textit{state-compositionality} from the dataset, rather than \textit{action-compositionality} conducted in prior imitation-style methods. We dumb this new approach Policy-guided Offline RL (\texttt{POR}). \texttt{POR} demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline RL. We also highlight the benefits of \texttt{POR} in terms of improving with supplementary suboptimal data and easily adapting to new tasks by only changing the guide-poicy.Comment: Oral @ NeurIPS 2022, code at https://github.com/ryanxhr/PO

A large eddy simulation turbulence model for estuary using spline correction

2006-2007 > Academic research: refereed > Publication in refereed journalVersion of RecordPublishe

Comparative studies on wood structure and microtensile properties between compression and opposite wood fibers of Chinese fir plantation

The microtensile properties of mechanically isolated compression wood (CW) and opposite wood (OW) tracheids of Chinese fir (Cunninghamia lanceolata) were investigated and discussed with respect to their structure. Major differences in the tensile modulus and ultimate tensile stress were found between CW and OW fibers. Compared to OW, CW showed a larger cellulose microfibril angle, less cellulose content and probably more pits, resulting in lower tensile properties. These findings contribute to a further understanding of the structural–mechanical relationships of Chinese fir wood at the cell and cell wall level, and provide a scientific basis for better utilization of plantation softwood