Data augmentation (DA) is a crucial technique for enhancing the sample
efficiency of visual reinforcement learning (RL) algorithms. Notably, employing
simple observation transformations alone can yield outstanding performance
without extra auxiliary representation tasks or pre-trained encoders. However,
it remains unclear which attributes of DA account for its effectiveness in
achieving sample-efficient visual RL. To investigate this issue and further
explore the potential of DA, this work conducts comprehensive experiments to
assess the impact of DA's attributes on its efficacy and provides the following
insights and improvements: (1) For individual DA operations, we reveal that
both ample spatial diversity and slight hardness are indispensable. Building on
this finding, we introduce Random PadResize (Rand PR), a new DA operation that
offers abundant spatial diversity with minimal hardness. (2) For multi-type DA
fusion schemes, the increased DA hardness and unstable data distribution result
in the current fusion schemes being unable to achieve higher sample efficiency
than their corresponding individual operations. Taking the non-stationary
nature of RL into account, we propose a RL-tailored multi-type DA fusion scheme
called Cycling Augmentation (CycAug), which performs periodic cycles of
different DA operations to increase type diversity while maintaining data
distribution consistency. Extensive evaluations on the DeepMind Control suite
and CARLA driving simulator demonstrate that our methods achieve superior
sample efficiency compared with the prior state-of-the-art methods.Comment: NeurIPS 2023 poste