Autonomous vehicles (AV) are expected to reshape future transportation
systems, and decision-making is one of the critical modules toward high-level
automated driving. To overcome those complicated scenarios that rule-based
methods could not cope with well, data-driven decision-making approaches have
aroused more and more focus. The datasets to be used in developing data-driven
methods dramatically influences the performance of decision-making, hence it is
necessary to have a comprehensive insight into the existing datasets. From the
aspects of collection sources, driving data can be divided into vehicle,
environment, and driver related data. This study compares the state-of-the-art
datasets of these three categories and summarizes their features including
sensors used, annotation, and driving scenarios. Based on the characteristics
of the datasets, this survey also concludes the potential applications of
datasets on various aspects of AV decision-making, assisting researchers to
find appropriate ones to support their own research. The future trends of AV
dataset development are summarized