In the autonomous driving system, trajectory prediction plays a vital role in
ensuring safety and facilitating smooth navigation. However, we observe a
substantial discrepancy between the accuracy of predictors on fixed datasets
and their driving performance when used in downstream tasks. This discrepancy
arises from two overlooked factors in the current evaluation protocols of
trajectory prediction: 1) the dynamics gap between the dataset and real driving
scenario; and 2) the computational efficiency of predictors. In real-world
scenarios, prediction algorithms influence the behavior of autonomous vehicles,
which, in turn, alter the behaviors of other agents on the road. This
interaction results in predictor-specific dynamics that directly impact
prediction results. As other agents' responses are predetermined on datasets, a
significant dynamics gap arises between evaluations conducted on fixed datasets
and actual driving scenarios. Furthermore, focusing solely on accuracy fails to
address the demand for computational efficiency, which is critical for the
real-time response required by the autonomous driving system. Therefore, in
this paper, we demonstrate that an interactive, task-driven evaluation approach
for trajectory prediction is crucial to reflect its efficacy for autonomous
driving