The debut of ChatGPT has recently attracted the attention of the natural
language processing (NLP) community and beyond. Existing studies have
demonstrated that ChatGPT shows significant improvement in a range of
downstream NLP tasks, but the capabilities and limitations of ChatGPT in terms
of recommendations remain unclear. In this study, we aim to conduct an
empirical analysis of ChatGPT's recommendation ability from an Information
Retrieval (IR) perspective, including point-wise, pair-wise, and list-wise
ranking. To achieve this goal, we re-formulate the above three recommendation
policies into a domain-specific prompt format. Through extensive experiments on
four datasets from different domains, we demonstrate that ChatGPT outperforms
other large language models across all three ranking policies. Based on the
analysis of unit cost improvements, we identify that ChatGPT with list-wise
ranking achieves the best trade-off between cost and performance compared to
point-wise and pair-wise ranking. Moreover, ChatGPT shows the potential for
mitigating the cold start problem and explainable recommendation. To facilitate
further explorations in this area, the full code and detailed original results
are open-sourced at https://github.com/rainym00d/LLM4RS