With the development of blockchain technology, smart contracts have become an
important component of blockchain applications. Despite their crucial role, the
development of smart contracts may introduce vulnerabilities and potentially
lead to severe consequences, such as financial losses. Meanwhile, large
language models, represented by ChatGPT, have gained great attentions,
showcasing great capabilities in code analysis tasks. In this paper, we
presented an empirical study to investigate the performance of ChatGPT in
identifying smart contract vulnerabilities. Initially, we evaluated ChatGPT's
effectiveness using a publicly available smart contract dataset. Our findings
discover that while ChatGPT achieves a high recall rate, its precision in
pinpointing smart contract vulnerabilities is limited. Furthermore, ChatGPT's
performance varies when detecting different vulnerability types. We delved into
the root causes for the false positives generated by ChatGPT, and categorized
them into four groups. Second, by comparing ChatGPT with other state-of-the-art
smart contract vulnerability detection tools, we found that ChatGPT's F-score
is lower than others for 3 out of the 7 vulnerabilities. In the case of the
remaining 4 vulnerabilities, ChatGPT exhibits a slight advantage over these
tools. Finally, we analyzed the limitation of ChatGPT in smart contract
vulnerability detection, revealing that the robustness of ChatGPT in this field
needs to be improved from two aspects: its uncertainty in answering questions;
and the limited length of the detected code. In general, our research provides
insights into the strengths and weaknesses of employing large language models,
specifically ChatGPT, for the detection of smart contract vulnerabilities