The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?

ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users’ questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research

The effect of base rate sensitization on end-user query performance moderated by conscientiousness

End users with extensive experience with an organization’s data can often detect query errors when query results do not correspond to their ex ante expectations. Many end users, for example, newly hired business analysts, however, compose queries on unfamiliar data. Their lack of familiarity means that they may be less able to evaluate the reasonableness of their query results. Although additional query experience will eventually give them the familiarity with the data that they need, in the interim, they may not recognize incorrect results from flawed queries. This paper develops and tests base rate sensitization as a means of enabling end users to improve their query performance. Contrary to the hypotheses, sensitizing end users to base rates, as a means of improving their assessments of the likely correctness of their query results, was not associated with significantly fewer query errors on a consistent basis. In a post hoc analysis, participant conscientiousness was found to moderate query performance. Participants of high conscientiousness that were sensitized to base rates made fewer query errors than those not sensitized. In contrast, base rate-sensitized participants with low conscientiousness made more errors than those not sensitized. In this interaction, high conscientiousness participants were able to take advantage of base rate information while low conscientiousness participants appeared to be hindered by base rate sensitization