The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?

ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users’ questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research

A model for and the effects of information request ambiguity on end-user query performance

The increasing reliance of organizations on information technology, which prompts everyone to expect faster responses to information needs, is propelling end users to satisfy many information requests they receive by querying databases themselves. This paper develops and tests a model for the effects of information request ambiguity on end-user query performance where performance is measured by the number of errors in user- developed queries, the time taken to complete queries, and end usersí confidence in the correctness of their queries. Based on preliminary analysis of participantsí performance, end-user query performance was significantly degraded by the presence of ambiguity in information requests. The model identifies seven ambiguities: lexical, syntactical, inflective, pragmatic, extraneous, emphatic, and suggestive. Organizations whose participants rely on e-mail to communicate information requests or whose work teams experience rapid personnel turnover may be especially vulnerable to the debilitating effects of ambiguities on information requests

The effects of normalization on end-user query errors: An experimental evaluation

As a result of the shortage of professional programmers to extract timely information from databases, end-users are increasingly developing their own database queries. Because end-user querying is error-prone, characterizing the sources of query errors and using that knowledge to improve the effectiveness of end-user query development can improve the quality of information used for decision making. This paper reports the results of an experiment that investigated the effect of normalization level on query errors. The results show that query errors vary with the normalization level of the database structure and confirm previous findings about query errors increasing with task complexity. End-users querying a first normal form data structure make fewer errors than end-users querying an unnormalized data structure or a third normal form data structure. Furthermore, end-users querying a third normal form data structure make fewer errors than end-users querying an unnormalized data structure

The effects of intrinsic motivation on knowledge work: Performance in database query construction

Not available

Understanding Conceptual Schemas: Exploring the Role of Application and IS Domain Knowledge

Although information systems (IS) problem solving involves knowledge of both the IS and application domains, little attention has been paid to the role of application domain knowledge. In this study, which is set in the context of conceptual modeling, we examine the effects of both IS and application domain knowledge on different types of schema understanding tasks: syntactic and semantic comprehension tasks and schema-based problem-solving tasks. Our thesis was that while IS domain knowledge is important in solving all such tasks, the role of application domain knowledge is contingent upon the type of understanding task under investigation. We use the theory of cognitive fit to establish theoretical differences in the role of application domain knowledge among the different types of schema understanding tasks. We hypothesize that application domain knowledge does not influence the solution of syntactic and semantic comprehension tasks for which cognitive fit exists, but does influence the solution of schema-based problem-solving tasks for which cognitive fit does not exist. To assess performance on different types of conceptual schema understanding tasks, we conducted a laboratory experiment in which participants with high- and low-IS domain knowledge responded to two equivalent conceptual schemas that represented high and low levels of application knowledge (familiar and unfamiliar application domains). As expected, we found that IS domain knowledge is important in the solution of all types of conceptual schema understanding tasks in both familiar and unfamiliar applications domains, and that the effect of application domain knowledge is contingent on task type. Our findings for the EER model were similar to those for the ER model. Given the differential effects of application domain knowledge on different types of tasks, this study highlights the importance of considering more than one application domain in designing future studies on conceptual modeling