Asking generative artificial intelligence the right questions improves writing performance

Abstract

Generative Artificial Intelligence (GenAI) tools are widely used by learners and this trend is poised to continue. However, little is known about whether and how GenAI use impacts learning and performance. This study aimed to investigate the effect of GenAI on performance by examining a key affordance of GenAI—seeking help via question asking. We compared the questions that learners asked GenAI versus a human tutor online during a writing task. Using quantitative ethnographic methods, we found that: (a) participants in the GenAI condition asked significantly more questions compared to those in the Tutor condition; (b) GenAI participants tended to ask one-off questions, while Tutor participants tended to have longer conversational exchanges; (c) GenAI participants tended to question pragmatically, asking direct questions about conceptual and procedural knowledge, while Tutor participants tended to make indirect request for feedback; (d) question asking, as measured by epistemic network analysis, mediated the relationship between experimental condition and performance—the more pragmatic the questions, and thus the more like questions in the GenAI condition, the better the performance; and (e) questions in the GenAI condition were driven by social coordination and knowledge deficits, while questions in the Tutor condition were driven by social coordination and establishing common ground. These findings suggest learners may be less hesitant to admit knowledge deficits and more willing to repair them when interacting with GenAI compared to human tutors. Thus, GenAI can be a useful educational tool when improved performance is the goal and human tutoring may benefit from creating a space where learners are more comfortable revealing a lack of knowledge.</p