DevGPT: Studying Developer-ChatGPT Conversations

Abstract

The emergence of large language models (LLMs) such as ChatGPT has disrupted the landscape of software development. Many studies are investigating the quality of responses generated by ChatGPT, the efficacy of various prompting techniques, and its comparative performance in programming contests, to name a few examples. Yet, we know very little about how ChatGPT is actually used by software developers. What questions do developers present to ChatGPT? What are the dynamics of these interactions? What is the backdrop against which these conversations are held, and how do the conversations feedback into the artifacts of their work? To close this gap, we introduce DevGPT, a curated dataset which encompasses 17,913 prompts and ChatGPT's responses including 11,751 code snippets, coupled with the corresponding software development artifacts -- ranging from source code, commits, issues, pull requests, to discussions and Hacker News threads -- to enable the analysis of the context and implications of these developer interactions with ChatGPT.Comment: MSR 2024 Mining Challenge Proposa