Comparing Code Explanations Created by Students and Large Language Models

Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course ($n \approx 1000$) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.Comment: 8 pages, 3 figures. To be published in Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V.

On-Demand Collaboration in Programming

In programming, on-demand assistance occurs when developers seek support for their tasks as needed. Traditionally, this collaboration happens within teams and organizations in which people are familiar with the context of requests and tasks. More recently, this type of collaboration has become ubiquitous outside of teams and organizations, due to the success of paid online crowdsourcing marketplaces (e.g., Upwork) and free online question-answering websites (e.g., Stack Overflow). Thousands of requests are posted on these platforms on a daily basis, and many of them are not addressed in a timely manner for a variety of reasons, including requests that often lack sufficient context and access to relevant artifacts. In consequence, on-demand collaboration often results in suboptimal productivity and unsatisfactory user experiences. This dissertation includes three main parts: First, I explored the challenges developers face when requesting help from or providing assistance to others on demand. I have found seven common types of requests (e.g., seeking code examples) that developers use in various projects when an on-demand agent is available. Compared to studying existing supporting systems, I suggest eight key system features to enable more effective on-demand remote assistance for developers. Second, driven by these findings, I designed and developed two systems: 1) CodeOn, a system that enables more effective task hand-offs (e.g., rich context capturing) between end-user developers and remote helpers than exciting synchronous support systems by allowing asynchronous responses to on-demand requests; and 2) CoCapture, a system that enables interface designers to easily create and then accurately describe UI behavior mockups, including changes they want to propose or questions they want to ask about an aspect of the existing UI. Third, beyond software development assistance, I also studied intelligent assistance for embedded system development (e.g., Arduino) and revealed six challenges (e.g., communication setup remains tedious) that developers have during on-demand collaboration. Through an imaginary study, I propose four design implications to help develop future support systems with embedded system development. This thesis envisions a future in which developers in all kinds of domains can effortlessly make context-rich, on-demand requests at any stage of their development processes, and qualified agents (machine or human) can quickly be notified and orchestrate their efforts to promptly respond to the requests.PHDInformationUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/166144/1/yanchenm_1.pd