19 research outputs found
Comparing Code Explanations Created by Students and Large Language Models
Reasoning about code and explaining its purpose are fundamental skills for
computer scientists. There has been extensive research in the field of
computing education on the relationship between a student's ability to explain
code and other skills such as writing and tracing code. In particular, the
ability to describe at a high-level of abstraction how code will behave over
all possible inputs correlates strongly with code writing skills. However,
developing the expertise to comprehend and explain code accurately and
succinctly is a challenge for many students. Existing pedagogical approaches
that scaffold the ability to explain code, such as producing exemplar code
explanations on demand, do not currently scale well to large classrooms. The
recent emergence of powerful large language models (LLMs) may offer a solution.
In this paper, we explore the potential of LLMs in generating explanations that
can serve as examples to scaffold students' ability to understand and explain
code. To evaluate LLM-created explanations, we compare them with explanations
created by students in a large course () with respect to
accuracy, understandability and length. We find that LLM-created explanations,
which can be produced automatically on demand, are rated as being significantly
easier to understand and more accurate summaries of code than student-created
explanations. We discuss the significance of this finding, and suggest how such
models can be incorporated into introductory programming education.Comment: 8 pages, 3 figures. To be published in Proceedings of the 2023
Conference on Innovation and Technology in Computer Science Education V.
Automatically Generating CS Learning Materials with Large Language Models
Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and
Codex, now enable software developers to generate code based on a natural
language prompt. Within computer science education, researchers are exploring
the potential for LLMs to generate code explanations and programming
assignments using carefully crafted prompts. These advances may enable students
to interact with code in new ways while helping instructors scale their
learning materials. However, LLMs also introduce new implications for academic
integrity, curriculum design, and software engineering careers. This workshop
will demonstrate the capabilities of LLMs to help attendees evaluate whether
and how LLMs might be integrated into their pedagogy and research. We will also
engage attendees in brainstorming to consider how LLMs will impact our field.Comment: In Proceedings of the 54th ACM Technical Symposium on Computing
Science Educatio
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
Identifying and resolving logic errors can be one of the most frustrating
challenges for novices programmers. Unlike syntax errors, for which a compiler
or interpreter can issue a message, logic errors can be subtle. In certain
conditions, buggy code may even exhibit correct behavior -- in other cases, the
issue might be about how a problem statement has been interpreted. Such errors
can be hard to spot when reading the code, and they can also at times be missed
by automated tests. There is great educational potential in automatically
detecting logic errors, especially when paired with suitable feedback for
novices. Large language models (LLMs) have recently demonstrated surprising
performance for a range of computing tasks, including generating and explaining
code. These capabilities are closely linked to code syntax, which aligns with
the next token prediction behavior of LLMs. On the other hand, logic errors
relate to the runtime performance of code and thus may not be as well suited to
analysis by LLMs. To explore this, we investigate the performance of two
popular LLMs, GPT-3 and GPT-4, for detecting and providing a novice-friendly
explanation of logic errors. We compare LLM performance with a large cohort of
introductory computing students solving the same error detection
task. Through a mixed-methods analysis of student and model responses, we
observe significant improvement in logic error identification between the
previous and current generation of LLMs, and find that both LLM generations
significantly outperform students. We outline how such models could be
integrated into computing education tools, and discuss their potential for
supporting students when learning programming
Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book
Advances in natural language processing have resulted in large language
models (LLMs) that are capable of generating understandable and sensible
written text. Recent versions of these models, such as OpenAI Codex and GPT-3,
can generate code and code explanations. However, it is unclear whether and how
students might engage with such explanations. In this paper, we report on our
experiences generating multiple code explanation types using LLMs and
integrating them into an interactive e-book on web software development. We
modified the e-book to make LLM-generated code explanations accessible through
buttons next to code snippets in the materials, which allowed us to track the
use of the explanations as well as to ask for feedback on their utility. Three
different types of explanations were available for students for each
explainable code snippet; a line-by-line explanation, a list of important
concepts, and a high-level summary of the code. Our preliminary results show
that all varieties of explanations were viewed by students and that the
majority of students perceived the code explanations as helpful to them.
However, student engagement appeared to vary by code snippet complexity,
explanation type, and code snippet length. Drawing on our experiences, we
discuss future directions for integrating explanations generated by LLMs into
existing computer science classrooms
Comparing Code Explanations Created by Students and Large Language Models
Funding Information: We are grateful for the grant from the Ulla Tuominen Foundation to the first author. Publisher Copyright: © 2023 Owner/Author.Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.Peer reviewe
Engaging Youth, Serving Community: Social Change Lessons from a 4-H Rural Youth Development Program
Recommended from our members
The Implications of Large Language Models for CS Teachers and Student
The introduction of Large Language Models (LLMs) has generated a significant amount of excitement both in industry and among researchers. Recently, tools that leverage LLMs have made their way into the classroom where they help students generate code and help instructors generate learning materials. There are likely many more uses of these tools -- both beneficial to learning and possibly detrimental to learning. To help ensure that these tools are used to enhance learning, educators need to not only be familiar with these tools, but with their use and potential misuse. The goal of this BoF is to raise awareness about LLMs and to build a learning community around their use in computing education. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed discussion leaders, including undergraduate researchers, to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education