Prompt Problems: A New Programming Exercise for the Generative AI Era

Large Language Models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards code reading, comprehension and evaluation of LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly for the first time in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.Comment: Accepted to SIGCSE'24. arXiv admin note: substantial text overlap with arXiv:2307.1636

Plagiarism Deterrence in CS1 Through Keystroke Data

Recent work in computing education has explored the idea of analyzing and grading using the process of writing a computer program rather than just the final submitted code. We build on this idea by investigating the effect on plagiarism when the process of coding, in the form of keystroke logs, is submitted for grading in addition to the final code. We report results from two terms of a university CS1 course in which students submitted keystroke logs. We find that when students are required to submit a log of keystrokes together with their written code they are less likely to plagiarize. In this paper we explore issues of implementation, adoption, deterrence, anxiety, and privacy. Our keystroke logging software is available in the form of an IDE plugin in a public plugin repository

CodEval

Grading coding assignments call for a lot of work. There are numerous aspects of the code that need to be checked, such as compilation errors, runtime errors, the number of test cases passed or failed, and plagiarism. Automated grading tools for programming assignments can be used to help instructors and graders in evaluating the programming assignments quickly and easily. Creating the assignment on Canvas is again a time taking process and can be automated. We developed CodEval, which instantly grades the student assignment submitted on Canvas and provides feedback to the students. It also uploads, creates, and edits assignments, thereby making the whole experience streamlined and quick for instructors and students. It is simple to use, easily integrated with the learning management system, and has a low learning curve. This report shows the background, implementation, and results of using CodEval for programming courses

Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style

The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses, and recent work has focused on improving the quality of automatically generated feedback. However, there is a relative lack of data directly comparing student outcomes when receiving computer-generated feedback and human-written feedback. This paper addresses this gap by splitting one 90-student class into two feedback groups and analyzing differences in the two cohorts' performance. The class is an intro to AI with programming HW assignments. One group of students received detailed computer-generated feedback on their programming assignments describing which parts of the algorithms' logic was missing; the other group additionally received human-written feedback describing how their programs' syntax relates to issues with their logic, and qualitative (style) recommendations for improving their code. Results on quizzes and exam questions suggest that human feedback helps students obtain a better conceptual understanding, but analyses found no difference between the groups' ability to collaborate on the final project. The course grade distribution revealed that students who received human-written feedback performed better overall; this effect was the most pronounced in the middle two quartiles of each group. These results suggest that feedback about the syntax-logic relation may be a primary mechanism by which human feedback improves student outcomes.Comment: Published in SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science Educatio

Accurate Estimation of Time-on-Task While Programming

In a recent study, students were periodically prompted to self-report engagement while working on computer programming assignments in a CS1 course. A regression model predicting time-on-task was proposed. While it was a significant improvement over ad-hoc estimation techniques, the study nevertheless suffered from lack of error analysis, lack of comparison with existing methods, subtle complications in prompting students, and small sample size. In this paper we report results from a study with an increased number of student participants and modified prompting scheme intended to better capture natural student behavior. Furthermore, we perform a cross-validation analysis on our refined regression model and present the resulting error bounds. We compare with threshold approaches and find that, in at least one context, a simple 5-minute threshold of inactivity is a reasonable estimate for whether a student is on-task or not. We show that our approach to modeling student engagement while programming is robust and suitable for identification of students in need of intervention, understanding engagement behavior, and estimating time taken on programming assignments

What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming

Metacognition and self-regulation are popular areas of interest in programming education, and they have been extensively researched outside of computing. While computing education researchers should draw upon this prior work, programming education is unique enough that we should explore the extent to which prior work applies to our context. The goal of this systematic review is to support research on metacognition and self-regulation in programming education by synthesizing relevant theories, measurements, and prior work on these topics. By reviewing papers that mention metacognition or self-regulation in the context of programming, we aim to provide a benchmark of our current progress towards understanding these topics and recommendations for future research. In our results, we discuss eight common theories that are widely used outside of computing education research, half of which are commonly used in computing education research. We also highlight 11 theories on related constructs (e.g., self-efficacy) that have been used successfully to understand programming education. Towards measuring metacognition and self-regulation in learners, we discuss seven instruments and protocols that have been used and highlight their strengths and weaknesses. To benchmark the current state of research, we examined papers that primarily studied metacognition and self-regulation in programming education and synthesize the reported interventions used and results from that research. While the primary intended contribution of this paper is to support research, readers will also learn about developing and supporting metacognition and self-regulation of students in programming courses

Experiences from Teaching Automated Testing with CrowdSorcerer

Software testing is an important process when ensuring a program's quality. However, testing has not traditionally been a very substantial part of computer science education. Some attempts to integrate it into the curriculum has been made but best practices still prove to be an open question. This thesis discusses multiple attempts of teaching software testing during the years. It also introduces CrowdSorcerer, a system for gathering programming assignments with tests from students. It has been used in introductory programming courses in University of Helsinki. To study if the students benefit from creating assignments with CrowdSorcerer, we analysed the number of assignments and tests they created and if they correlate with their performance in a testing-related question in the course exam. We also gathered feedback from the students on their experiences from using CrowdSorcerer. Looking at the results, it seems that more research on how to teach testing would be beneficial. Improving CrowdSorcerer would also be a good idea

Programming Process, Patterns and Behaviors: Insights from Keystroke Analysis of CS1 Students

With all the experiences and knowledge, I take programming as granted. But learning to program is still difficult for a lot of introductory programming students. This is also one of the major reasons for a high attrition rate in CS1 courses. If instructors were able to identify struggling students then effective interventions can be taken to help them. This thesis is a research done on programming process data that can be collected non-intrusively from CS1 students when they are programming. The data and their findings can be leveraged in understanding students’ thought process, detecting patterns and identifying behaviors that could possibly help instructors to identify struggling students, help them and design better courses

Choosing Code Segments to Exclude from Code Similarity Detection

When student programs are compared for similarity as a step in the detection of academic misconduct, certain segments of code are always sure to be similar but are no cause for suspicion. Some of these segments are boilerplate code (e.g. public static void main String [] args) and some will be code that was provided to students as part of the assessment specification. This working group explores these and other types of code that are legitimately common in student assessments and can therefore be excluded from similarity checking. From their own institutions, working group members collected assessment submissions that together encompass a wide variety of assessment tasks in a wide variety of programming languages. The submissions were analysed to determine what sorts of code segment arose frequently in each assessment task. The group has found that common code can arise in programming assessment tasks when it is required for compilation purposes; when it reflects an intuitive way to undertake part or all of the task in question; when it can be legitimately copied from external sources; and when it has been suggested by people with whom many of the students have been in contact. A further finding is that the nature and size of the common code fragments vary with course level and with task complexity. An informal survey of programming educators confirms the group's findings and gives some reasons why various educators include code when setting programming assignments.Peer reviewe