Let's Ask Students About Their Programs, Automatically

Students sometimes produce code that works but that its author does not comprehend. For example, a student may apply a poorly-understood code template, stumble upon a working solution through trial and error, or plagiarize. Similarly, passing an automated functional assessment does not guarantee that the student understands their code. One way to tackle these issues is to probe students' comprehension by asking them questions about their own programs. We propose an approach to automatically generate questions about student-written program code. We moreover propose a use case for such questions in the context of automatic assessment systems: after a student's program passes unit tests, the system poses questions to the student about the code. We suggest that these questions can enhance assessment systems, deepen student learning by acting as self-explanation prompts, and provide a window into students' program comprehension. This discussion paper sets an agenda for future technical development and empirical research on the topic

Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests

Background and Context: Over the past year, large language models (LLMs) have taken the world by storm. In computing education, like in other walks of life, many opportunities and threats have emerged as a consequence. Objectives: In this article, we explore such opportunities and threats in a specific area: responding to student programmers' help requests. More specifically, we assess how good LLMs are at identifying issues in problematic code that students request help on. Method: We collected a sample of help requests and code from an online programming course. We then prompted two different LLMs (OpenAI Codex and GPT-3.5) to identify and explain the issues in the students' code and assessed the LLM-generated answers both quantitatively and qualitatively. Findings: GPT-3.5 outperforms Codex in most respects. Both LLMs frequently find at least one actual issue in each student program (GPT-3.5 in 90% of the cases). Neither LLM excels at finding all the issues (GPT-3.5 finding them 57% of the time). False positives are common (40% chance for GPT-3.5). The advice that the LLMs provide on the issues is often sensible. The LLMs perform better on issues involving program logic rather than on output formatting. Model solutions are frequently provided even when the LLM is prompted not to. LLM responses to prompts in a non-English language are only slightly worse than responses to English prompts. Implications: Our results continue to highlight the utility of LLMs in programming education. At the same time, the results highlight the unreliability of LLMs: LLMs make some of the same mistakes that students do, perhaps especially when formatting output as required by automated assessment systems. Our study informs teachers interested in using LLMs as well as future efforts to customize LLMs for the needs of programming education.Comment: 13 pages, 1 figure. To be published in Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (ICER '23 V1

Usability and Usage of Interactive Features in an Online Ebook for CS Teachers

There are too few secondary school computing teachers to meet international needs for growing secondary school computing education. Our group has created an ebook to help prepare secondary teachers to teach the programming and big data concepts in the new AP Computer Science Principles course. The ebook was designed using principles from educational psychology, specifically worked examples and cognitive load. The ebook interleaves worked examples and interactive practice activities, which we believe will lead to more efficient and effective learning than more typical approaches to learning programming. This paper reports the results from initial studies of our ebook. First, we conducted a usability study comparing three different ebook platforms. Next, we conducted a study of teacher use of the ebook. Ten teachers worked through the first eight chapters of the ebook at their own pace. Five of the ten teachers completed the first eight chapters which is a 50% completion rate. Significantly, teachers who used more of the interactive features in the ebook did better on the post-tests and reported higher confidence in their ability to teach the material than teachers who used few of the interactive features

Conceptions and Misconceptions about Computational Thinking among Italian Primary School Teachers

International audienceMany advanced countries are recognizing more and more the importance of teaching computing, in some cases even as early as in primary school. "Computational thinking" is the term often used to denote the conceptual core of computer science or "the way a computer scientist thinks", as Wing put it. Such term - given also the lack of a widely accepted definition - has become a "buzzword" meaning different things to different people. We investigated the Italian primary school teachers' conceptions about computational thinking by analyzing the results of a survey (N=972) conducted in the context of "Programma il Futuro" project. Teachers have been asked to provide a definition of computational thinking and to answer three additional related closed-ended questions. The analysis shows that, while almost half of teachers (43.4%) have included in their definitions some fundamental elements of computational thinking, very few (10.8%) have been able to provide an acceptably complete definition. On a more positive note, the majority is aware that computational thinking is not characterized by coding or by the use of information technology

Non-invasive markers of airway inflammation and remodeling in childhood asthma

To evaluate the relationship between pro-inflammatory and pro-remodeling mediators and severity and control of asthma in children, the levels of IL-8, MMP-9, TIMP-1 in induced sputum supernatants, the number of sputum eosinophils, as well as FeNO, were investigated in 35 asthmatic children, 12 with intermittent (IA) and 23 with moderate asthma (MA), and 9 controls (C). The patients with asthma were followed for 1 yr and sputum was obtained twice during the follow-up. Biomarker levels were correlated with the number of exacerbations. We found that IL-8, MMP-9, TIMP-1 and the numbers of eosinophils in induced sputum, as well as FeNO, were increased in children with IA and MA in comparison to C. The ongoing inflammation was confirmed by increased nuclear p65 NF-kappaB subunit localization in sputum cells. In MA, FeNO measurements, sputum eosinophils and IL-8 levels, positively correlated with the occurrence of disease exacerbations during a 1-yr follow-up. According to FeNO, sputum eosinophils and IL-8 sputum concentrations, and the number of exacerbations, two distinct phenotypes of MA were identified. This study shows that the presence of bronchial inflammation is detectable in the airways of some IA, as well as in the airways of MA, despite the regular ICS treatment. This study also proposes the need to perform large prospective studies to confirm the importance of measuring specific biomarkers in induced sputum, concomitantly to FeNO analyses, to assess sub-clinical airway inflammation and disease control in children with asthma