Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models

In the challenging field of introductory programming, high enrollments and failure rates drive us to explore tools and systems to enhance student outcomes, especially automated tools that scale to large cohorts. This paper presents and evaluates the dcc --help tool, an integration of a Large Language Model (LLM) into the Debugging C Compiler (DCC) to generate unique, novice-focused explanations tailored to each error. dcc --help prompts an LLM with contextual information of compile- and run-time error occurrences, including the source code, error location and standard compiler error message. The LLM is instructed to generate novice-focused, actionable error explanations and guidance, designed to help students understand and resolve problems without providing solutions. dcc --help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We analysed a subset of these error/explanation pairs to evaluate their properties, including conceptual correctness, relevancy, and overall quality. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code. Our findings, observations and reflections following deployment indicate that dcc-help provides novel opportunities for scaffolding students' introduction to programming.Comment: 7 pages, 2 figures. Accepted in SIGCSE'2

The use of manipulative materials in early place value instruction: a cognitive load perspective

Teaching mathematical concepts is often accompanied by the use of worked examples, and the use of manipulative materials. Worked examples have been shown to be an effective method of instruction with novice learners, as shown by higher test performance and shorter acquisition times. Often worked examples are accompanied by illustrations of manipulative materials, and the physical use of such materials. One such example is the use of Multi-base Arithmetic Blocks (MAB) to teach place value. Whilst the use of worked examples alongside illustrations of MAB in instructional material is common, their efficacy has not been investigated in young students. Using the concepts of cognitive load theory, which investigates how the learner’s limited working memory and vast long-term memory can be used to efficiently design educational material, this research examines the effects of using worked examples alongside MAB to teach place value to young students. Experiment 1 examined whether it was possible to facilitate a more efficient transition from the manipulative material to the abstract concept of place value. Using the abacus and MAB, and two methods of instruction, the results of Experiment 1 showed no significant differences in student performance between the methods of instruction, or the manipulative materials. Using worked examples in Experiment 2, no significant differences in the performance of students using either the abacus or MAB, were found. Experiment 3 reduced the level of the reading comprehension in the instructional material. The control group, with no access to MAB, performed better in the post-test than the group using MAB. Experiment 4 examined whether the use of MAB produced a redundancy effect, by providing identical information in three different formats. No significant differences were found due to the complexity of the instructional material. In Experiment 5, the expertise of students with respect to the experimental materials was decreased. The non-MAB group performed better in the post-test and the delayed test, than the group using MAB. Thus, despite the widespread use of MAB in primary school to teach place value, this research suggests that the blocks may produce a redundancy effect, leading to an increased extraneous load, and negatively affecting learning

The effect of personality in sample selection for usability testing

The aim of this thesis is to determine whether selecting usability testing participants on the basis of their personality, as measured by the Myers-Briggs Type Indicator (MBTI) extraversion/introversion scale, can enhance the results obtained in usability testing in a web context. This thesis assesses whether extraverted subjects can uncover a more significant subset of problems and, in particular, a larger number of more severe problems, than introverts during usability testing. If this were the case, then the process of usability testing can become more efficient by using participants who are able to provide the highest quality feedback and are able to find the greatest number of, and the most severe problems leading to a decrease in the number of necessary participants and a reduction in project costs and a higher return on investment. Forty-three randomly selected candidates were given the MBTI test. Of these, twenty qualified as either extraverts or introverts and were, therefore, selected to take part in the study. Two sample groups were constituted, one comprising ten extraverts and the other ten introverts. Each of these participants was then asked to take part in a formal laboratory-based usability testing with an e-commerce website. The participants were required to complete a total of five tasks. Performance metrics from these tasks were observed and recorded, with the main focus being on the number and severity of usability problems found by each of the participants. The severity rating was assigned based on an average severity score taken from four evaluators. This was then combined with the frequency of problem occurrence to give an overall severity rating to the usability problem ranging from two to eight, with two representing a low-level cosmetic problem and eight representing a major usability problem, which rendered the system unusable. The study established that extraverts found more usability problems than introverts (p<0.001), and also a larger number of more severe problems. Additionally, extraverts found on average 96% of all unique usability problems, whereas introverts found just 28% of all such problems. A strong positive correlation was found between the degree of extraversion and the overall number of usability problems found, using a Pearson correlation coefficient R=0.85 (p<0.01). Extraverted participants were more confident in their feedback and more comfortable voicing their opinions than introverted participants. Overall, extraverts talked more often, as measured by the words-per-second metric (p<0.001), and instigated more helpful commentary than introverts; through the feedback of extraverted subjects, 100% of all Category 6 high severity usability problems were uncovered. Based on the results of the usability testing carried out, this study has recommended a selection process for participants in usability testing, based on their levels of extraversion and their realistic use of the product. It is hoped that this approach to the selection of participants will ensure that the greatest number of, and the most severe, usability problems are found during usability testing. This will lead to a more efficient testing process and make it possible to decrease sample sizes in usability testing without reducing the quality of the results obtained, thereby providing a decrease in project costs and a higher return on investment

The adaptive virtual workshop : maintaining student engagement through an on-line adaptive resource for engineering design education

The use of ATs in teaching engineering design has resulted in improvements to the way educators are able to analyse student needs. This can further translate to improved student engagement and increased understanding of the subject matter, achieved through the improved, timely and individualised feedback mechanisms. The preliminary results of this research suggest that ATs do not necessarily result in higher student performance can improve overall performance of students in the subject and increase their understanding of the subject matter. Addressing This will in turn allow for better structure to the design of courses and the ability to efficiently understand and address any gaps in student knowledge. Preliminary results indicate that the inclusion of ATs in the teaching curriculum does not necessarily translate to higher marks or deeper student engagement; no statistically significant differences were seen in the final exam results in the subject. However, initial data suggests a higher overall pass rate for the subject with the inclusion of multiple ATs. Overall student feedback has suggested that students enjoyed using the ATs and would like these used more often in their subject. One third of students surveyed found that the ATs give them a better understanding of the subject matter. More research is required to support these preliminary results