On Novices\u27 Interaction with Compiler Error Messages: A Human Factors Approach

The difficulty in understanding compiler error messages can be a major impediment to novice student learning. To alleviate this issue, multiple researchers have run experiments enhancing compiler error messages in automated assessment tools for programming assignments. The conclusions reached by these published experiments appear to be conflicting. We examine these experiments and propose five potential reasons for the inconsistent conclusions concerning enhanced compiler error messages: (1) students do not read them, (2) researchers are measuring the wrong thing, (3) the effects are hard to measure, (4) the messages are not properly designed, (5) the messages are properly designed, but students do not understand them in context due to increased cognitive load. We constructed mixed-methods experiments designed to address reasons 1 and 5 with a specific automated assessment tool, Athene, that previously reported inconclusive results. Testing student comprehension of the enhanced compiler error messages outside the context of an automated assessment tool demonstrated their effectiveness over standard compiler error messages. Quantitative results from a 60 minute one-on-one think-aloud study with 31 students did not show substantial increase in student learning outcomes over the control. However, qualitative results from the one-on-one thinkaloud study indicated that most students are reading the enhanced compiler error messages and generally make effective changes after encountering them

Beyond Automated Assessment: Building Metacognitive Awareness in Novice Programmers in CS1

The primary task of learning to program in introductory computer science courses (CS1) cognitively overloads novices and must be better supported. Several recent studies have attempted to address this problem by understanding the role of metacognitive awareness in novices learning programming. These studies have focused on teaching metacognitive awareness to students by helping them understand the six stages of learning so students can know where they are in the problem-solving process, but these approaches are not scalable. One way to address scalability is to implement features in an automated assessment tool (AAT) that build metacognitive awareness in novice programmers. Currently, AATs that provide feedback messages to students can be said to implement the fifth and sixth learning stages integral to metacognitive awareness: implement solution (compilation) and evaluate implemented solution (test cases). The computer science education (CSed) community is actively engaged in research on the efficacy of compile error messages (CEMs) and how best to enhance them to maximize student learning and it is currently heavily disputed whether or not enhanced compile error messages (ECEMs) in AATs actually improve student learning. The discussion on the effectiveness of ECEMs in AATs remains focused on only one learning stage critical to metacognitive awareness in novices: implement solution. This research carries out an ethnomethodologically-informed study of CS1 students via think-aloud studies and interviews in order to propose a framework for designing an AAT that builds metacognitive awareness by supporting novices through all six stages of learning. The results of this study provide two important contributions. The first is the confirmation that ECEMs that are designed from a human-factors approach are more helpful for students than standard compiler error messages. The second important contribution is that the results from the observations and post-assessment interviews revealed the difficulties novice programmers often face to developing metacognitive awareness when using an AAT. Understanding these barriers revealed concrete ways to help novice programmers through all six stages of the problem-solving process. This was presented above as a framework of features, which when implemented properly, provides a scalable way to implicitly produce metacognitive awareness in novice programmers

Not seeing the forest for the trees: Novice programmers and the SOLO taxonomy

This paper reports on the authors use of the SOLO taxonomy to describe differences in the way students and educators solve small code reading exercises. SOLO is a general educational taxonomy, and has not previously been applied to the study of how novice programmers manifest their understanding of code. Data was collected in the form of written and think-aloud responses from students (novices) and educators (experts), using exam questions. During analysis, the responses were mapped to the different levels of the SOLO taxonomy. From think-aloud responses, the authors found that educators tended to manifest a SOLO relational response on small reading problems, whereas students tended to manifest a multistructural response. These results are consistent with the literature on the psychology of programming, but the work in this paper extends on these findings by analyzing the design of exam questions. Copyright 2006 ACM

CodeProcess Charts: Visualizing the Process of Writing Code

Instructors of computer programming courses evaluate student progress on code submissions, exams, and other activities. The evaluation of code submissions is typically a summative assessment that gives very little insight into the process the student used when designing and writing the code. Thus, a tool that offers instructors a view into how students actually write their code could have broad impacts on assessment, intervention, instructional design, and plagiarism detection. In this article we propose an interactive software tool with a novel visualization that includes both static and dynamic views of the process that students take to complete computer programming assignments. We report results of an exploratory think-aloud study in which instructors offer thoughts as to the utility and potential of the tool. In the think-aloud study, we observed that the instructors easily identified multiple coding strategies (or the lack of thereof), were able to recognize plagiarism, and noticed a clear need for wider dissemination of tools for visualizing the programming process.Peer reviewe

An Exploration of Novice Programmers' Comprehension of Conditionals in Imperative and Functional Programming

2Students of introductory programming courses are expected to develop higher-order thinking skills to inspect, understand and modify code. However, although novices can correctly write small programs, they appear to lack a more abstract, comprehensive grasp of basic constructs, such as conceiving the overall effect of alternative conditional flows. This work takes a little-explored perspective on the comprehension of tiny programs by asking students to reason about reversing conditionals in either an imperative or a functional context. More specifically, besides deciding if the given constructs can be reversed, students had to justify their choice by writing a reversing program or by providing suitable counterexamples. The students’ answers to four reversibility questions have been analysed through the lens of the SOLO taxonomy. 45% of students correctly identified the reversibility for the four code items; furthermore, more than 50% of each cohort were able to provide correct justifications for at least three of their four answers. Most incorrect answers were due to failures to consider border cases or to edit the conditional expressions appropriately to reverse the construct. Differences in comprehension between functional and imperative languages are explored indicating the explicit else paths of the functional examples facilitate comprehension compared with the implicit else (no update) of its imperative counterpart.partially_openopenMirolo, Claudio; Izu, CruzMirolo, Claudio; Izu, Cru

An In-Depth Look at Learning Computer Language Syntax in a High-Repetition Practice Environment

Students in an introductory computer science course generally have difficulty producing code that follows the arrangement rules known as syntax. Phanon was created to help students practice writing correct code that follows the rules of syntax. Previous research suggests this tool has helped students improve their exam scores and strengthen effectiveness in the course. A study was conducted to observe students while they complete the syntax exercises to find meaningful patterns in the steps the students take to complete an exercise. Evidence to support high intrinsic load was found throughout the study, which is a measure of difficulty learning a subject. The syntax exercise design’s ineffectiveness, known as the extraneous cognitive load, was minimal throughout the study. It was also found that even if students seem to take longer completing the syntax exercises, it does not reflect a decrease in their performance for the class. This supports a theory that syntax is a separate process from problem-solving and mastering it can help students focus their cognitive process on problem-solving. Finding ordinary moments of comprehension or struggle can provide insight into how improvements can be made in Phanon and computer science teaching methods. The effectiveness of Phanon can be applied to students with a variety of programming experience

Increasing the Effectiveness of Error Messages in a Computer Programming and Simulation Tool

abstract: Each programming language has a compiler associated with it which helps to identify logical or syntactical errors in the program. These compiler error messages play important part in the form of formative feedback for the programmer. Thus, the error messages should be constructed carefully, considering the affective and cognitive needs of programmers. This is especially true for systems that are used in educational settings, as the messages are typically seen by students who are novice programmers. If the error messages are hard to understand then they might discourage students from understanding or learning the programming language. The primary goal of this research is to identify methods to make the error messages more effective so that students can understand them better and simultaneously learn from their mistakes. This study is focused on understanding how the error message affects the understanding of the error and the approach students take to solve the error. In this study, three types of error messages were provided to the students. The first type is Default type error message which is an assembler centric error message. The second type is Link type error message which is a descriptive error message along with a link to the appropriate section of the PLP manual. The third type is Example type error message which is again a descriptive error message with an example of the similar type of error along with correction step. All these error types were developed for the PLP assembly language. A think-aloud experiment was designed and conducted on the students. The experiment was later transcribed and coded to understand different approach students take to solve different type of error message. After analyzing the result of the think-aloud experiment it was found that student read the Link type error message completely and they understood and learned from the error message to solve the error. The results also indicated that Link type was more helpful compare to other types of error message. The Link type made error solving process more effective compared to other error types.Dissertation/ThesisMasters Thesis Software Engineering 201

Novice Programmers' Reasoning About Reversing Conditional Statements

We want undergraduate students to develop higher-order thinking skills that enable them to master program behaviour. Nonetheless, many students of both introductory and advanced programming courses appear to struggle with the abstraction required for this purpose. In particular, a recent think-aloud study showed a group of students were able to reason about and reverse the effect of assignments and vector updates, but most of them failed when asked to reverse a seemingly simple conditional statement. We have extended that study by assigning a similar task to two cohorts of novice programmers as part of their final exam paper. Students' answers, including code and short explanations, have been analysed through the lens of the SOLO taxonomy: 28% of the solutions were correct (classified as relational); an additional 23% were partially correct but failed to identify the overlap between the two paths of the conditional statement (classified as multistructural). Furthermore, the concept of reversibility and related ``low-ceiling'' tasks, such as the one discussed in this study, could be useful resources for educators to assess and develop students' understanding of program behaviour