Going SOLO to assess novice programmers

This paper explores the programming knowledge of novices using Biggs' SOLO taxonomy. It builds on previous work of Lister et al. (2006) and addresses some of the criticisms of that work. The research was conducted by studying the exam scripts for 120 introductory programming students, in which three specific questions were analyzed using the SOLO taxonomy. The study reports the following four findings: when the instruction to students used by Lister et al. - "In plain English, explain what the following segment of Java code does" - is replaced with a less ambiguous instruction, many students still provide multistructural responses; students are relatively consistent in the SOLO level of their answers; student responses on SOLO reading tasks correlate positively with performance on writing tasks; postgraduates students manifest a higher level of thinking than undergraduates. Copyright 2008 ACM

Introductory programming: a systematic literature review

As computing becomes a mainstream discipline embedded in the school curriculum and acts as an enabler for an increasing range of academic disciplines in higher education, the literature on introductory programming is growing. Although there have been several reviews that focus on specific aspects of introductory programming, there has been no broad overview of the literature exploring recent trends across the breadth of introductory programming. This paper is the report of an ITiCSE working group that conducted a systematic review in order to gain an overview of the introductory programming literature. Partitioning the literature into papers addressing the student, teaching, the curriculum, and assessment, we explore trends, highlight advances in knowledge over the past 15 years, and indicate possible directions for future research

Comparing Small Programs for Equivalence: A Code Comprehension Task for Novice Programmers

Novice programmers should develop program comprehension skills as they learn to code so that they are able both to read and reason about code created by others, and to reflect on their code when writing, debugging or extending it. This work takes a little-explored perspective on the comprehension of small programs by asking students to decide if two code segments are equivalent or not in terms of carrying out the same computation. A variation of Euclid's algorithm, that extends the greatest common divisor calculation to more than two numbers, was chosen for this work, as it has an adequate level of complexity and its semantics are not obvious. Four program transformations of the original code were developed: two transformations were equivalent and two were not. 73.5% of students were able to identify correctly the four options and 75.5% provided good insights on the equivalent program flow to justify their choices. The overall task has a SOLO mean of 3.19, which indicates code equivalence is a suitable and approachable task to analyse program execution at novice level. In addition, the data analysis suggests that students' code-reading abilities beyond basic tracing may be generally underestimated and we should investigate how to bridge the potential gap between reasoning about program execution and extracting its purpose

Reliably Classifying Novice Programmer Exam Responses using the SOLO Taxonomy

Abstract: Past papers of the BRACElet project have described an approach to teaching and assessing students where the students are presented with short pieces of code, and are instructed to explain, in plain English, what the code does. The student responses to these types of questions can be analysed according to the SOLO taxonomy. Some students display an understanding of the code as a single, functional whole, while other students cannot âsee the forest for the treesâ . However, classifying student responses into the taxonomy is not always straightforward. This paper analyses the reliability of the SOLO taxonomy as a means of categorising student responses. The paper derives an augmented set of SOLO categories for application to the programming domain, and proposes a set of guidelines for researchers to use

Abstraction in action: K-5 teachers' uses of levels of abstraction, particularly the design level, in teaching programming

Research indicates that understanding levels of abstraction (LOA) and being able to move between the levels is essential to programming success. For K-5 contexts LOA levels have been named: problem, design, code and running the code. In a qualitative exploratory study, five K-5 teachers were interviewed on their uses of LOA, particularly the design level, in teaching programming and other subjects. Using PCK elements to analyse responses, the teachers interviewed used design as an instructional strategy and for assessment. The teachers used design as an aide memoire and the expert teachers used design: as a contract for pair-programming; to work out what they needed to teach; for learners to annotate with code snippets (to transition across LOA); for learners to self-assess and to assess ‘do-ability’. The teachers used planning in teaching writing to scaffold learning and promote self-regulation revealing their insight in student understanding. One issue was of the teachers' knowledge of terms including algorithm and code; a concept of ‘emergent algorithms’ is proposed. Findings from the study suggest design helps learners learn to program in the same way that planning helps learners learn to write and that LOA, particularly the design level, may provide an accessible exemplar of abstraction in action. Further work is needed to verify whether the study's results are generalisable more widely

An Exploration of Novice Programmers' Comprehension of Conditionals in Imperative and Functional Programming

2Students of introductory programming courses are expected to develop higher-order thinking skills to inspect, understand and modify code. However, although novices can correctly write small programs, they appear to lack a more abstract, comprehensive grasp of basic constructs, such as conceiving the overall effect of alternative conditional flows. This work takes a little-explored perspective on the comprehension of tiny programs by asking students to reason about reversing conditionals in either an imperative or a functional context. More specifically, besides deciding if the given constructs can be reversed, students had to justify their choice by writing a reversing program or by providing suitable counterexamples. The students’ answers to four reversibility questions have been analysed through the lens of the SOLO taxonomy. 45% of students correctly identified the reversibility for the four code items; furthermore, more than 50% of each cohort were able to provide correct justifications for at least three of their four answers. Most incorrect answers were due to failures to consider border cases or to edit the conditional expressions appropriately to reverse the construct. Differences in comprehension between functional and imperative languages are explored indicating the explicit else paths of the functional examples facilitate comprehension compared with the implicit else (no update) of its imperative counterpart.partially_openopenMirolo, Claudio; Izu, CruzMirolo, Claudio; Izu, Cru

Novice Programmers' Reasoning About Reversing Conditional Statements

We want undergraduate students to develop higher-order thinking skills that enable them to master program behaviour. Nonetheless, many students of both introductory and advanced programming courses appear to struggle with the abstraction required for this purpose. In particular, a recent think-aloud study showed a group of students were able to reason about and reverse the effect of assignments and vector updates, but most of them failed when asked to reverse a seemingly simple conditional statement. We have extended that study by assigning a similar task to two cohorts of novice programmers as part of their final exam paper. Students' answers, including code and short explanations, have been analysed through the lens of the SOLO taxonomy: 28% of the solutions were correct (classified as relational); an additional 23% were partially correct but failed to identify the overlap between the two paths of the conditional statement (classified as multistructural). Furthermore, the concept of reversibility and related ``low-ceiling'' tasks, such as the one discussed in this study, could be useful resources for educators to assess and develop students' understanding of program behaviour

The Many Ways of the BRACElet Project

This paper provides a retrospective snapshot of the first two years of a multi-institutional multi-national study (MIMN) in Computer Science Education called the BRACElet Project. This study has been inquiring into how novice programmers comprehend and write computer programs. The context for the study is outlined, together with details of how it has evolved and those who have participated. Some challenges encountered during the project are highlighted and pointers for the successful conduct of such a study are provided. The paper concludes by noting pitfalls to be avoided, some open research questions, and current plans for furthering the project

Empirical Studies of Novices Learning Programming

This thesis is concerned with the problems that novices have in learning to program: in particular it is concerned with the difficulties experienced by novices learning at a distance, using instructional materials which have been designed especially for novices. One of the major problems for novices is how to link the new information which they encounter with their existing knowledge. Du Boulay, O'Shea and Monk (1981) suggest helping novices to bridge the gap between their existing knowledge and new information by teaching via a conceptual model, which serves to explain the new information in familiar terms.In this thesis the difficulties which novices have when learning to program with the help of a conceptual model were investigated. The curricula and conceptual models of four different programming languages are examined, all of which were designed to teach novices. Du Boulay, O'Shea and Monk (1981) have suggested criteria for analysing conceptual models. It is argued that these criteria, however, do not address the presentation of the conceptual model, and so are insufficient to evaluate them. An additional form of analysis was proposed and used, in addition to the criteria offered by Du Boulay et al. This is a way of describing the conceptual model which distinguishes three views of the conceptual model: state, procedure and function, and which highlights the different aspects which are important for the novice learner by identifying the different kinds of knowledge which are necessary to understand the conceptual model. This analysis of the conceptual models showed that the environments are not as exemplary as the du Boulay et al's criteria suggest, and indicated that three of the environments, SOLO, PT501 and DESMOND, lack a functional representation, and that the fourth, Open Logo, has other different problems.An empirical study was carried out to study the transfer effects of learning two of the languages, a high level and a low level language, sequentially. There was no evidence for such transfer effects. The difficulties novices have in learning the four different languages were also investigated. These studies show that even though the novices were studying environments designed for novices learning at a distance, they did not develop good levels of competence, and the problems they had fall into two main categories: programming and pedagogical.Although the different languages had different aims and curricula, novices had some problems which were common to all or most of the languages. These included understanding flow of control, developing and using programming plans, developing accurate mental models, and in the high level languages, understanding recursion. It is argued that some of these problems are related to the conceptual models. In particular, the difficulties novices had in developing and using plan knowledge, which is one of their main problems, can be explained by the lack of an appropriate functional description in the languages.The subjects' pedagogical problems arose from the relationship between the style and structure of the curriculum, its content, and the subjects themselves. In all the four texts the teaching material is very carefully structured and it is suggested that this may encourage the learner to adopt an over-dependent attitude towards the text, and in some cases, to work at an inappropriate syntactic level.The relationship between the distance learning situation and the novice programmer is discussed, and recommendations are made for improving the curricula used for teaching novices programming

High-school students' mastery of basic flow-control constructs through the lens of reversibility

High-school students specialising in computing fields need to develop the abstraction skills required to understand and create programs. Novices' difficulties at high-school level, ranging from mastery of the "notional machine"to recognition of a program's purpose, have not been investigated as extensively as at tertiary level. This work explores high-school students' code comprehension by asking to reason about reversing conditional and iteration constructs. A sample of 205 K11 - 13 students from different institutions were asked to engage in a set of "reversibility tasklets". For each code fragment, they need to identify if its computation is reversible and either provide the code to reverse or an example of a value that cannot be reversed. For 4 such items, after extracting the recurrent patterns in students' answers, we have carried out an analysis within the framework of the SOLO taxonomy. Overall, 74% of answers correctly identified if the code was reversible but only 42% could provide the full explanation/code. The rate of relational answers varies from 51% down to 21%, the poorest performance arising for a small array-processing loop (and although 65% of the subjects had correctly identified the loop as reversible). The instruction level did not have a strong impact on performance, indicating such tasks are suitable for K11, when the basic flow-control constructs are usually introduced. In particular, the reversibility concept could be a useful pedagogical instrument both to assess and to help develop students' program comprehension