Software Verification and Graph Similarity for Automated Evaluation of Students' Assignments

In this paper we promote introducing software verification and control flow graph similarity measurement in automated evaluation of students' programs. We present a new grading framework that merges results obtained by combination of these two approaches with results obtained by automated testing, leading to improved quality and precision of automated grading. These two approaches are also useful in providing a comprehensible feedback that can help students to improve the quality of their programs We also present our corresponding tools that are publicly available and open source. The tools are based on LLVM low-level intermediate code representation, so they could be applied to a number of programming languages. Experimental evaluation of the proposed grading framework is performed on a corpus of university students' programs written in programming language C. Results of the experiments show that automatically generated grades are highly correlated with manually determined grades suggesting that the presented tools can find real-world applications in studying and grading

An integrated approach to courseware

Software engineering is becoming increasingly important as an engineering discipline, and its teaching in universities and other higher education institutions should be of high quality. In this paper we describe a tool (BOSS — the Boss Online Submission System) which aids the education of software engineers. BOSS allows students to submit programming assignments online, and to run black-box tests on their programs prior to submission. Instructors can use BOSS to assist in marking such assignments by allowing submitted programs to be tested against multiple data sets. We describe how BOSS helps in the teaching of specific conceptual aspects of software engineering, and how it addresses some of the practical issues involved in teaching large student numbers in a pedagogically neutral manner

Marking complex assignments using peer assessment with an electronic voting system and an automated feedback tool

The work described in this paper relates to the development and use of a range of initiatives in order to mark complex masters' level assignments related to the development of computer web applications. In the past such assignments have proven difficult to mark since they assess a range of skills including programming, human computer interaction and design. Based on the experience of several years marking such assignments, the module delivery team decided to adopt an approach whereby the students marked each other's practical work using an electronic voting system (EVS). The results of this are presented in the paper along with statistical comparison with the tutors' marking, providing evidence for the efficacy of the approach. The second part of the assignment related to theory and documentation. This was marked by the tutors using an automated feedback tool. It was found that the time to mark the work was reduced by more than 30% in all cases compared to previous years. More importantly it was possible to provide good quality individual feedback to learners rapidly. Feedback was delivered to all within three weeks of the test submission datePeer reviewe

An evaluation of electronic individual peer assessment in an introductory programming course

[Abstract]: Peer learning is a powerful pedagogical practice delivering improved outcomes over conventional teacher-student interactions while offering marking relief to instructors. Peer review enables learning by requiring students to evaluate the work of others. PRAISE is an on-line peer-review system that facilitates anonymous review and delivers prompt feedback from multiple sources. This study is an evaluation of the use of PRAISE in an introductory programming course. Use of the system is examined and attitudes of novice programmers towards the use of peer review are compared to those of students from other disciplines, raising a number of interesting issues. Recommendations are made to introductory programming instructors who may be considering peer review in assignments

Feedback Generation for Performance Problems in Introductory Programming Assignments

Providing feedback on programming assignments manually is a tedious, error prone, and time-consuming task. In this paper, we motivate and address the problem of generating feedback on performance aspects in introductory programming assignments. We studied a large number of functionally correct student solutions to introductory programming assignments and observed: (1) There are different algorithmic strategies, with varying levels of efficiency, for solving a given problem. These different strategies merit different feedback. (2) The same algorithmic strategy can be implemented in countless different ways, which are not relevant for reporting feedback on the student program. We propose a light-weight programming language extension that allows a teacher to define an algorithmic strategy by specifying certain key values that should occur during the execution of an implementation. We describe a dynamic analysis based approach to test whether a student's program matches a teacher's specification. Our experimental results illustrate the effectiveness of both our specification language and our dynamic analysis. On one of our benchmarks consisting of 2316 functionally correct implementations to 3 programming problems, we identified 16 strategies that we were able to describe using our specification language (in 95 minutes after inspecting 66, i.e., around 3%, implementations). Our dynamic analysis correctly matched each implementation with its corresponding specification, thereby automatically producing the intended feedback.Comment: Tech report/extended version of FSE 2014 pape

Plan-based delivery composition in intelligent tutoring systems for introductory computer programming

In a shell system for the generation of intelligent tutoring systems, the instructional model that one applies should be variable independent of the content of instruction. In this article, a taxonomy of content elements is presented in order to define a relatively content-independent instructional planner for introductory programming ITS's; the taxonomy is based on the concepts of programming goals and programming plans. Deliveries may be composed by the instantiation of delivery templates with the content elements. Examples from two different instructional models illustrate the flexibility of this approach. All content in the examples is taken from a course in COMAL-80 turtle graphics