A metacognitive feedback scaffolding system for pedagogical apprenticeship

This thesis addresses the issue of how to help staff in Universities learn to give feedback with the main focus on helping teaching assistants (TAs) learn to give feedback while marking programming assignments. The result is an innovative approach which has been implemented in a novel computer support system called McFeSPA. The design of McFeSPA is based on an extensive review of the research literature on feedback. McFeSPA has been developed based on relevant work in educational psychology and Artificial Intelligence in EDucation (AIED) e.g. scaffolding the learner, ideas about andragogy, feedback patterns, research into the nature and quality of feedback and cognitive apprenticeship. McFeSPA draws on work on feedback patterns that have been proposed within the Pedagogical Patterns Project (PPP) to provide guidance on structuring the feedback report given to the student by the TA. The design also draws on the notion of andragogy to support the TA. McFeSPA is the first Intelligent Tutoring System (ITS) that supports adults learning to help students by giving quality feedback. The approach taken is more than a synthesis of these key ideas: the scaffolding framework has been implemented both for the domain of programming and the feedback domain itself; the programming domain has been structured for training TAs to give better feedback and as a framework for the analysis of students’ performance. The construction of feedback was validated by a small group of TAs. The TAs employed McFeSPA in a realistic situation that was supported by McFeSPA which uses scaffolding to support the TA and then fade. The approach to helping TAs become better feedback givers, which is instantiated in McFeSPA, has been validated through an experimental study with a small group of TAs using a triangulation approach. We found that our participants learned differently by using McFeSPA. The evaluation indicates that 1) providing content scaffolding (i.e. detailed feedback about the content using contingent hints) in McFeSPA can help almost all TAs increase their knowledge/understanding of the issues of learning to give feedback; 2) providing metacognitive scaffolding (i.e. each level of detailed feedback in contingent hint, this can also be general pop-up messages in using the system apart from feedback that encourage the participants to give good feedback) in McFeSPA helped all TAs reflect on/rethink their skills in giving feedback; and 3) when the TAs obtained knowledge about giving quality feedback, providing adaptable fading of TAs using McFeSPA allowed the TAs to learn alone without any support

Recognising the design decisions in Prolog programs as a prelude to critiquing

This thesis presents an approach by which an automated teaching system can analyse the design of novices' Prolog programs for tutorial critiquing. Existing methodologies for tutorial analysis of programs focus on the kind of small pro¬ gramming examples that are used only in the early stages of teaching. If an automated teaching system is to be widely useful, it must cover a substantial amount of the teaching syllabus, and a critiquing system must be able to analyse and critique programs written during the later stages of the syllabus.The work is motivated by a study of students' Prolog programs which were written as assessed exercises towards the end of their course. These programs all work (in some sense), yet they reveal a wide range of design (laws (bodges) for which some form of tutoring would be useful. They present problems for any automated analysis in terms of the size of the programs, the number of individual decisions that must be made to create each program and the range of correct and incorrect decisions that may be made in each case.This study identifies two areas in the analysis of students' program in which further work is needed. Existing work has focussed only on the design and implementation decisions that relate closely to the programming language. That is not sufficient for these slightly more advanced programs, for which decisions in the problem domain must also be recognised. Existing work has focussed on the different ways to implement code, but in these programs the students also make decisions about which data structures are to be used. These decisions must also be part of an analysis.The thesis provides an approach which represents both decisions in the domain of the problem being solved and decisions about how to implement them in Prolog. Decisions in the problem domain are represented by tasks (for code) and by domain objects (for data structures). Decisions that are specific to the Prolog implementation are represented by prototypes which encapsulate standard programming techniques (for code) and by a polymorphic data type language (for data structures). Issues in devising these representations are discussed.An analysis-by synthesis approach is used for code recognition. This is aug¬ mented by a procedure called "clausal split" which isolates novel or poorly de¬ signed parts of an implementation. Following an incomplete analysis of the program by synthesis, the results of this analysis provide the basis for making inferences about the parts of the program that have not been understood. For analysing data structures, a type inference mechanism is combined with inference about the parts of domain objects. Inferred data type information is also used to limit search, both for synthesis and analysis.An architecture using this approach has been implemented. The success of the architecture is assessed on student's programs. From this assessment it is clear that much further work remains to be done, but the results are hopeful

An application of formal semantics to student modelling : an investigation in the domain of teaching Prolog

This thesis reports on research undertaken in an exploration of the use of formal semantics for student modelling in intelligent tutoring systems. The domain chosen was that of tutoring programming languages and within that domain Prolog was selected to be the target language for this exploration. The problem considered is one of how to analyse students' errors at a level which allows diagnosis to be more flexible and meaningful than is possible with the 'mal-rules' and 'bugcatalogue' approach of existing systems. The ideas put forward by Robin Milner [1980] in his Calculus of Communicating Systems (CCS) form the basis of the formalism which is proposed as a solution to this problem. Based on the findings of an empirical investigation, novices' misconceptions of control flow in Prolog was defined as a suitable area in which to explore the application of this solution. A selection of Prolog programs used in that investigation was formally described in terms of CCS. These formal descriptions were used by a production rule system to generate a number of the incomplete or faulty models of Prolog execution which were identified in the first empirical study. In a second empirical study, a machine-analysis tool, designed to be part of a diagnostic tutoring module, used these models to diagnose students' misconceptions of Prolog control flow. This initial application of CCS to student modelling showed that the models of Prolog execution generated by the system could be used successfully to detect students' misunderstandings. Results from the research reported here indicate that the use of formal semantics to model programming languages has a useful contribution to make to the task of student modelling

Hint generation in programming tutors

Programming is increasingly recognized as a useful and important skill. Online programming courses that have appeared in the past decade have proven extremely popular with a wide audience. Learning in such courses is however not as effective as working directly with a teacher, who can provide students with immediate relevant feedback. The field of intelligent tutoring systems seeks to provide such feedback automatically. Traditionally, tutors have depended on a domain model defined by the teacher in advance. Creating such a model is a difficult task that requires a lot of knowledgeengineering effort, especially in complex domains such as programming. A potential solution to this problem is to use data-driven methods. The idea is to build the domain model by observing how students have solved an exercise in the past. New students can then be given feedback that directs them along successful solution paths. Implementing this approach is particularly challenging for programming domains, since the only directly observable student actions are not easily interpretable. We present two novel approaches to creating a domain model for programming exercises in a data-driven fashion. The first approach models programming as a sequence of textual rewrites, and learns rewrite rules for transforming programs. With these rules new student-submitted programs can be automatically debugged. The second approach uses structural patterns in programs’ abstract syntax trees to learn rules for classifying submissions as correct or incorrect. These rules can be used to find erroneous parts of an incorrect program. Both models support automatic hint generation. We have implemented an online application for learning programming and used it to evaluate both approaches. Results indicate that hints generated using either approach have a positive effect on student performance

Ask-Elle:an Adaptable Programming Tutor for Haskell Giving Automated Feedback

Ask-Elle is a tutor for learning the higher-order, strongly-typed functional programming language Haskell. It supports the stepwise development of Haskell programs by verifying the correctness of incomplete programs, and by providing hints. Programming exercises are added to Ask-Elle by providing a task description for the exercise, one or more model solutions, and properties that a solution should satisfy. The properties and model solutions can be annotated with feedback messages, and the amount of flexibility that is allowed in student solutions can be adjusted. The main contribution of our work is the design of a tutor that combines (1) the incremental development of different solutions in various forms to a programming exercise with (2) automated feedback and (3) teacher-specified programming exercises, solutions, and properties. The main functionality is obtained by means of strategy-based model tracing and property-based testing. We have tested the feasibility of our approach in several experiments, in which we analyse both intermediate and final student solutions to programming exercises, amongst others

Expert-Generated and Auto-Generated Socratic Tutoring Systems For Code Comprehension

Programming skills are a vital part of many disciplines but can be challenging to teach and learn. Thus, the programming courses are considered difficult and a major stumbling block. To overcome these challenges, students could benefit from extensive individual support such as tutoring, but there are simply not enough qualified tutors available to meet rising demands.A potential solution is the development of intelligent tutoring systems (ITSs), which offer individualized, one-on-one instruction. Such systems can offer the support to make programming instruction more effective, scalable and reduce existing teachers\u27 workloads.This dissertation demonstrates how conversational ITSs and the Socratic method of teaching can improve a novice\u27s understanding of programming concepts and, in particular, the scaffolding of code comprehension processes. Furthermore, this work provides a novel method to automatically author a Socratic dialogue-based ITS. Indeed, two major outcomes of this work are a Socratic dialogue-based ITS and an automated dialogue authoring tool, which generates full Socratic dialogue from Java source code.The key objectives of this dissertation were, first, to determine whether the Socratic method would be effective at eliciting learners to engage in self-explanations with the help of the Socratic Tutor ITS and, second, to assess the quality of Socratic Author\u27s auto-generated tutorial dialogue. Thus, the work presented here sought to answer two main research questions: (1) can a Socratic ITS lead to improved code comprehension? and (2) to what extent can Socratic dialogue be generated automatically?In sum, this research helps establish a relationship between code comprehension and the use of the Socratic method in learning computer programming. Furthermore, the work introduces a novel approach for generating Socratic dialogue from source code with examples for the Java programming language. The auto-authoring tool could help teachers and ITS developers create tutorial dialogues automatically from Java code without requiring nondomain knowledge. To the best of our knowledge, no such auto-generation of tutorial dialogues from source code has been done before and thus constituting a premiere