Evaluating and improving adaptive educational systems with learning curves

Personalised environments such as adaptive educational systems can be evaluated and compared using performance curves. Such summative studies are useful for determining whether or not new modifications enhance or degrade performance. Performance curves also have the potential to be utilised in formative studies that can shape adaptive model design at a much finer level of granularity. We describe the use of learning curves for evaluating personalised educational systems and outline some of the potential pitfalls and how they may be overcome. We then describe three studies in which we demonstrate how learning curves can be used to drive changes in the user model. First, we show how using learning curves for subsets of the domain model can yield insight into the appropriateness of the model’s structure. In the second study we use this method to experiment with model granularity. Finally, we use learning curves to analyse a large volume of user data to explore the feasibility of using them as a reliable method for fine-tuning a system’s model. The results of these experiments demonstrate the successful use of performance curves in formative studies of adaptive educational systems

Evaluating the effects of adaptively presenting worked examples, erroneous examples and problem solving in a constraint-based tutor.

Learning from Problem Solving (PS), Worked Examples (WE) and Erroneous Examples (ErrEx) have all been proven to be effective learning strategies in Intelligent Tutoring Systems. A worked example consists of a problem statement, its solution, and additional explanations, and therefore provides a high level of assistance to students. Many studies have shown the benefits of learning from WEs and PS in ITSs. An erroneous example (ErrEx) presents an incorrect solution and requires students to find and correct errors, therefore helping the student to solve problems. Erroneous examples may also help students become better at evaluating problem solutions. In this project, we aim to investigate how to maximize learning by adaptively providing learning activities for students based on their performance in the domain of Structured Query Language (SQL). The project was conducted in the context of SQL-Tutor, which is a constraint-based tutor that teaches SQL. A series of studies conducted during the project produced promising results. Our first study demonstrated that a fixed sequence of WE/PS pairs and ErrEx/PS pairs (WPEP) resulted in improved problem solving and that it also benefitted students with different levels of prior SQL knowledge. We then introduced an adaptive strategy in the second study, which decided what learning activities (WE, ErrEx with one or two errors, or PS) to provide to the student based on his/her performance on problem solving. We found that students who studied with the adaptive strategy improved their post-test scores on conceptual, procedural, and debugging questions (i.e., analyzing the solution, explaining the errors, and then making appropriate corrections) with significantly fewer learning activities. The final study compared the enhanced adaptive strategy to the self-selection strategy, as well as compared the enhanced adaptive strategy to the original adaptive strategy from the second study. The results show that the enhanced adaptive strategy is superior to the self-selection strategy. However, the original adaptive strategy was the better choice compared to the enhanced adaptive strategy, for students with varying levels of prior knowledge

The effect of adapting feedback generality in ITS

Intelligent tutoring systems achieve much of their success by adapting to individual students. One potential avenue for personalization is feedback generality. This paper presents two evaluation studies that measure the effects of modifying feedback generality in a web-based Intelligent Tutoring System (ITS) based on the analysis of student models. The object of the experiments was to measure the effectiveness of varying feedback generality, and to determine whether this could be performed en masse or if personalization is needed. In an initial trial with a web-based ITS it appeared that it is feasible to use a mass approach to select appropriate concepts for generalizing feedback. A second study gave conflicting results and showed a relationship between generality and ability, highlighting the need for feedback to be personalized to individual students’ needs

The Effect of Adapting Feedback Generality in ITS

Abstract. Intelligent tutoring systems achieve much of their success by adapting to individual students. One potential avenue for personalization is feedback generality. This paper presents two evaluation studies that measure the effects of modifying feedback generality in a web-based Intelligent Tutoring System (ITS) based on the analysis of student models. The object of the experiments was to measure the effectiveness of varying feedback generality, and to determine whether this could be performed en masse or if personalization is needed. In an initial trial with a web-based ITS it appeared that it is feasible to use a mass approach to select appropriate concepts for generalizing feedback. A second study gave conflicting results and showed a relationship between generality and ability, highlighting the need for feedback to be personalized to individual students ’ needs.