The Utility of Clustering in Prediction Tasks

We explore the utility of clustering in reducing error in various prediction tasks. Previous work has hinted at the improvement in prediction accuracy attributed to clustering algorithms if used to pre-process the data. In this work we more deeply investigate the direct utility of using clustering to improve prediction accuracy and provide explanations for why this may be so. We look at a number of datasets, run k-means at different scales and for each scale we train predictors. This produces k sets of predictions. These predictions are then combined by a na\"ive ensemble. We observed that this use of a predictor in conjunction with clustering improved the prediction accuracy in most datasets. We believe this indicates the predictive utility of exploiting structure in the data and the data compression handed over by clustering. We also found that using this method improves upon the prediction of even a Random Forests predictor which suggests this method is providing a novel, and useful source of variance in the prediction process.Comment: An experimental research report, dated 11 September 201

A quasi-experimental evaluation of an on-line formative assessment and tutoring system.

ASSISTment is a web-based math tutor designed to address the need for timely student assessment while simultaneously providing instruction, thereby avoiding lost instruction time that typically occurs during assessment. This paper presents a quasi-experiment that evaluates whether ASSISTment use has an effect on improving middle school students\u27 year-end test scores. The data was collected from 1240 seventh graders in three treatment schools and one comparison school. Posttest (7th grade year-end test) results indicate, after adjusting for the pretest (6th grade year-end test), that students in the treatment schools significantly outperformed students in the comparison school and the difference was especially present for special education students. A usage analysis reveals that greater student use of ASSISTments is associated with greater learning consistent with the hypothesis that it is useful as a tutoring system. We also 2 found evidence consistent with the hypothesis that teachers adapt their whole class instruction based on overall student performance in ASSISTments. Namely, increased teacher use (i.e., having more students use the system more often) is associated with greater learning among students with little or no use suggesting that those students may have benefited from teachers adapting their whole-class instruction based on what they learned from ASSISTment use reports. These results indicate potential for using technology to provide students instruction during assessment and to give teachers fast and continuous feedback on student progress

Towards Enabling Collaboration in Intelligent Tutoring Systems

Intelligent Tutoring Systems have historically been shown [Koedinger, Anderson, Hadley & Mark, 1997; Morgan & Ritter, 2002] to be an effective means of educating an audience. While there is great benefit from such systems they are generally very costly to build and maintain. Currently it is estimated that 200 hours of time is required to produce one hour of Intelligent Tutoring Systems content [Murray, 2002; Anderson, 1993]. For Intelligent Tutoring Systems to be widely accepted in the classroom environment there needs to be a tool set that allows for the most novice user to maintain and grow the system with minimal cost. The goal of this work is to create such a tool set targeted towards the Assistments Project [Razzaq, Feng, Nuzzo-Jones, Heffernan, & et. al., 2005] and enable teacher collaboration within the system. A goal of the Assistments Project is to provide a means for teachers to obtain meaningful data from the system that they utilize in the classroom environment thus enabling a comprehensive learning solution

Knowledge Engineering for Intelligent Tutoring Systems: Using machine learning assistance to help humans tag questions to skills based upon the words in the questions.

Building a mapping between items and their related knowledge components, while difficult and time consuming, is central to the task of developing affective intelligent tutoring systems. Improving performance on this task by creating a semi-automatic skill encoding system would facilitate the development of such systems. The goal of this project is to explore techniques involved in text classification to the end of improving the time required to correctly tag items with their associated skills

Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems

A major issue in Intelligent Tutoring Systems is off-task student behavior, especially performance-based gaming, where students systematically exploit tutor behavior in order to advance through a curriculum quickly and easily, with as little active thought directed at the educational content as possible. The goal of this research was to develop a passive visual indicator to deter and prevent off-task gaming behavior without active intervention, via graphical feedback to the student and teachers. Traditional active intervention approaches were also constructed for comparison purposes. Our passive graphical intervention has been well received by teachers, and results suggest that this technique is effective at reducing off-task gaming behavior

A Methodology for Evaluating Predictions of Transfer and an Empirical Application to Data from a Web-Based Intelligent Tutoring System: How to Improve Knowledge Tracing in Dialog Based Tutors

Cognitive Science is interested in being able to develop methodologies for analyzing human learning and performance data. Intelligent tutoring systems need good cognitive models that can predict student performance. Cognitive models of human processing are also useful in tutoring because well-designed curriculums need to understand the common components of knowledge that students need to be able to employ (cite Koedinger paper and algebra stuff). A common concern is being able to predict when transfer should happen. We describe a methodology (first used by Koedinger, 2001) that uses empirical data and cognitively principled task analysis to evaluate the fit of cognitive models. This methodology seems particularly useful when you are trying to find evidence for “hidden” knowledge components that are hard to assess because they are confounded with accessing other knowledge components. We present this methodology as well as an illustration showing how we are trying to use this method to answer an important cognitive science issue

Increasing parent engagement in student learning using an Intelligent Tutoring System

In this paper we present the results of a study that demonstrates the ability of an Intelligent Tutoring System (ITS) to increase parental engagement in student learning. We developed a parent notification feature for the web-based ASSISTment ITS that allows parents to log into their own accounts and access detailed data about their students’ performance. We then invited parents from a local middle school to create accounts and answer a survey assessing how engaged they felt they were in their students’ education. We ran a 60 day study during which we sent messages home to parents regarding what their students were studying in class and how they were performing. After having them take a post-survey, we found that parents felt significantly more engaged in their students’ education. Additionally, the messages significantly increased how frequently parents logged in to check reports on their students’ performance data using the ASSISTment system. Qualitative feedback from both parents and teachers was extremely positive

Using Learning Decomposition and Bootstrapping with Randomization to Compare the Impact

Abstract. A basic question of instructional interventions is how effective it is in promoting student learning. This paper presents a study to determine the relative efficacy of different instructional strategies by applying an educational data mining technique, learning decomposition. We use logistic regression to determine how much learning is caused by different methods of teaching the same skill, relative to each other. We compare our results with a previous study, which used classical analysis techniques and reported no main effect. Our results show that there is a marginal difference, suggesting giving students scaffolding questions is less effective at promoting student learning than providing them delayed feedback. Our study utilizes learning decomposition, an easier and quicker approach of evaluating the quality of ITS interventions than experimental studies. We also demonstrate the usage of computer-intensive approach, bootstrapping, for hypothesis testing in educational data mining area.

Cost-Effective Content Creation with Variabilization

Traditional intelligent tutoring systems have been successful at fostering learning, but very few systems have been built due to the high cost of creation. It has been reported that it takes between 100 to 1000 hours to produce a single hour of tutoring content. Our previous research reported a reduction in the cost of authoring content through the use of pseudo-tutors, constructs that mimic cognitive tutors but are limited in scope to a single problem. Although the extreme reduction in complexity allowed teachers with no background in intelligent tutoring systems to build effective tutoring content, building multiple questions within a particular skill set required significant repetition of content. In the current work, we add some complexity back into the system by allowing teachers to generalize pseudo-tutors through the use of variables that can alter the contextual and numerical data used in the problem.We report evidence that variabilization reduces the cost of authoring similar skill problems by a factor of two. Further, this factor increases linearly with the number of instances of the problem created. We also suggest that the additional complexity is not a hindrance to teachers adopting the system and some repetition of tutoring content is acceptable to students

Can we insist students reach proficiency on homework? Yes!

This study involved a comparison between the conditions of Mastery Mode against Non-Mastery. In the Mastery Mode, selected problems were mastery problems. A student, who got an incorrect answer, went into the Mastery Mode in which he/she had to get three consecutive problems correct testing the same skill. Although this process took long, it forced the student to master the subject matter. In the Non-Mastery Mode, the students were given two chances for each problem while no tutoring was provided. It was observed through the results that there was a significant difference between the Mastery Mode and Non-Mastery with the p-value of 0.003. The effect of the study was towards the mastery mode and students learned significantly more in this condition with an effect size of 0.52. The implication of this experiment is that mastery learning is an efficient technique which can be incorporated in homework to make students put effort in learning the content at different time intervals, thus, increasing the overall learning gain