Calibration of Confidence Judgments in Elementary Mathematics: Measurement, Development, and Improvement

Self-regulated learning (SRL), the ability to set goals and monitor and control progress toward these goals, is an important part of a positive mathematical disposition. Within SRL, accurate metacognitive monitoring is necessary to drive control processes. Students who display this accuracy are said to be calibrated, and although calibration is a growing area of research within Educational Psychology, unanswered questions remain about the nature of calibration: how it should be measured, its role as a dynamic aspect of metacognition, and how best to improve it. This dissertation uses a rich source of data on student calibration and achievement within an online mathematics curriculum (ST Math) to approach these questions and present results on calibration as representative of a complex system of metacognition.This dissertation presents evidence that calibration is best represented as two separate monitoring processes, one for confidence and one for uncertainty; these processes can be operationalized through the measures of Sensitivity and Specificity. In Study 1, comparisons with other commonly used measures of calibration indicate that Sensitivity and Specificity have a relative robustness to most patterns of missing data and greater strength as predictors. Other commonly used calibration measures suffer greatly from missing data inherent in real-world patterns of question answering. Study 2 characterizes metacognitive monitoring as part of a dynamic system that varies depending on task. In this study, variance in calibration is associated with variance in performance gain within the same student across ST Math quizzes. Both Sensitivity and Specificity are predictors of this gain, but greater confidence when correct (Sensitivity) is more strongly associated with performance gains between quizzes than is greater uncertainty when incorrect (Specificity).Study 3 evaluates the potential of ST Math as a calibration intervention. After a year's practice and feedback with ST Math, students display greater Specificity, but lower Sensitivity, indicating that ST Math made the students more uncertain. Study 3 also explores how change in calibration is related to change in achievement, finding no relation between growth in calibration and growth in achievement, either within or outside of ST Math

Fraction Errors in a Digital Mathematics Environment: Latent Class and Transition Analysis

Student struggles with fractions are well documented, and due to fractions’ importance to later mathematics achievement, identification of the errors students make when solving fraction problems is an area of interest for both researchers and teachers. Within this study, we examine data on student fraction problem errors in pre- and post-quizzes in a digital mathematics environment. Students (n = 1,431) were grouped by prevalence of error types using latent class analysis. Three different classes of error profiles were identified in the pre-quiz data. A latent transition analysis was then used to determine if class membership and class structure changed from pre- to post-quiz. In both pre- and post-quiz, there was a class of students who appeared to be guessing and a class of students who performed well. One class structure was consistent with the idea that early fraction learners rely heavily on whole number principles. Identification of co-occurrence of and changes to fraction errors has implications for curricular design and pedagogical decisions, especially in light of movements toward personalized learning systems

Khan Academy as Supplemental Instruction: A Controlled Study of a Computer-Based Mathematics Intervention

Khan Academy is a large and popular open educational resource (OER) with little empirical study into its impact on student achievement in mathematics when used in schools. In this study, we examined the use of Khan Academy as a mathematics intervention among seventh grade students over a 4-week period versus a control group. We also compared differences between students who had supplemental mathematics instruction and those who had not. In both cases, we found no statistically significant differences in student test scores. Khan Academy has several internal metrics used to track student performance and use. We found significant relationships between these metrics and student test scores in this study. Khan Academy and other OER provide access to information and knowledge to large numbers of the population. This research adds to the discourse methods by which Khan Academy and other OER may affect learners

Predictive Student Modelling in an Online Reading Platform

Use of technology-enhanced education and online learning systems has become more popular, especially after COVID-19. These systems capture a rich array of data as students interact with them. Predicting student performance is an essential part of technology-enhanced education systems to enable the generation of hints and provide recommendations to students. Typically, this is done through use of data on student interactions with questions without utilizing important data on the temporal ordering of students’ other interaction behavior, (e.g., reading, video watching). In this paper, we hypothesize that to predict students’ question performance, it is necessary to (i) consider other learning activities beyond question-answering and (ii) understand how these activities are related to question-solving behavior. We collected middle school physical science students’ data within a K12 reading platform, Actively Learn. This platform provides reading-support to students and collects trace data on their use of the system. We propose a transformer-based model to predict students' question scores utilizing question interaction and reading-related behaviors. Our findings show that integrating question attempts and reading-related behaviors results in better predictive power compared to using only question attempt features. The interpretable visualization of the transformer’s attention can be helpful for teachers to make tailored interventions in students’ learning

Construct Confounding Among Predictors of Mathematics Achievement

Identifying which early mathematics skills have the largest effects on later mathematics achievement has important implications. However, regression-based estimates often rely on untested assumptions: (a) Scores on different mathematics skills reflect unique constructs, and (b) other factors affecting early and later mathematics achievement are fully controlled. We illustrate a process to test these assumptions with a sample of third and fourth graders who completed measures of mathematics skills, working memory and motivation, and standardized mathematics and English language arts tests. Factor analyses indicated that mathematics skills largely reflect the same underlying construct. The skills that loaded highest on the general factor most predicted both later mathematics and English language arts, even after adjusting for working memory and motivation. Findings suggest that relations between earlier mathematics and later achievement largely reflected more general factors that contribute to children’s learning. We discuss the importance of establishing construct validity in correlational studies

Raising the stakes: How students' motivation for mathematics associates with high- and low-stakes test achievement

This study uses data from an urban school district to examine the relation between students' motivational beliefs about mathematics and high- versus low-stakes math test performance. We use ordinary least squares and quantile regression analyses and find that the association between students' motivation and test performance differs based on the stakes of the exam. Students' math self-efficacy and performance avoidance goal orientation were the strongest predictors for both exams; however, students' math self-efficacy was more strongly related to achievement on the low-stakes exam. Students' motivational beliefs had a stronger association at the low-stakes exam proficiency cutoff than they did at the high-stakes passing cutoff. Lastly, the negative association between performance avoidance goals and high-stakes performance showed a decreasing trend across the achievement distribution, suggesting that performance avoidance goals are more detrimental for lower achieving students. These findings help parse out the ways motivation influences achievement under different stakes

Profiles of Instructor Responses to Emergency Distance Learning

To understand instruction during the spring 2020 transition to emergency distance learning (EDL), we surveyed a sample of instructors teaching undergraduate EDL courses at a large university in the southwest. We asked them how frequently they used and how confident they were in their ability to implement each of nine promising practices, both for their spring 2020 EDL course and a time when they previously taught the same course face-to-face (F2F). Using latent class analysis, we examined how behavioral frequencies and confidence clustered to form meaningful groups of instructors, how these groups differed across F2F and EDL contexts, and what predicted membership in EDL groupings. Results suggest that in the EDL context, instructors fell into one of three profiles in terms of how often they used promising practices: Highly Supportive, Instructor Centered, and More Detached. When moving from the F2F to EDL context, instructors tended to shift “down” in terms of their profile—for example, among F2F Highly Supportive instructors, 34% shifted to the EDL Instructor Centered profile and 30% shifted to the EDL More Detached Profile. Instructors who reported lower self-efficacy for EDL practices were also more likely to end up in the EDL More Detached profile. These results can assist universities in understanding instructors' needs in EDL, and what resources, professional development, and institutional practices may best support instructor and student experiences