Visualizaciones del análisis temporal del aprendizaje para aumentar el conocimiento durante la evaluación

Les representacions visuals de dades de traces generades per l’alumnat durant les activitats d’aprenentatge ajuden tant els estudiants com els professors a interpretar-les intuïtivament i a percebre’n amb rapidesa aspectes amagats. En aquest treball descrivim la visualització de dades de traces temporals durant el procés d’avaluació. L’estudi tenia un doble objectiu: a) descriure la implicació dels estudiants en el procés d’avaluació pel que fa a temps esmerçat i factors temporals associats amb característiques concretes de l’aprenentatge, i b) explorar els factors que influeixen en la intenció comportamental del professorat quant a emprar el sistema proposat com a sistema d’informació i les seves percepcions de l’efectivitat i l’acceptació del nostre enfocament. Les visualitzacions proposades s’han examinat en un estudi amb 32 professors d’ensenyament secundari. Vàrem adoptar una metodologia de recerca basada en el disseny i vàrem utilitzar un instrument d’enquesta –basada en el model d’acceptació de l’anàlisi de l’aprenentatge– per mesurar l’impacte esperat de les visualitzacions proposades. L’anàlisi de les troballes indica que a) els factors temporals es poden utilitzar per visualitzar el comportament dels estudiants durant l’avaluació, i b) la visualització de la dimensió temporal del comportament dels estudiants augmenta el coneixement del professor pel que fa al progrés dels alumnes, a possibles conceptes erronis (per exemple, endevinar la resposta correcta) i a les dificultats de la tasca.Visual representations of student-generated trace data during learning activities help both students and instructors interpret them intuitively and perceive hidden aspects of these data quickly. In this paper, we elaborate on the visualization of temporal trace data during assessment. The goals of the study were twofold: a) to depict students’ engagement in the assessment procedure in terms of time spent and temporal factors associated with learning-specific characteristics, and b) to explore the factors that influence the teachers’ Behavioural Intention to use the proposed system as an information system and their perceptions of the effectiveness and acceptance of our approach. The proposed visualizations have been explored in a study with 32 Secondary Education teachers. We adopted a design-based research methodology and employed a survey instrument – based on the Learning Analytics Acceptance Model (LAAM) – in order to measure the expected impact of the proposed visualizations. The analysis of the findings indicates that a) temporal factors can be used for visualizing students’ behaviour during assessment, and b) the visualization of the temporal dimension of students’ behaviour increases teachers’ awareness of students’ progress, possible misconceptions (e.g., guessing the correct answer) and task difficulty. Las representaciones visuales de datos de trazas generados por el alumnado durante las actividades de aprendizaje ayudan tanto a los estudiantes como a los profesores a interpretarlos intuitivamente y a percibir con rapidez aspectos ocultos. En este trabajo, describimos la visualización de datos de trazas temporales durante la evaluación. El estudio tenía un doble objetivo: a) describir la implicación de los estudiantes en el proceso de evaluación en cuanto a tiempo invertido y factores temporales asociados con características concretas del aprendizaje, y b) explorar los factores que influyen en la intención comportamental del profesorado en cuanto a emplear el sistema propuesto como sistema de información y sus percepciones de la efectividad y la aceptación de nuestro enfoque. Las visualizaciones propuestas se han examinado en un estudio con 32 profesores de educación secundaria. Adoptamos una metodología de investigación basada en el diseño y utilizamos un instrumento de encuesta –basada en el modelo de aceptación del análisis del aprendizaje– para medir el impacto esperado de las visualizaciones propuestas. El análisis de los hallazgos indica que a) los factores temporales se pueden utilizar para visualizar el comportamiento de los estudiantes durante la evaluación, y b) la visualización de la dimensión temporal del comportamiento de los estudiantes aumenta el conocimiento del profesor respecto al progreso de los alumnos, posibles conceptos erróneos (por ejemplo, adivinar la respuesta correcta) y dificultad de la tarea.

General Features in Knowledge Tracing to Model Multiple Subskills, Temporal Item Response Theory, and Expert Knowledge

Knowledge Tracing is the de-facto standard for inferring student knowledge from performance data. Unfortunately, it does not allow modeling the feature-rich data that is now possible to collect in modern digital learning environments. Because of this, many ad hoc Knowledge Tracing variants have been proposed to model a specific feature of interest. For example, variants have studied the effect of students’ individual characteristics, the effect of help in a tutor, and subskills. These ad hoc models are successful for their own specific purpose, but are specified to only model a single specific feature. We present FAST (Feature Aware Student knowledge Tracing), an efficient, novel method that allows integrating general features into Knowledge Tracing. We demonstrate FAST’s flexibility with three examples of feature sets that are relevant to a wide audience. We use features in FAST to model (i) multiple subskill tracing, (ii) a temporal Item Response Model implementation, and (iii) expert knowledge. We present empirical results using data collected from an Intelligent Tutoring System. We report that using features can improve up to 25% in classification performance of the task of predicting student performance. Moreover, for fitting and inferencing, FAST can be 300 times faster than models created in BNT-SM, a toolkit that facilitates the creation of ad hoc Knowledge Tracing variants

Computational Psychometrics for Item-based Computerized Adaptive Learning

With advances in computer technology and expanded access to educational data, psychometrics faces new opportunities and challenges for enhancing pattern discovery and decision-making in testing and learning. In this dissertation, I introduced three computational psychometrics studies for solving the technical problems in item-based computerized adaptive learning (CAL) systems related to dynamic measurement, diagnosis, and recommendation based on Bayesian item response theory (IRT). For the first study, I introduced a new knowledge tracing (KT) model, dynamic IRT (DIRT), which can iteratively update the posterior distribution of latent ability based on moment match approximation and capture the uncertainty of ability change during the learning process. For dynamic measurement, DIRT has advantages in interpretation, flexibility, computation cost, and implementability. For the second study, A new measurement model, named multilevel and multidimensional item response theory with Q matrix (MMIRT-Q), was proposed to provide fine-grained diagnostic feedback. I introduced sequential Monte Carlo (SMC) for online estimation of latent abilities. For the third study, I proposed the maximum expected ratio of posterior variance reduction criterion (MERPV) for testing purposes and the maximum expected improvement in posterior mean (MEIPM) criterion for learning purposes under the unified framework of IRT. With these computational psychometrics solutions, we can improve the students’ learning and testing experience with accurate psychometrics measurement, timely diagnosis feedback, and efficient item selection

Rule-based item construction.:Analysis with and comparison of linear logistic test models and cognitive diagnostic models with two item types

Die Dissertation behandelt die Darstellung und Evaluierung regelgeleiteter Aufgabenkonstruktion am Beispiel von figuralen Reasoning- und mathematischen Textaufgaben sowie die Anwendung und den Vergleich von linear logistischen Testmodellen (LLTMs) und Kognitiven Diagnosemodellen (CDMs) als statistische Analysemethoden. Die Ergebnisse zeigen Rasch-Skalierbarkeit der Aufgaben und demonstrieren einen präzisen Aufgabenkonstruktions- und Analyseprozess. Die LLTM-Varianten liefern wichtige Einblicke in kognitive Lösungsprozesse und in die Zusammensetzung der Aufgabenschwierigkeit für beide Aufgabentypen genauso wie für einen implementierten Aufgabencloning-Ansatz und longitudinale Datenstrukturen. In der CDM-Anwendung zeigen sich erhebliche Modellierungsprobleme und Unangemessenheit des Ansatzes für die vorliegenden Aufgabenbeispiele. Hinweise bezüglich der Aufgabenkonstruktion, der statistischen Modelle und der Interpretation der Ergebnisse für Anwendung und Forschung werden herausgestellt. The dissertation focuses on demonstration and evaluation of rule-based item construction of figural reasoning items and mathematical word problems and application as well as comparison of LLTMs and CDMs as statistical analysis methods. Results show Rasch scalability of items, confirm the importance of the chosen basic parameter sets and demonstrate precise item construction and analysis processes. It is shown how LLTM and its variants can contribute substantial insights into cognitive solution processes and composition of item difficulty in relational reasoning and mathematical word problems and also for item cloning and longitudinal data. However, CDM application detects severe modeling problems and misfit. Application hints regarding test item construction as well as statistical model application and interpretation of results for practitioners and researchers are pointed out

When practice does not make perfect: Differentiating between productive and unproductive persistence

Research has suggested that persistence in the face of challenges plays an important role in learning. However, recent work on wheel-spinning—a type of unproductive persistence where students spend too much time struggling without achieving mastery of skills—has shown that not all persistence is uniformly beneficial for learning. For this reason, Study 1 used educational data-mining techniques to determine key differences between the behaviors associated with productive persistence and wheel-spinning in ASSISTments, an online math learning platform. This study’s results indicated that three features differentiated between these two modes of persistence: the number of hints requested in any problem, the number of bottom-out hints in the last eight problems, and the variation in the delay between solving problems of the same skill. These findings suggested that focusing on number of hints can provide insight into which students are struggling, and encouraging students to engage in longer delays between problem solving is likely helpful to reduce their wheel-spinning. Using the same definition of productive persistence in Study 1, Study 2 attempted to investigate the relationship between productive persistence and grit using Duckworth and Quinn’s (2009) Short Grit Scale. Correlational results showed that the two constructs were not significantly correlated with each other, providing implications for synthesizing literature on student persistence across computer-based learning environments and traditional classrooms

Implementation of a diagnostic classification model for middle-school physics

This dissertation provides a start-to-finish description of development, administration, and validation for an online middle-school physics test using a DCM framework with response-time. The first paper illustrated the process of implementing DCM with a careful selection of the content domain and a simulation approach for a Q-matrix construction. The results were promising despite some items that showed inadequate fit and quality. The second paper is a narration of a step-by-step validation process for the effects of the DCM-scored physics assessment on learning and teaching. While evidence was found to support multiple validity arguments and the usefulness of the diagnostic feedback, validity threats were also found because one of the students identified multiple strategies in solving some of the questions. The third paper investigates the potential benefits of incorporating response-time into the diagnostic model. Although the response-time variable did not improve the classification estimation in this case, different types of relations between the ability, time variable, or other ancillary variables should be investigated in the future. Although limitations were found in the dissertation, multiple actions could be taken to refine this process in future research, and the process could still be generalized into other domains and as guidelines for researchers and educators interested in DCM application