Cognitive diagnosis modeling and applications to assessing learning

Chapter 1: Cognitive diagnosis models (CDMs) are restricted latent class models designed to assess test takers' mastery on a set of skills or attributes. With a wide range of applications in education and in psychopathology, various CDMs have been proposed and fitted to response data from different scenarios. Recently, Xu (2017) derived sufficient conditions for identifying model parameters of a restricted latent class model, which generalizes many existing CDMs. We propose a Bayesian estimation algorithm for this restricted latent class model. The model is applied to the Examination for the Certificate of Proficiency in English language assessment data (e.g., Henson et al., 2007). Chapter 2: There has been a growing interest in measuring students' growth over time. CDMs were traditionally used to measure students' skill mastery at a static time point, but recently, they have been used in many longitudinal models to track students' changes in skill acquisition over time. In this chapter, we propose a longitudinal learning model, where different kinds of skill hierarchies were considered, and the reduced-reparameterized unified model (r-RUM) or the noisty input, deterministic-``and''-gate (NIDA) model is used to measure students' skill mastery at each time point. This model is fitted to the Spatial Rotation data set (e.g., Wang et al., 2016), and different models were compared using Bayesian model comparison methods. Chapter 3: The increased popularity of computer-based testing has enabled researchers to collect various types of process data, including response times. Extensive research has been conducted on the joint modeling of response accuracy and response times. Recent research on CDMs begins to explore the relationship between speed and accuracy to understand students’ fluency of applying the mastered skills, in addition to mastery information, in a learning environment. In this chapter, we propose a mixture hidden Markov Diagnostic Classification Model framework for learning with response times and response accuracy. Such a model accounts for the heterogeneities in learning styles among students by modeling the different learning and response behaviors among subgroups. The proposed model is evaluated through a simulation study in terms of parameter recovery. Chapter 4: We introduce an R package, hmcdm, that can be used to fit several longitudinal models for learning under the cognitive diagnosis framework. The package allows users to simulate item responses (and response times if applicable) under several learning models, to fit the models using Markov Chain Monte Carlo (MCMC) methods, to compute point estimates of parameters based on the MCMC samples, and to evaluate and compare different models using Deviance Information Criterion and posterior predictive probabilities

Evaluating NLG Evaluation Metrics: A Measurement Theory Perspective

We address the fundamental challenge in Natural Language Generation (NLG) model evaluation, the design and validation of evaluation metrics. Recognizing the limitations of existing metrics and issues with human judgment, we propose using measurement theory, the foundation of test design, as a framework for conceptualizing and evaluating the validity and reliability of NLG evaluation metrics. This approach offers a systematic method for defining "good" metrics, developing robust metrics, and assessing metric performance. In this paper, we introduce core concepts in measurement theory in the context of NLG evaluation and key methods to evaluate the performance of NLG metrics. Through this framework, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics, ultimately contributing to the advancement of robust and effective NLG models in real-world settings

Modeling Learner Heterogeneity: A Mixture Learning Model With Responses and Response Times

The increased popularity of computer-based testing has enabled researchers to collect various types of process data, including test takers' reaction time to assessment items, also known as response times. In recent studies, the relationship between speed and accuracy in a learning setting was explored to understand students' fluency changes over time in applying the mastered skills in addition to skill mastery. This can be achieved by modeling the changes in response accuracy and response times throughout the learning process. We propose a mixture learning model that utilizes the response times and response accuracy. Such a model accounts for the heterogeneities in learning styles among learners and may provide instructors with valuable information, which can be used to design individualized instructions. A Bayesian modeling framework is developed for parameter estimation and the proposed model is evaluated through a simulation study and is fitted to a real data set collected from a computer-based learning system for spatial rotation skills

Bacillus megaterium BMJBN02 induces the resistance of grapevine against downy mildew

Grape downy mildew caused by Plasmopara viticola is one of the most destructive diseases of grapes. All grape cultivars are susceptible to P. viticola. However, the resistance of grape plants could be induced in plant defense with some help of microbes. In this study, Bacillus megaterium BMJBN02 obtained from farmland soil was shown to regulate the resistance of grapevine against downy mildew. The salicylic acid (SA) content and the expression of pathogenesis-related (PR) genes of grapes under different treatments were examined using high-performance liquid chromatography-mass spectrometry (HPLC-MS) and reverse transcription- quantitative polymerase chain reaction (RT-qPCR), and it was found that SA content and the expression of PR genes could play a role in regulating the resistance of grapevine against downy mildew. The five-year plot experiment showed that the resistance effectiveness of isolate BMJBN02 was approximately equal to that of 0.1 % nicotinyl morpholine (commercial fungicide). Therefore, this study provides a valuable candidate method that uses B. megaterium BMJBN02 by regulating the resistance of grape against downy mildew for quality and yield of grape in commercial productivity

ProcData: An R Package for Process Data Analysis

Process data refer to data recorded in the log files of computer-based items. These data, represented as timestamped action sequences, keep track of respondents' response processes of solving the items. Process data analysis aims at enhancing educational assessment accuracy and serving other assessment purposes by utilizing the rich information contained in response processes. The R package ProcData presented in this article is designed to provide tools for processing, describing, and analyzing process data. We define an S3 class "proc" for organizing process data and extend generic methods summary and print for class "proc". Two feature extraction methods for process data are implemented in the package for compressing information in the irregular response processes into regular numeric vectors. ProcData also provides functions for fitting and making predictions from a neural-network-based sequence model. These functions call relevant functions in package keras for constructing and training neural networks. In addition, several response process generators and a real dataset of response processes of the climate control item in the 2012 Programme for International Student Assessment are included in the package

Identifying student profiles in a digital mental rotation task: insights from the 2017 NAEP math assessment

Mental rotation (MR), a key aspect of spatial reasoning, is highly predictive of success in STEM fields. This study analyzed strategies employed by 27,600 eighth-grade students during a digital MR task from the 2017 National Assessment of Educational Progress (NAEP) in mathematics. Utilizing K-means cluster analysis to categorize behavioral and performance patterns, we identified four distinct profiles: Cognitive Offloaders (15% of the sample), Internal Visualizers (55%), External Visualizers (5%), and Non-Triers (25%). Cognitive Offloaders, skilled at minimizing cognitive load by eliminating incorrect options, demonstrated the highest MR accuracy rates at 45%. Internal Visualizers, relying less on digital tools and more on mental strategies, achieved robust performance with an average score of 38%. External Visualizers, despite their extensive use of assistive tools and greater time investment, scored an average of 36%. Non-Triers showed minimal engagement and correspondingly the lowest performance, averaging 29%. These findings not only underscore the diverse strategies students adopt in solving MR tasks but also emphasize the need for educational strategies that are tailored to accommodate different cognitive styles. By integrating MR training into the curriculum and enhancing teacher preparedness to support diverse learning needs, this study advocates for educational reforms to promote equitable outcomes in mathematics and broader STEM fields