A compensatory model for simultaneously setting cutting scores for selection-placement-mastery decisions

A method is proposed for optimizing cutting scores for a selection-placement-mastery problem simultaneously. A simultaneous approach has two advantages over separate optimization. First, test scores used in previous decisions can be used as "prior data" in later decisions, increasing the efficiency of the decisions. Then, more realistic utility structures can be defined using final success criteria in utility functions for earlier decisions. An important distinction is made between weak and strong decision rules. Weak rules are allowed to be a function of prior test scores. Conditions for optimal rules to be monotone are presented, and it is shown that optimal weak monotone rules are compensatory by nature. Results from an empirical example of instructional decision making illustrate the differences between simultaneous and separate approaches. Subjects were 71 medical students receiving interactive video or computer-aided instruction

Balanced difficulty task finder: an adaptive recommendation method for learning tasks based on the concept of state of flow

publishedVersio

Identifiable Cognitive Diagnosis with Encoder-decoder for Modelling Students' Performance

Cognitive diagnosis aims to diagnose students' knowledge proficiencies based on their response scores on exam questions, which is the basis of many domains such as computerized adaptive testing. Existing cognitive diagnosis models (CDMs) follow a proficiency-response paradigm, which views diagnostic results as learnable embeddings that are the cause of students' responses and learns the diagnostic results through optimization. However, such a paradigm can easily lead to unidentifiable diagnostic results and the explainability overfitting problem, which is harmful to the quantification of students' learning performance. To address these problems, we propose a novel identifiable cognitive diagnosis framework. Specifically, we first propose a flexible diagnostic module which directly diagnose identifiable and explainable examinee traits and question features from response logs. Next, we leverage a general predictive module to reconstruct response logs from the diagnostic results to ensure the preciseness of the latter. We furthermore propose an implementation of the framework, i.e., ID-CDM, to demonstrate the availability of the former. Finally, we demonstrate the identifiability, explainability and preciseness of diagnostic results of ID-CDM through experiments on four public real-world datasets

Development of a Computerized Adaptive Testing for Internet Addiction

Internet addiction disorder has become one of the most popular forms of addiction in psychological and behavioral areas, and measuring it is growing increasingly important in practice. This study aimed to develop a computerized adaptive testing to measure and assess internet addiction (CAT-IA) efficiently. Four standardized scales were used to build the original item bank. A total of 59 polytomously scored items were finally chosen after excluding 42 items for failing the psychometric evaluation. For the final 59-item bank of CAT-IA, two simulation studies were conducted to investigate the psychometric properties, efficiency, reliability, concurrent validity, and predictive validity of CAT-IA under different stopping rules. The results showed that (1) the final 59 items met IRT assumptions, had high discrimination, showed good item-model fit, and were without DIF; and (2) the CAT-IA not only had high measurement accuracy in psychometric properties but also sufficient efficiency, reliability, concurrent validity, and predictive validity. The impact and limitations of CAT-IA were discussed, and several suggestions for future research were provided

Learning from Sequential User Data: Models and Sample-efficient Algorithms

Recent advances in deep learning have made learning representation from ever-growing datasets possible in the domain of vision, natural language processing (NLP), and robotics, among others. However, deep networks are notoriously data-hungry; for example, training language models with attention mechanisms sometimes requires trillions of parameters and tokens. In contrast, we can often access a limited number of samples in many tasks. It is crucial to learn models from these `limited\u27 datasets. Learning with limited datasets can take several forms. In this thesis, we study how to select data samples sequentially such that downstream task performance is maximized. Moreover, we study how to introduce prior knowledge in the deep networks to maximize prediction performance. We focus on four sequential tasks: computerized adaptive testing in psychometrics, sketching in recommender systems, knowledge tracing in computer-assisted education, and career path modeling in the labor market. In the first two tasks, we devise novel sample-efficient algorithms to query a minimal number of sequential samples to improve future predictions. We propose a Bilevel Optimization-Based framework for computerized adaptive testing to learn a data-driven question selection algorithm that improves existing data selection policies. We also tackle the sketching problem in the recommender system, with the task of recommending the next item using a stored subset of prior data samples. In this setting, we develop a data-driven sequential selection algorithm that tackles evolving downstream task distribution. In the last two tasks, we devise novel neural models to introduce prior knowledge exploiting limited data samples. For knowledge tracing, we propose a novel neural architecture, inspired by cognitive and psychometric models, to improve the prediction of students\u27 future performance and utilize the labeled data samples efficiently. For career path modeling, we propose a novel and interpretable monotonic nonlinear state-space model to analyze online user professional profiles and provide actionable feedback and recommendations to users on how they can reach their career goals. The data-driven differentiable data selection algorithms for the first two tasks open up future directions to query (a non-differentiable operation) a minimal number of samples optimally to maximize prediction performance. The structures, introduced in the neural architecture for the models in the last two tasks using prior knowledge, open up future directions to learn deep models augmented with prior knowledge using limited data samples

Prognostic-based Life Extension Methodology with Application to Power Generation Systems

Practicable life extension of engineering systems would be a remarkable application of prognostics. This research proposes a framework for prognostic-base life extension. This research investigates the use of prognostic data to mobilize the potential residual life. The obstacles in performing life extension include: lack of knowledge, lack of tools, lack of data, and lack of time. This research primarily considers using the acoustic emission (AE) technology for quick-response diagnostic. To be specific, an important feature of AE data was statistically modeled to provide quick, robust and intuitive diagnostic capability. The proposed model was successful to detect the out of control situation when the data of faulty bearing was applied. This research also highlights the importance of self-healing materials. One main component of the proposed life extension framework is the trend analysis module. This module analyzes the pattern of the time-ordered degradation measures. The trend analysis is helpful not only for early fault detection but also to track the improvement in the degradation rate. This research considered trend analysis methods for the prognostic parameters, degradation waveform and multivariate data. In this respect, graphical methods was found appropriate for trend detection of signal features. Hilbert Huang Transform was applied to analyze the trends in waveforms. For multivariate data, it was realized that PCA is able to indicate the trends in the data if accompanied by proper data processing. In addition, two algorithms are introduced to address non-monotonic trends. It seems, both algorithms have the potential to treat the non-monotonicity in degradation data. Although considerable research has been devoted to developing prognostics algorithms, rather less attention has been paid to post-prognostic issues such as maintenance decision making. A multi-objective optimization model is presented for a power generation unit. This model proves the ability of prognostic models to balance between power generation and life extension. In this research, the confronting objective functions were defined as maximizing profit and maximizing service life. The decision variables include the shaft speed and duration of maintenance actions. The results of the optimization models showed clearly that maximizing the service life requires lower shaft speed and longer maintenance time

Process Data Applications in Educational Assessment

The widespread adoption of computer-based testing has opened up new possibilities for collecting process data, providing valuable insights into the problem-solving processes that examinees engage in when answering test items. In contrast to final response data, process data offers a more diverse and comprehensive view of test takers, including construct-irrelevant characteristics. However, leveraging the potential of process data poses several challenges, including dealing with serial categorical responses, navigating nonstandard formats, and handling the inherent variability. Despite these challenges, the incorporation of process data in educational assessments holds immense promise as it enriches our understanding of students' cognitive processes and provides additional insights into their interactive behaviors. This thesis focuses on the application of process data in educational assessments across three key aspects. Chapter 2 explores the accurate assessment of a student's ability by incorporating process data into the assessment. Through a combination of theoretical analysis, simulations, and empirical study, we demonstrate that appropriately integrating process data significantly enhances assessment precision. Building upon this foundation, Chapter 3 takes a step further by addressing not only the target attribute of interest but also the nuisance attributes present in the process data to mitigate the issue of differential item functioning. We present a novel framework that leverages process data as proxies for nuisance attributes in item response functions, effectively reducing or potentially eliminating differential item functioning. We validate the proposed framework using both simulated data and real data from the PIAAC PSTRE items. Furthermore, this thesis extends beyond the analysis of existing tests and explores enhanced strategies for item administration. Specifically, in Chapter 4, we investigate the potential of incorporating process data in computerized adaptive testing. Our adaptive item selection algorithm leverages information about individual differences in both measured proficiency and other meaningful traits that can influence item informativeness. A new framework for process-based adaptive testing, encompassing real-time proficiency scoring and item selection is presented and evaluated through a comprehensive simulation study to demonstrate the efficacy