Using Connectionist Models to Evaluate Examinees’ Response Patterns to Achievement Tests

The attribute hierarchy method (AHM) applied to assessment engineering is described. It is a psychometric method for classifying examinees’ test item responses into a set of attribute mastery patterns associated with different components in a cognitive model of task performance. Attribute probabilities, computed using a neural network, can be estimated for each examinee thereby providing specific information about the examinee’s attribute-mastery level. The pattern recognition approach described in this study relies on an explicit cognitive model to produce the expected response patterns. The expected response patterns serve as the input to the neural network. The model also yields the cognitive test specifications. These specifications identify the examinees’ attribute patterns which are used as output for the neural network. The purpose of the statistical pattern recognition analysis is to estimate the probability that an examinee possess specific attribute combinations based on their observed item response patterns. Two examples using student response data from a sample of algebra items on the SAT illustrate our pattern recognition approach

The effects of mixture-induced local dependence on diagnostic classification

Diagnostic Classification Models (DCMs) have been extensively researched in recent psychometric literature for providing mastery skill profiles for diagnostic feedback (Henson, Templin, & Willse, 2009). DCMs are multidimensional confirmatory latent class models (LCMs) where latent classes represent skill mastery profiles and latent attributes are categorical (mastery or non-mastery). DCMs make a central assumption that once mastery profiles are accounted for that items are independent, referred to as local independence (LI). Construct irrelevant variance (e.g., differential item functioning (DIF), speededness, test wiseness, item-to-skill misspecification) or underrepresentation (extra dimensionality, inappropriate definitional grain-size of defined skills) could introduce systematic within-class variation which would violate LI. Using connections of LCMs with mixture IRT models, this study explores the effects of introducing systematic within-class variation on diagnostic classification. The log-linear cognitive diagnosis model (LCDM) is extended to include continuous abilities, akin to a multidimensional item response theory (MIRT) model with underling mixtures due to skill mastery/nonmastery. Data were then simulated for different ability variances related to distribution overlap conditions. Multiple DCMs are then fit using the LCDM framework in a simulation study. Impact on classification and local dependence detection are summarized. It was found that as mixture overlap increased due to companion ability variance that diagnostic classification in DCMs greatly suffered, but can be detected by Yen’s Q3. The relationship of the degree of inaccuracy and effect sizes based on ability variance and group separation is delineated. Recommendations for practitioners are given along with areas for future study

I’ve (Urn)ed This: An Application and Criterion-based Evaluation of the Urnings Algorithm

There is increased interest in personalized learning and making e-learning environments more adaptable. Some e-learning systems may use an Item Response Theory (IRT)-based assessment system. An important distinction between assessment and learning contexts is that learner proficiency is expected to remain constant across an assessment, while it is expected to change over time in a learning context. Constant learner proficiency during an assessment enables conventional approaches to estimating person and item parameters using IRT. These IRT-based systems could be abandoned for alternative approaches to modeling learners and system learning content, but assessments may provide more functions than adapting learning material to students. Thus, there is the question, how can e-learning systems with IRT-based assessment components more dynamically adapt their learning content? Is there a solution that leverages IRT for adapting the learning content of the system? A promising solution is the Urnings algorithm. Like other candidate algorithms, it is computationally light, but this algorithm has mechanisms for preventing variance inflation and is suitable for e-learning contexts. It also provides a measure of uncertainty around estimates. It has been studied both through simulations and applications to e-learning systems. Results are promising; however, there has not been an application of the Urnings algorithm to an e-learning context where there are conventionally estimated person parameters to compare the algorithm estimates to. This study addresses this gap by applying the Urnings algorithm to a K–8 reading and mathematics learning platform. In data from this platform, we have person parameter estimates across academic years from an in-system diagnostic assessment. Results from this study will help industry researchers understand the feasibility of the Urnings algorithm for large e-learning systems with IRT-based assessment components

The Impact of Information Quantity and Quality on Parameter Estimation for a Selection of Dynamic Bayesian Network Models with Latent Variables

abstract: Dynamic Bayesian networks (DBNs; Reye, 2004) are a promising tool for modeling student proficiency under rich measurement scenarios (Reichenberg, in press). These scenarios often present assessment conditions far more complex than what is seen with more traditional assessments and require assessment arguments and psychometric models capable of integrating those complexities. Unfortunately, DBNs remain understudied and their psychometric properties relatively unknown. If the apparent strengths of DBNs are to be leveraged, then the body of literature surrounding their properties and use needs to be expanded upon. To this end, the current work aimed at exploring the properties of DBNs under a variety of realistic psychometric conditions. A two-phase Monte Carlo simulation study was conducted in order to evaluate parameter recovery for DBNs using maximum likelihood estimation with the Netica software package. Phase 1 included a limited number of conditions and was exploratory in nature while Phase 2 included a larger and more targeted complement of conditions. Manipulated factors included sample size, measurement quality, test length, the number of measurement occasions. Results suggested that measurement quality has the most prominent impact on estimation quality with more distinct performance categories yielding better estimation. While increasing sample size tended to improve estimation, there were a limited number of conditions under which greater samples size led to more estimation bias. An exploration of this phenomenon is included. From a practical perspective, parameter recovery appeared to be sufficient with samples as low as N = 400 as long as measurement quality was not poor and at least three items were present at each measurement occasion. Tests consisting of only a single item required exceptional measurement quality in order to adequately recover model parameters. The study was somewhat limited due to potentially software-specific issues as well as a non-comprehensive collection of experimental conditions. Further research should replicate and, potentially expand the current work using other software packages including exploring alternate estimation methods (e.g., Markov chain Monte Carlo).Dissertation/ThesisDoctoral Dissertation Family and Human Development 201

Nonparametric diagnostic classification analysis for testlet based tests

Diagnostic classification Diagnostic Classification Models (DCMs) are multidimensional confirmatory latent class models that can classify individuals into different classes based on their attribute mastery profiles. While DCMs represent the more prevalent parametric approach to diagnostic classification analysis, the Hamming distance method, a newly developed nonparametric diagnostic classification method, is quite promising in that it does not require fitting a statistical model and is less demanding on sample size. However, both parametric and nonparametric approach have assumptions of local item independency, which is often violated by testlet based tests. This study proposed a conditional-correlation based nonparametric approach to assess testlet effect and a set of testlet Hamming distance methods to account for the testlet effects in classification analyses. Simulation studies were conducted to evaluate the proposed methods. In the conditional-correlation approach, the testlet effects were computed as the average item-pair correlations within the same testlet by conditioning on attribute profiles. The inverse of the testlet effect was then used in testlet Hamming distance method to weight the Hamming distances for that particular testlet. Simulation studies were conducted to evaluate the proposed methods in conditions with varying sample size, testlet effect size, testlet size, balance of testlet size, and balance of testlet effect size. Although the conditional-correlation based approach often underestimated true testlet effect sizes, it was still able to detect the relative size of different testlet effects. The developed testlet Hamming distance methods seem to be an improvement over the estimation methods that ignore testlet effects because they provided slightly higher classification accuracy where large testlet effects were present. In addition, Hamming distance method and maximum likelihood estimation are robust to local item dependency caused by low to moderate testlet effects. Recommendations for practitioners and study limitations were provided

Diagnosing Primary Pupils Learning of the Concept of After in The Topic Time Through Knowledge States by Using Cognitive Diagnostic Assessment

Purpose - Knowledge state specifies pupils mastery level and informs about their strength and weaknesses in the tested domain. This study attempted to diagnose primary pupils learning the concept of after through their knowledge states by using Cognitive Diagnostic Assessment (CDA). Methodology - This study employed a survey research design to gauge pupils' knowledge states for the concept of finding the date after a specific number of days from a given date [abbreviated as the concept of after]. Quantitative data from the pupils pattern of response to the items in Cognitive Diagnostic Assessment (CDA) were collected and analyzed. Items in the CDA were designed by three experienced Mathematics Education researchers and content validated by a panel of seven expert primary mathematics teachers. It was then administered to 238 Grade Six pupils from 11 primary schools in Penang, Malaysia. The pupils item responses were interpreted into knowledge states and mastery levels. Findings - The overall analysis showed that there were 18 knowledge states diagnosed in the concept of after. This large number of knowledge states indicated the specificity of pupils mastery level and thus provided detailed information about their strengths and weaknesses in the concept of after. The findings of this study imply that primary pupils face different levels of difficulty when they are learning the topic of Time. Significance - This method of diagnosing pupils knowledge in terms of mastery level of each attribute tested is different from a conventional diagnostic test which provides only the final score for each pupil. By knowing the pupils knowledge states, teachers can make use of this fine-grained information to enable them to carry out differentiated instructional planning and other remedial work more effectively. Pupils can also use this information to monitor their own learning by maintaining their strengths and overcoming their weaknesses to cope with their own studies

A comparison of parameter estimation algorithms for estimating a polytomous log-linear cognitive diagnosis model for polytomous attributes

Parameter estimation techniques such as an expectation-maximization (EM) algorithm have been used ubiquitously to estimate cognitive diagnosis models (CDM). The primary goal of this study was to utilize polytomous attributes in the polytomous log-linear cognitive diagnosis model (P-LCDM-PA), which is a special case of the general polytomous diagnostic model (GPDM) for polytomous attributes. Then, due to exponentially increasing the number of latent classes, explore the feasibility and efficiency in addition to the quality of parameter estimation of the stochastic expectation-maximization (SEM) and Metropolis-Hastings Robbins-Monro (MH-RM) algorithms relative to the EM algorithm. The SEM and MH-RM algorithms may be more computationally advantageous over an EM algorithm when there exist many latent classes. As the number of measured attributes increases in a diagnostic assessment, the number of latent classes increases exponentially. The large number of classes is even more problematic when polytomous attribute levels are introduced in the diagnostic assessment. The large number of classes becomes computationally challenging when estimating a model using an EM algorithm because for each respondent, the probability of class membership is computed for every latent class. Simulation experiments were conducted examining item parameter recovery in the P-LCDM-PA, correct classification rates, and computational time between the three algorithms

Refining Prerequisite Skill Structure Graphs Using Randomized Controlled Trials

Prerequisite skill structure graphs represent the relationships between knowledge components. Prerequisite structure graphs also propose the order in which students in a given curriculum need to be taught specific knowledge components in order to assist them build on previous knowledge and improve achievement in those subject domains. The importance of accurate prerequisite skill structure graphs can therefore not be overemphasized. In view of this, many approaches have been employed by domain experts to design and implement these prerequisite structures. A number of data mining techniques have also been proposed to infer these knowledge structures from learner performance data. These methods have achieved varied degrees of success. Moreover, to the best of our knowledge, none of the methods have employed extensive randomized controlled trials to learn about prerequisite skill relationships among skills. In this dissertation, we motivate the need for using randomized controlled trials to refine prerequisite skill structure graphs. Additionally, we present PLACEments, an adaptive testing system that uses a prerequisite skill structure graph to identify gaps in students’ knowledge. Students with identified gaps are assisted with more practice assignments to ensure that the gaps are closed. PLACEments additionally allows for randomized controlled experiments to be performed on the underlying prerequisite skill structure graph for the purpose of refining the structure. We present some of the different experiment categories which are possible in PLACEments and report the results of one of these experiment categories. The ultimate goal is to inform domain experts and curriculum designers as they create policies that govern the sequencing and pacing of contents in learning domains whose content lend themselves to sequencing. By extension students and teachers who apply these policies benefit from the findings of these experiments

The Analysis of Student Traces for Q-Matrix Refinement and Knowledge Tracing

Nous assistons à une effervescense de l’auto-apprentissage rendue possible par l’Internet et les environnements d’apprentissage. L’accessibilité des MOOCS et des environnements d’apprentissage informatisés en est une manifestation. En contrepartie, l’apprenant perd le guidage personnalisé d’un tuteur humain et le développement d’environnements d’apprentissage adaptatifs vise à combler cette lacune. Afin d’offrir guidage et personnalisation au long du processus d’apprentissage, il est essentiel de bien évaluer les connaissances acquises de l’apprenant et d’adapter le matériel didactique en conséquence. Les recherches dans les domaines des tutoriels intelligents et de l’analytique des données éducationnelles visent essentiellement à développer des modèles de connaissances pouvant offrir le support à la personnalisation de l’apprentissage. Je propose dans cette thèse de nouvelles approches à la modélisation des connaissances apprenants autour de deux axes. Le premier porte sur l’objectif de valider les connaissances et compétences sous-jacentes à des tâches à partir de données. La classification de questions et exercices en une taxonomie d’objectifs d’apprentissage est un exemple pratique d’identification de compétences sousjacentes que les enseignants et pédagogues font couramment. Les chercheurs du domaine de la modélisation cognitive (Cognitive Diagnostic Modeling) vont plus loin en identifiant plusieurs connaissances et compétences derrière un seul problème à résoudre par exemple. Cet exercice est intrinsèquemet difficile et sujet aux erreurs. Les recherches pour faciliter la validation des connaissances sous-jacentes sont connues sous le nom du raffinement d’une Q-matrice qui représente l’alignement des tâches aux connaissances requises. La dernière décennie a été témoin de développements importants autour des approches basées sur les données pour effectuer le raffinement de Q-matrices. Ce processus de raffinement peut être considéré comme un problème de classification : pour chaque alignement tâche-connaissance défini par l’expert, l’algorithme de classification doit décider s’il est correct ou incorrect. Alors que la majorité des algorithmes portent sur une décision par alignement individuel, nous proposons une approche de classification basée sur des algorithme multi-classe où l’ensemble des connaissances requises par une tâche est soumises, plutôt que chaque connaissance individuellement. Les résultats de l’approche démontrent que le raffinement est généralement de meilleur qualité que les techniques de l’état de l’art. Le second axe vise à améliorer les modèles d’apprentissage profond pour l’évaluation des connaissances de l’apprenant à partir de traces séquentielles du succès ou échecs aux tâches. Nous tablons sur un modèle d’évaluation de connaissances capable de capturer l’évolution temporelle du profil de connaissance qui évolue au long du processus d’apprentissage de l’apprenant. Les algorithmes d’apprentissage profond utilisant une architecture LSTM (Long Short-Term Memory) aspirent à cet objectif de mémoriser les informations temporelles et réussissent effectivement à mieux prédire les performances des apprenants. Mais le profil de connaissance constitue un mécanisme plus explicite de l’état de connaissance atteint et plus efficace pour synthétiser cet état. Nous intégrons donc ce mécanisme à une architecture LSTM et à une architecture de réseau de mémoires (memory networks) afin de valider cette hypothèse. Le profil de connaissance est modélisé sous forme de classes et cette information est encodée par un vecteur binaire de longueur unitaire (one-hot) qui est fourni en entrée aux modèles d’apprentissage profond.----------ABSTRACT: The growth of self-learning, enabled by the availability on the Internet of different forms of didactic material such as MOOCs and tutoring systems, increases in turn the relevance of personalized instructions for students in adaptive learning environment. For providing adaptive and personalized learning instructions, the assessment of student’s mastery of a topic and the estimation of when she actually knows how to answer problems correctly is recognized as paramount in the fields of learning analytics and educational data mining community. In this dissertation, I propose novel approaches for building skills and student learning models along two axes. The first axis is to recover and ensure the quality of skills sets behind problems in learning system. The second axis is on improving the predictive accuracy of students’ performance based on student ability profile on skills and considering of difficulty of the problem dynamically. Both of these axes are complementary and essential in knowledge assessment of future educational learning systems for equipping intelligent agents to provide adaptive instructions and independent learning environment for students. The first axis is referred to as the Q-matrix refinement problem and consists in validating an expert-defined mapping of exercises and tasks to underlying skills. The last decade has witnessed a wealth of data driven approaches aiming to refine expert-defined mappings. This refinement can be seen as a classification problem: for each possible mapping of task to skill, the classifier has to decide whether the expert’s advice is correct, or incorrect. Whereas most algorithms are working at the level of individual mappings, we introduce an approach based on a multi-label classification algorithm that is trained on the mapping of a task to all skills simultaneously. This approach improves Q-matrix validation methods by using supervised multi-label classifier. Results show it outperforms the existing Q-matrix refinement techniques. The second axis aims to improve deep learning models of skills assessment based on sequential data. The student skills model needs to capture the temporal nature of student knowledge, changing over time, based on the learning transferred from previous practice. Deep learning has achieved a large amount of success in student performance prediction with models relying on Long short-term memory (LSTM). We proposed two approaches called Deep Knowledge Tracing and Dynamic Student Classification (DKT-DSC) and Dynamic Student Classification on Memory Networks (DSCMN) based on LSTM and key-value memory networks. We apply k-means clustering to capture students’ temporal ability profile at each time interval, which serves as a transfer learning mechanism across student’s long-term learning process. DKT DSC can capture temporal ability profile, utilize ability profile in assessment of knowledge mastery state simultaneously. The second approach, DSCMN, utilizes problem difficulty in prediction of student performance. According to experimental results, these approaches show improvements in student performance prediction over other state-of-the-art methods (such as BKT, PFA, etc.)