Bringing Learning Back In: Examining Three Psychometric Models for Evaluating Learning Progression Theories

Learning progressions provide potentially valuable information to teachers about how to develop a scope and sequence for a group of learning objectives. However, for the learning progressions to be valuable, the progressions must be supported. Although there are several approaches and models that can be used to evaluate the validity of a learning progression, there is a dearth of research examining the advantages and limitations of each approach. The purpose of this study was to examine a multi-dimensional item response theory model and two cognitive diagnostic models (DINA and HO-DINA) for evaluating two learning progressions via a simulation study. In addition, the models were applied to empirical data to determine if the models provided consistent results. The results from the investigation indicated that five methods of using the model and statistical methods derived from them to testify learning level order could complement each other. None of the methods worked dominantly better than the others but they all deemed useful in certain contexts. With respect to assessing the possible links among levels across progressions, the degree to which the model recovered the true information in the simulation studies varied depending on the model and the magnitude of the difference between the learning levels. The more distant the levels were, the more accurate the model became at recovering the true classification. For the empirical analysis, three models provided convergent evidence to support almost all the aspects of the theory underlying two progressions considered in this study. Statistical results also suggested a few revisions to make the theory more in line with the empirical evidence. Four limitations were discussed, and six future directions were elaborated to address the drawbacks of this study. Finally, three practical implications were presented as take-away messages from this dissertation

Nonparametric diagnostic classification analysis for testlet based tests

Diagnostic classification Diagnostic Classification Models (DCMs) are multidimensional confirmatory latent class models that can classify individuals into different classes based on their attribute mastery profiles. While DCMs represent the more prevalent parametric approach to diagnostic classification analysis, the Hamming distance method, a newly developed nonparametric diagnostic classification method, is quite promising in that it does not require fitting a statistical model and is less demanding on sample size. However, both parametric and nonparametric approach have assumptions of local item independency, which is often violated by testlet based tests. This study proposed a conditional-correlation based nonparametric approach to assess testlet effect and a set of testlet Hamming distance methods to account for the testlet effects in classification analyses. Simulation studies were conducted to evaluate the proposed methods. In the conditional-correlation approach, the testlet effects were computed as the average item-pair correlations within the same testlet by conditioning on attribute profiles. The inverse of the testlet effect was then used in testlet Hamming distance method to weight the Hamming distances for that particular testlet. Simulation studies were conducted to evaluate the proposed methods in conditions with varying sample size, testlet effect size, testlet size, balance of testlet size, and balance of testlet effect size. Although the conditional-correlation based approach often underestimated true testlet effect sizes, it was still able to detect the relative size of different testlet effects. The developed testlet Hamming distance methods seem to be an improvement over the estimation methods that ignore testlet effects because they provided slightly higher classification accuracy where large testlet effects were present. In addition, Hamming distance method and maximum likelihood estimation are robust to local item dependency caused by low to moderate testlet effects. Recommendations for practitioners and study limitations were provided

Factor analytic models and cognitive diagnostic models: how comparable are they?--a comparison of R-RUM and compensatory MIRT model with respect to cognitive feedback

The necessity and importance of cognitive diagnosis is being realized by more and more researchers. As a result, a number of models have been defined for cognitive diagnosis--the IRT-based discrete cognitive diagnosis models (ICDMs) and the traditional continuous latent trait models. However, there is a lack of literature that compares the newly defined ICDMs based on constrained latent class models to more traditional approaches such as a multidimensional factor analytic model. The purpose of this study is to compare the feedback provided to examinees using a multidimensional item response model (MIRT) versus feedback provided using an ICDM. Specifically, a Monte Carlo study was used to compare the diagnostic results from the R-RUM, a noncompensatory model with dichotomous abilities, to diagnoses made based on the 2PL CMIRT model, a compensatory model with continuous abilities. A fully crossed design was used to consider the effects of test quality, Q-matrix structure and inter-attribute correlation on the agreement rates of the diagnostic feedback for examinees between these two models. Given that one of the factors of this study is "test quality", an initial study was performed to explore the possible relationship between test quality (including estimated model parameters) based on the models used to characterize examinee responses. In addition, because these models provide examinee information in different ways (one discrete and one continuous), a method using logistic regression, which is used to discretize the continuous estimates provided by the 2PL CMIRT, is discussed as a way to maximize diagnostic agreement between these two models. The significance of this study is that, if the two models agree consistently across the experimental conditions, model selection for cognitive purposes can be based largely on the preference of the researcher, which is informed by an underlying theory and assessment purposes. However, if the two models do not agree consistently, this study will help (1) to identify situations where the two models agree or disagree consistently and (2) to explore the feasibility of using the MIRT model for classifying examinees cognitively

The effects of mixture-induced local dependence on diagnostic classification

Diagnostic Classification Models (DCMs) have been extensively researched in recent psychometric literature for providing mastery skill profiles for diagnostic feedback (Henson, Templin, & Willse, 2009). DCMs are multidimensional confirmatory latent class models (LCMs) where latent classes represent skill mastery profiles and latent attributes are categorical (mastery or non-mastery). DCMs make a central assumption that once mastery profiles are accounted for that items are independent, referred to as local independence (LI). Construct irrelevant variance (e.g., differential item functioning (DIF), speededness, test wiseness, item-to-skill misspecification) or underrepresentation (extra dimensionality, inappropriate definitional grain-size of defined skills) could introduce systematic within-class variation which would violate LI. Using connections of LCMs with mixture IRT models, this study explores the effects of introducing systematic within-class variation on diagnostic classification. The log-linear cognitive diagnosis model (LCDM) is extended to include continuous abilities, akin to a multidimensional item response theory (MIRT) model with underling mixtures due to skill mastery/nonmastery. Data were then simulated for different ability variances related to distribution overlap conditions. Multiple DCMs are then fit using the LCDM framework in a simulation study. Impact on classification and local dependence detection are summarized. It was found that as mixture overlap increased due to companion ability variance that diagnostic classification in DCMs greatly suffered, but can be detected by Yen’s Q3. The relationship of the degree of inaccuracy and effect sizes based on ability variance and group separation is delineated. Recommendations for practitioners are given along with areas for future study

Effects on individual level behaviour in mackerel (Scomber scombrus) of sub-lethal capture related stressors: Crowding and hypoxia

Stress to fish during harvest in wild capture fisheries is known to negatively influence subsequent survival in catches that are released. Therefore, if fisheries are to be conducted sustainably, there is a need to promote good fish welfare during the capture process. Purse seine fishing is a widespread and efficient fishing method. However, capture and release of fish from purse seines (a process called “slipping”) can result in extremely high mortality in small pelagic schooling species. The objective of this study was to establish behavioural indicators of sub-lethal stress in Atlantic mackerel (Scomber scombrus) that may be used to set safe threshold limits for use in commercial purse seine fishing, in order to ensure good fish welfare and thereby minimise slipping mortality. Controlled mesocosm scale experiments with schools of mackerel in net pens were undertaken to determine behavioural responses to simulated purse seine capture stressors of “crowding”, “hypoxia” and “crowding & hypoxia”. Crowding (at 30 kg.m-3) was achieved by reducing the volume of the net pen, while hypoxia (to 40% oxygen saturation) was achieved by surrounding the net pen with a tarpaulin bag to prevent water exchange. Using video analysis, we investigated behavioural responses in nearest neighbour distances, nearest neighbour angular deviations, tail beat amplitude and tail beat frequency (TBF). Of the metrics considered, only TBF showed a response; a significant increase to “crowding” (42% increase) and “crowding & hypoxia” (38% increase) was found. The increase in TBF in response to “hypoxia” alone (29% increase) was not significant. We therefore conclude that increases in tail beat frequency may be used as an indicator of sub-lethal purse seine capture stress in mackerel that may have utility in minimising post slipping mortality.publishedVersio

Rule-based item construction.:Analysis with and comparison of linear logistic test models and cognitive diagnostic models with two item types

Die Dissertation behandelt die Darstellung und Evaluierung regelgeleiteter Aufgabenkonstruktion am Beispiel von figuralen Reasoning- und mathematischen Textaufgaben sowie die Anwendung und den Vergleich von linear logistischen Testmodellen (LLTMs) und Kognitiven Diagnosemodellen (CDMs) als statistische Analysemethoden. Die Ergebnisse zeigen Rasch-Skalierbarkeit der Aufgaben und demonstrieren einen präzisen Aufgabenkonstruktions- und Analyseprozess. Die LLTM-Varianten liefern wichtige Einblicke in kognitive Lösungsprozesse und in die Zusammensetzung der Aufgabenschwierigkeit für beide Aufgabentypen genauso wie für einen implementierten Aufgabencloning-Ansatz und longitudinale Datenstrukturen. In der CDM-Anwendung zeigen sich erhebliche Modellierungsprobleme und Unangemessenheit des Ansatzes für die vorliegenden Aufgabenbeispiele. Hinweise bezüglich der Aufgabenkonstruktion, der statistischen Modelle und der Interpretation der Ergebnisse für Anwendung und Forschung werden herausgestellt. The dissertation focuses on demonstration and evaluation of rule-based item construction of figural reasoning items and mathematical word problems and application as well as comparison of LLTMs and CDMs as statistical analysis methods. Results show Rasch scalability of items, confirm the importance of the chosen basic parameter sets and demonstrate precise item construction and analysis processes. It is shown how LLTM and its variants can contribute substantial insights into cognitive solution processes and composition of item difficulty in relational reasoning and mathematical word problems and also for item cloning and longitudinal data. However, CDM application detects severe modeling problems and misfit. Application hints regarding test item construction as well as statistical model application and interpretation of results for practitioners and researchers are pointed out

Validation of the Item-Attribute Matrix in TIMSS-Mathematics Using Multiple Regression and the LSDM

For many cognitive diagnostic models, the item-attribute matrix (or Q-matrix) is an essential component which displays the relationship between items and their latent attributes or skills in knowledge and cognitive processes. However, it is a challenge to develop an effective Q-matrix.The purposes of this study were (1) to validate of the item-attribute matrix using two levels of attributes (Level 1 attributes and Level 2 sub-attributes), and (2) through retrofitting the diagnostic models to the mathematics test of the Trends in International Mathematics and Science Study (TIMSS), to evaluate the construct validity of TIMSS mathematics assessment by comparing the results of two assessment booklets. Item data were extracted from Booklets 2 and 3 for the 8th grade in TIMSS 2007, which included a total of 49 mathematics items and every student\u27s response to every item. The study developed three categories of attributes at two levels: content, cognitive process (TIMSS or new), and comprehensive cognitive process (or IT) based on the TIMSS assessment framework, cognitive procedures, and item type. At level one, there were 4 content attributes (number, algebra, geometry, and data and chance), 3 TIMSS process attributes (knowing, applying, and reasoning), and 4 new process attributes (identifying, computing, judging, and reasoning). At level two, the level 1 attributes were further divided into 32 sub-attributes. There was only one level of IT attributes (multiple steps/responses, complexity, and constructed-response). Twelve Q-matrices (4 originally specified, 4 random, and 4 revised) were investigated with eleven Q-matrix models (QM1 ~ QM11) using multiple regression and the least squares distance method (LSDM). Comprehensive analyses indicated that the proposed Q-matrices explained most of the variance in item difficulty (i.e., 64% to 81%). The cognitive process attributes contributed to the item difficulties more than the content attributes, and the IT attributes contributed much more than both the content and process attributes. The new retrofitted process attributes explained the items better than the TIMSS process attributes. Results generated from the level 1 attributes and the level 2 attributes were consistent. Most attributes could be used to recover students\u27 performance, but some attributes\u27 probabilities showed unreasonable patterns. The analysis approaches could not demonstrate if the same construct validity was supported across booklets. The proposed attributes and Q-matrices explained the items of Booklet 2 better than the items of Booklet 3. The specified Q-matrices explained the items better than the random Q-matrices

Design, Development and Biomechanical Analysis of Scaffolds for Augmentation of Rotator Cuff Repairs

Rotator cuff tears are a source of debilitating pain that commonly affects more than 40 of our aging population. Despite advances in surgical treatment, the failure rate of rotator cuff repairs is as high as 20-90 . Extracellular matrix (ECM) derived scaffolds have recently been investigated as augmentation devices for rotator cuff repairs, but none has yet demonstrated both the appropriate biological and mechanical properties for mitigating re-tears and enhancing healing. This dissertation proposes to engineer the mechanical properties of allograft fascia lata in a manner that will allow its use as an augmentation device for rotator cuff repairs. This dissertation also aims to develop a simple quasi-linear spring-network model for rotator cuff repairs to elucidate the basic biomechanics of these repairs. The central hypothesis is that engineered fascia lata will have suture retention strength similar to that of human rotator cuff tendon (̃250N), even after in vivo implantation. The specific aims are to engineer the mechanical properties of allograft fascia lata ECM and to subsequently evaluate the host response and concomitant mechanical properties of the engineered (reinforced) fascia in a rat model. Further, this dissertation will also develop and validate a spring-network model for simplified rotator cuff repairs. Studies presented in this dissertation demonstrate stitching as a technology to engineer the suture retention and stiffness of allograft (human derived) fascia lata ECM. Stitching fascia ECM with braided, resorbable, polymer fibers in a unique, controlled manner increased the suture retention load of reinforced fascia scaffolds by six fold over non-reinforced fascia. Additionally, the suture retention properties of reinforced fascia scaffolds were comparable to that of human rotator cuff tendon (̃250N) at time zero and even after in vivo implantation for twelve weeks. Except for the increased presence of foreign body giant cells in areas concentrated around the polymer fibers, the host re