2010 Mathematics Subject Classification: 62P15.As the cognitive operations are hypothesized according to a cognitive theory in the context of a study, they are latent (hidden) in nature and cannot be measured and scored directly from the test. The least squares distance model (LSDM) of cognitive diagnosis uses estimates of the item parameters under a specific item-response theory (IRT) model to provide estimates of the probability of a person to process correctly any cognitive attribute given the person’s location on the IRT logit scale. In this paper a methodology for comparing two (or more) groups of individuals, according to their performance on a given set of cognitive attributes is presented

Atanasov, Dimitar

Dimitrov, Dimiter M.

Bulgarian Digital Mathematics Library at IMI-BAS

  Provided for non-commercial research and educational use.   Not for reproduction, distribution or commercial use. The attached copy is furnished for non-commercial research and education use only.Authors are permitted to post this version of the article to their personal websites or institutional repositories and to share with other researchers in the form of electronic reprints.Other uses, including reproduction and distribution, or selling orlicensing copies, or posting to third party websites are prohibited.For further information on Pliska Studia Mathematica Bulgarica visit the website of the journal http://www.math.bas.bg/~pliska/or contact: Editorial OfficePliska Studia Mathematica BulgaricaInstitute of Mathematics and Informatics Bulgarian Academy of SciencesTelephone: (+359-2)9792818, FAX:(+359-2)971-36-49e-mail: pliska@math.bas.bgPliska Stud. Math. Bulgar. 22 (2013), 33–40STUDIA MATHEMATICABULGARICAGROUP COMPARISONS ON COGNITIVE ATTRIBUTESUSING THE LEAST SQUARES DISTANCE MODELOF COGNITIVE DIAGNOSISDimiter M. Dimitrov, Dimitar AtanasovAbstract. As the cognitive operations are hypothesized according to acognitive theory in the context of a study, they are latent (hidden) in natureand cannot be measured and scored directly from the test. The least squaresdistance model (LSDM) of cognitive diagnosis uses estimates of the itemparameters under a specific item-response theory (IRT) model to provideestimates of the probability of a person to process correctly any cognitiveattribute given the persons location on the IRT logit scale. In this paper amethodology for comparing two (or more) groups of individuals, accordingto their performance on a given set of cognitive attributes is presented.1. Introduction. The cognitive structure of a test in education, psychol-ogy, and other behavioral fields is typically defined by a set of cognitive oper-ations, processes or rules, referred to as cognitive attributes, and informationabout which attributes are required for the correct solution of each test item. Asthe attributes are hypothesized according to a cognitive theory in the contextof study, they are latent (hidden) in nature and cannot be measured and scoreddirectly. Knowledge about cognitive structures can help test developers, psychol-ogists, and educators to better understand the cognitive processes of thinking,learning, and performance. The least squares distance model of cognitive diagno-sis ([2], [3]) uses estimates of the item parameters under a specific item-response2010 Mathematics Subject Classification: 62P15Key words: cognitive diagnosis modeling, item response theory, assessment.34 D. M. Dimitrov, D. Atanasovtheory (IRT) model to provide estimates of the probability of a person to processcorrectly any cognitive attribute given the persons location on the IRT logit scale.The graphical depiction of such probabilities along the IRT logit scale for a givencognitive attribute is referred to its attribute probability curve (APC).One of the most popular and useful in practice methodology for studying theproperties of the test items is the so called Item Response Theory model (IRT)([1], [4]).In a 3-parametric IRT model the probability for a correct performance on agiven item from a student with ability on a logit scale θ can be represented as a3-parametric logistic curve. The difficulty parameter a represents the level of theability, required for a correct performance on the item. The discrimination pa-rameter b indicates how effectively the item discriminates between the examineeswho are relatively high on the criterion of interest against those who are relativelylow. The pseudo guessing parameter c represents the probability that examineeswith very low ability can guess the correct answer. Then the probability of acorrect performance is given by the following logistic function(1) P (θ) = c+ (1− c)exp(Db(θ − a))1 + exp(Db(θ − a)),where D is a constant, usually set to D = 1.7 when P (θ) approximates thenormal ogive curve. Plotted against θ it gives the so called Item CharacteristicCurves (ICC). An example of ICC for two items is presented on Figure 1. Theparameters of the Item 2 are shown. The difficulty of the item is the ability level,giving the probability of a correct performance equal to 0.5 if there is no guessing.The discrimination of the item is presented by the slope of the tangent at thepoint of difficulty. The guess parameter represents the probability for a correctitem response (just by guessing) from a subject with a small level of abilities.In general, in this example, Item 2 is more difficult, but less discriminative thanItem 1, having a larger value of the guessing parameter.LSDM ([2]) provides an interesting approach to a cognitive assessment. Cog-nitive diagnosis models are widely used in psychology and psychometrics in formof the conjunctive assumption that a correct item response is produced when allattributes required by the item are mastered. LSDM technique can be consid-ered as an extension of the classical models as it can deal with both conjunctiveand disjunctive assumptions ([3]). The basic idea behind the LSDM is that theprobability for correct performance on the item can be represented by a correctperformance on the required attributes. For example a conjunctive model forpositive response on the dichotomous item Dl, i = 1, . . . , N determined from aComparisons on Cognitive Attributes 35−3 −2 −1 0 1 2 30.10.20.30.40.50.60.70.80.91Item Characteristics Curves (ICC)  Item 1Item 2discriminationdifficultyguessFig. 1. Item Characteristic Curveset of attributes A1, . . . , Ak can be expressed as(2) P (Di = 1 | θ) =k∏l=1P (Al = 1 | θ)qil,where the values qil form the matrix Q and qil = 1, l = 1, . . . , k, if the item Direquires the attribute Al, and 0 otherwise.The purpose of this paper is to propose a LSDM-based probabilistic approachto comparing groups of examinees according to their performance on a set of cog-nitive attributes. Such groups can be defined, for example, by gender or differenttreatment conditions (say, experimental and control groups) in educational orpsychological research.For the computations and the representation of the results a MATLAB pack-age is used, available athttp://www.ir-statistics.net/index.cgi/software-irt,under the GNU License.2. The Algorithm. The proposed algorithm is demonstrated via a partic-ular example, taken from [2]. Let us have an IRT calibration and a Q-matrix ofa set of 10 items36 D. M. Dimitrov, D. AtanasovTable 1. IRT parameters of test itemsItem Difficulty Discr. Q-matrix1 −2.0300 0.6000 1 0 0 0 02 −1.2900 0.8100 0 1 0 0 03 −1.0300 0.7500 0 1 0 1 04 −1.5800 0.8100 0 1 0 0 05 0.5900 0.6200 0 1 1 0 06 −1.6500 0.7500 0 0 0 1 07 2.2200 0.5400 0 1 0 0 18 −1.4600 0.6500 0 0 0 1 09 2.5800 0.7500 0 0 0 1 110 −0.6600 0.5400 1 0 1 0 0First, the estimates of the test ability scores of the persons from the groupsbeing compared are obtained by fitting an appropriate IRT model to the sampledata (binary item scores, 1 = true, 0 = false). For example, on Table 2 the abilityscores for 60 persons, separated in three groups are provided. Let us note thatthe average ability in these groups is statistically different (with p-value equal to0.002).Second, the APCs of attributes (attributePerformance) that underlie thesuccess on test items (itemPerformance) are obtained through the use of theLSDM. The used functions are given in the following lines:itemPerformance = irt_item_performance(th,itemParameters)attributePerformance = lsdm(itemPerformance,Q,1)’opt.legend = 1;opt.colour = 1;plot_item(attributePerformance,th,opt)Third, these APCs are described analytically by fitting them to the 3-parameter logistic (3PL) model in IRT.attributeParameters = irt_fit(th, attributePerformance)Fourth, using this 3PL model of cognitive attributes, the probability for eachperson to process correctly a given attribute (i.e. the person’s attribute score) isobtained as a function of the test ability score of that person on the IRT logitscale.for g = [1,2,3] %for each groupComparisons on Cognitive Attributes 37−3 −2 −1 0 1 2 300.10.20.30.40.50.60.70.80.91  12345678910Fig. 2. Characteristic Curves of the test itemsperformanceForGroup{g} =irt_item_performance(ability(:,g),attributeParameters);end;Fifth, the obtained attribute scores for the persons in the study sample areused with an appropriate statistical method for group comparison (e.g., t-test,ANOVA, ANCOVA, and so forth). Here the MATLAB functions anova1 andmultcompare are used.for k = 1:5 % for each of the attributescompare = [];for g = [1,2,3] % for each groupcompare = [compare performanceForGroup{g}(k,:)’]end;[p,a,s] = anova1(compare)multcompare(s)end;According to the comparison presented on Figure 2 there is no between groupdifference based on performance of attribute A1. Whereas, a statistically sig-nificant difference between group 2 and group 3 can be found considering theperformance of the attribute A3 (Figure 2).38 D. M. Dimitrov, D. Atanasov−3 −2 −1 0 1 2 300.10.20.30.40.50.60.70.80.91  12345Fig. 3. Characteristic Curves of the attributes (APC)−3 −2 −1 0 1 2 300.10.20.30.40.50.60.70.80.91Fig. 4. Fit of the attribute curvesComparisons on Cognitive Attributes 39Table 2. Persons ability measuresPerson Group 1 Group 2 Group 31 0.5377 0.1715 0.89782 1.8339 −1.7075 0.75863 −2.2588 0.2172 1.31924 0.8622 1.1302 1.31295 0.3188 −0.0111 0.13516 −1.3077 0.5347 0.96997 −0.4336 0.2269 0.83518 0.3426 −0.8034 1.62779 3.5784 −0.2061 2.093310 2.7694 −1.2873 2.109311 −1.3499 0.3884 0.136312 3.0349 −1.6471 1.077413 0.7254 −1.5689 −0.214114 −0.0631 −1.3095 −0.113515 0.7147 −3.4443 0.993216 −0.2050 0.9384 2.532617 −0.1241 −0.1748 0.230318 1.4897 −1.2549 1.371419 1.4090 0.8703 0.774420 1.4172 −2.2115 2.11743. Conclusions. The proposed methodology shows how a between-groupcomparison can be achieved, based on the performing on a set of cognitive at-tributes. The considered example demonstrates that although that there is a sta-tistically significant difference in the average person ability between the groups,this difference can’t be considered as statistically significant at the level of thecognitive attribute performance in general. This difference can be caused by adistinct performing on one or more attributes with no difference in the others.The results from such group comparisons on the latent cognitive attributesprovide a valuable feedback to educators and psychologists for improving as-Table 3. IRT parameters of the attributesAttribute Diff. Discr. Guess1 −2.2257 1.6543 0.36692 −1.6742 1.0383 0.15183 0.3911 0.6280 0.13494 −1.2786 0.4939 0.05785 2.4264 0.6432 0.011940 D. M. Dimitrov, D. Atanasov0.88 0.9 0.92 0.94 0.96 0.98 1 1.02 1.04 1.06321Fig. 5. Between group comparison ofattribute A10.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8321Fig. 6. Between group comparison ofattribute A3sessment, teaching strategies, and curriculum development in targeted areas oflearning, teaching, and behavioural intervention.The usage of specially developed software package is demonstrated.REFERENCES[1] L. Crocker, J. Algila. Introduction to Classical and Modern Test Theory,Warsworth, 1986.[2] D. M. Dimitrov. Least squares distance method of cognitive validationand analysis for binary items using their item response theory parameters.Applied Psychological Measurement 31 (2007), No 5, 367–387.[3] D. M. Dimitrov, D. V. Atanasov. Conjunctive and Disjunctive Exten-sions of the Least Squares Distance Model of Cognitive Diagnosis. Educa-tional and Psychological Measurement, 2011.[4] E. Smith, R. Smith. Introduction to Rasch Measurement. JAM Press,Maple Grove, Minnesota US, 2004.Dimitar AtanasovNew Bulgarian University21, Montevideo Blvd1618 Sofia, Bulgariae-mail: datanasov@nbu.bgDimiter DimitrovGeorge Mason University, Fairfax CampusWest Building 2007 4400University Dr. MS 6D2 Fairfax, VA 22030e-mail: ddimitro@gmu.edu

Group Comparisons on Cognitive Attributes Using the Least Squares Distance Model of Cognitive Diagnosis

http://sci-gems.math.bas.bg/jspui/bitstream/10525/2510/1/Pliska-22-2013-033-040.pdf

Group Comparisons on Cognitive Attributes Using the Least Squares Distance Model of Cognitive Diagnosis

Abstract

Similar works

Full text

Available Versions

Bulgarian Digital Mathematics Library at IMI-BAS