Multidimensional IRT models (MIRTM), developed in the fields of psychometrics
and ability assessment, are here considered in connection with the problem of evaluating customer satisfaction. Different models, that allow us to take into account more complex and, possibly, more realistic latent constructs than those usually assumed, are presented and discussed. Eventually, these models are applied to a real dataset, MCMC techniques for the estimation are implemented and analogies and differences with results from previous analyses on the same survey in the literature are discussed

F. Andreis

P. A. Ferrari

Multidimensional IRT models (MIRTM), developed in the fields of psychometrics
and ability assessment, are here considered in connection with the problem of evaluating customer satisfaction. Different models, that allow us to take into account more complex and, possibly, more realistic latent constructs than those usually assumed, are presented and discussed. Eventually, these models are applied to a real dataset, MCMC techniques for the estimation are implemented and analogies and differences with results from previous analyses on the same survey in the literature are discussed

AIR Universita degli studi di Milano

Quaderni di StatisticaVol. 14, 2012Multidimensional extensions of IRT models andtheir application to customer satisfactionevaluationFederico Andreis, Pier Alda FerrariDepartment of Economics, Business and Statistics - Universita` degli Studi di MilanoSummary: Multidimensional IRT models (MIRTM), developed in the fields of psycho-metrics and ability assessment, are here considered in connection with the problem ofevaluating customer satisfaction. Different models, that allow us to take into accountmore complex and, possibly, more realistic latent constructs than those usually assumed,are presented and discussed. Eventually, these models are applied to a real dataset,MCMC techniques for the estimation are implemented and analogies and differenceswith results from previous analyses on the same survey in the literature are discussed.Keywords: BINARY RESPONSES, COMPENSATORY MODELS, MIRT.1. IntroductionIt has become increasingly clear in the last decades that, in complex settings whereunobservable (latent) quantities are of interest, the usual IRT assumption of a singleunderlying component influencing the observable outcomes might be not realistic [5,Chap.3]. Multidimensional IRT models (MIRTM) arise from the fields of psychometricsand ability assessment (as do their ancestors, the unidimensional IRT models, UIRTM),their aim being to overcome this limitation. The rationale behind these techniques isto provide an instrument capable of describing the usually not trivial apparatus of skillsthat a person brings to a test, in order to obtain a diagnostic tool about several subscalessimultaneously and a way to model the interaction between examinees and test items’characteristics. It is important to point out, though, that this approach is not intended toprovide a measurement of the latent trait in the sense Rasch introduced [4] (i.e. objectivemeasurement), rather than to investigate from a modeling point of view the complexityof such unobservable phenomenon, thus following the statistical logic of model tun-ing in order to fit data, i.e. choosing the best tool available. Such distinction has been2 F. Andreis and P.A. Ferrari.highlighted since the beginnings: the Rasch model is a theoretical ideal, a definition ofmeasurement, as opposed to statistical modeling, which is a toolbox we reckon beinguseful to try and disentangle not trivial intertwinings within the underlying (often veryarticulated) reality. Along these lines, our interest is into evaluating the possibility ofemploying MIRTM in a field they haven’t previously been applied to, specifically cus-tomer satisfaction, assessing interpretability of models’ parameters and comparing thisapproach to current methodology.2. Multidimensional IRT modelsDifferent approaches to multi-dimensionality have been carried out in the literature,leading to the definition of two main classes of models, the ’compensatory’ and ’non(or partially) compensatory’ ones [5, Chap. 4] and to the introduction of the conceptsof ’between-items’ and ’within-items’ dimensionality [1]. Compensatory models allow,through proper parameterization, latent traits’ effects to compensate for each other, e.g.a high level of ability on one dimension can make up for a low level on another - thinkof an additive functional form for the parameters; non-compensatory (even though itwould be more correct to say ’partially’ compensatory) models do not admit such strongcompensation - think of a multiplicative form. Between-items and within-items dimen-sionality embody assumptions regarding how latent traits are represented by items ina questionnaire, i.e. if each item is related to one, and only one, of the latent traits(between-items), or is to be linked to more than one at the same time (within-items).Figure 1. Between-items (a) and within-items (b) dimensionality with two latent traits.MIRTM and Customer Satisfaction 3Multidimensional extensions of classic IRT models have been introduced; they per-mit to work with dichotomous or polytomous test responses, but also to include covari-ates [5, Chap. 4].Since MIRTM require a greater number of parameters to be estimated than unidimen-sional IRT models, estimation issues arise and are addressed. In the literature, MCMCmethods are advocated as a helpful tool to obtain accurate results, but implementationof the algorithms and assessment of convergence to the desired posterior distributionrequire careful assessment [6].3. Customer satisfaction assessment with MIRTMRecently, unidimensional IRT models have been applied to the field of customersatisfaction (CS) evaluation, via a convenient re-interpretation of the role of their pa-rameters [2]. Since it is legitimate to think about the existence of more than one latentfactor in CS-related investigations too, the aim of this work is to evaluate the possibilityof identifying and applying suitable MIRTM to this context.In order to better appreciate the meaning and additional contribution of the extension tomore than one latent trait, we first review the basic one-dimensional dichotomous model,whose expression is given by:P (Xij = x|θi, βj) = ex(θi−βj)1 + eθi−βj(1)where x = 0, 1 if the customer is, respectively, unsatisied or satisfied, θi is satisfactionfor the i−th respondent and (−βj) the quality of the j−th item. This model is alge-braically equivalent to that introduced by Rasch [4] in order to evaluate ability tests, θibeing the person ability and βj the item difficulty.The multidimensional extension we consider is called multidimensional 2 parameterlogistic (M2PL) model, and has the following expression:P (Xij = x|θi,aj , dj) = ex(a′jθi+dj)1 + ea′jθi+dj(2)where both aj and θi are m−dimensional vectors, so that the sum in the exponent of ecan be rewritten as a′jθi + dj =∑mt=1 ajtθit + dj . This model assumes that m latenttraits characterize the satisfaction, with single scores θit whose relevance is weighted bythe ajt parameters, known as items’ discrimination related to the t−th dimension, andhere intended to describe the relevance of the j−th item to the t−th trait. The M2PLbelongs to the class of compensatory models, since it adopts a parameterization that isa linear combination of the satisfaction parameters (one for each presumed dimensionof the latent construct) for each individual (customer/user), i.e. a high satisfaction valueθit on one latent dimension can compensate for a low θis on another.4 F. Andreis and P.A. Ferrari.The ajt parameters may be estimated from the data or be fixed according to particularassumptions. If all are assumed to be equal to a fixed value ajt = a∗, then a′jθi + dj =a∗∑mt=1 θit + dj , and we assume that the discriminating power of all the items is thesame, also across all dimensions; this means that different items have different quality,but the same relevance to each trait. If a∗ = 1, we obtain the initial model, with thepositions θi =∑mt=1 θit, dj = −βj , thus obtaining a mere reparameterization of theθi parameters from the unidimensional model, not taking into account dimensionality,i.e. collapsing the model back to one latent trait only. Another possibility is to fix theajt either to be zero or different from zero: this way, it becomes possible to embodyassumptions about between- or within-items dimensionality. If the researcher believes acertain item (or group of items) to be dependent on one latent trait only, for example thes−th, this can be implemented in the model by letting ajt = 0, t 6= s, and ais either tobe estimated or fixed to some non-zero value; this is the structure in Fig.1 - (a).4. ApplicationEventually, the models we discuss are applied to real data, presented in [3, Chap. 2];the dataset consists of responses (on a 1-5 Likert scale, that we dichotomize for our pur-poses) to a questionnaire concerning satisfaction for a large firm’s services. Analogiesand differences with previous IRT-oriented analyses on the same survey [3, Chap. 14]are investigated. The results are then compared and discussed.References[1] Adams, R.J., Wilson, M. and WANG, W. (1997), The MultidimensionalRandom Coefficients Multinomial Logit Model, Applied Psychological Mea-surement, 21-1, 1-23 .[2] De Battisti, F., Nicolini, G. and Salini, S. (2005), The Rasch Model toMeasure Service Quality, The ICFAI Journal of Services Marketing, III-3, 58-80.[3] Kennet, R.S. and Salini, S. (2012), Modern Analysis of Customer Satis-faction Surveys: with applications using R, New York: Wiley.[4] Rasch, G. (1960). Probabilistic models for some intelligence and attain-ment tests. (Copenhagen, Danish Institute for Educational Research).[5] Reckase, M.D. (2009), Multidimensional Item Response Theory, Springer,Statistics for Social and Behavioral Sciences.[6] Sinharay, S. (2004), Experiences With Markov Chain Monte Carlo Con-vergence Assessment in Two Psychometric Examples, Journal of Educationaland Behavioral Statistics, 29, 461-488 .

Multidimensional extensions of IRT models and
their application to customer satisfaction
evaluation

https://air.unimi.it/retrieve/handle/2434/179306/197397/short%20paper.pdf

Multidimensional extensions of IRT models and their application to customer satisfaction evaluation

Abstract

Similar works

Full text

Available Versions

AIR Universita degli studi di Milano