Meijer, Rob R.

English

University of Twente Research Information

Book ReviewTest ScoringEdited by David Thissen and Howard WainerMahwah, NJ: Lawrence Erlbaum, 2001, 422 pp., $99.95 (hardcover)ISBN 0-8058-3766-3This is a great book. Sometimes when you review a book you find yourself doing research andnot writing the review because the topics discussed in the book fascinate you, and doing researchis always more fun than writing reviews. This is that kind of book, at least for me. When I readchapter 7, “Item Response Theory Applied to Combinations of Multiple-Choice and ConstructedResponse Items—Scale Scores for Patterns of Summed Scores,” my thoughts were, “Here we havea good idea to compare subtest scores on an individual level; let’s see if it works.” It works if youtake a slightly different approach than that described in the book (Meijer, 2002).As the book title implies, this is a book about how to obtain test scores and in particular, how toobtain test scores from tests that consist of a combination of multiple-choice items and open-endedquestions. The book has nine chapters. After an introduction, where an overview of test scoringis given, the first part of the book (chapters 2, 3, and 4) deals with test scoring using classicaltest theory and item response theory (IRT) for dichotomously and polytomously scored items. Thesecond part (chapters 5 and 6) deals with the factor analytic approach for test scoring using bothdichotomously and polytomously scored items. Finally, the third part (chapters 7, 8, and 9) is aboutapplications and special problems in test scoring.The first chapter (Wainer and Thissen) provides an overview of test scoring. The purpose of thischapter is to sketch out what kinds of problems exist when scoring a test. The chapter discusses twotests: a computer skills test consisting of multiple-choice items and a reading test consisting of bothmultiple-choice items and open-ended questions. These two tests are used throughout the bookto illustrate the different scoring techniques. In the second chapter (Wainer and Thissen), scoringaccording to classical test theory is discussed, including the concept of reliability, Spearman-Brownformula, and coefficient alpha. Both the simple case, where each item measures the same construct,and a more complex case, where the test consists of different sections and reliability of compositescores needs to be determined, are discussed. The chapter ends with the description of differentmodels to characterize measurement error.In chapter 3 (Thissen and Orlando), IRT models for dichotomously scored items are discussed.The logistic one-, two-, and three-parameter item response models are presented, as well as thenormal ogive model, complete with the history and rationale behind this model. In a book abouttest scoring, the heart of the chapter deals with the estimation of proficiency, where the principlesof maximum likelihood, maximum a posteriori, and expected a posteriori estimation are presented.Applied Psychological Measurement, Vol. 27 No. 1, January 2003, 75–77DOI: 10.1177/014662102237799© 2003 Sage Publications 75Volume 27 Number 1 January 200376 APPLIED PSYCHOLOGICAL MEASUREMENTLogically, IRT models for polytomously scored items are discussed in chapter 4 (Thissen, Nelson,Rosa, & McLeod). Item parameter estimation and scale score estimation for polytomously scoreditems under the graded response, the nominal response, and the partial credit model are presented.Numerous examples are provided of scoring methods both for tests consisting of polytomouslyscored items and tests that combine dichotomously scored items and polytomously scored items.For example, how to score a third grade reading test that consists of 16 multiple-choice items and4 constructed response items is described.Chapter 5 (McLeod, Swygert, & Thissen) reviews factor analysis for items scored in two cate-gories. First, traditional factor analysis is described, followed by item factor analysis, multidimen-sional IRT, and full information factor analysis. At the end of the chapter, alternative methods toestablish multidimensionality such as DIMTEST are briefly discussed. Building on chapter 5, chap-ter 6 (Swygert, McLeod, & Thissen) describes factor analysis for items and testlets scored in morethan two categories. In this chapter, structural equating modeling is reviewed. Moreover, full infor-mation factor analysis for polytomous items is used to examine the degree to which multiple-choiceand constructed response items appear to measure the same construct. The conclusion at the endof the chapter is that “the theory underlying full-information models is complex . . . , the softwareitself is difficult to use” (p. 247) and that “no currently available implementation of full informationitem factor analysis for polytomous items permits confirmatory factor analysis” (p. 248).In chapter 7 (Rosa, Swygert, Nelson, & Thissen), a technique is discussed for combining sub-scale scores based on IRT. To illustrate the technique, the likelihoods for the summed scores on amultiple-choice section and an open-ended section are calculated and combined for each patternof summed scores. Thus, instead of patterns of item scores, patterns of summed scores are used. Inchapter 8 (Thissen, Nelson, & Swygert), this procedure is refined by using weighted linear combi-nations of the subscale scores. Finally, in chapter 9, an empirical Bayes estimation is introduced tocompute reliable estimates of (sub)scale scores. This “borrowing from strength” technique may beparticularly useful in the context where subscale scores are needed for diagnostic purposes despitethe fact that the shorter subscale scores have limited reliability.The strength of this book is that scoring solutions are presented for a diversity of real-worldscoring problems. In each chapter, first the theory is presented and then solutions are illustratedusing existing tests. Another strong point is the writing, which is clear and vivid. The book isoriented toward educational testing, the tests that are used in assessing educational achievement.The techniques discussed deal with practical solutions and are not overly concerned with generalityor theory building.What is the appropriate audience for this book? The technical level of the chapters varies greatly.Most chapters are relatively easy to read with some background in elementary statistics and/or IRT.On the other hand, chapters such as 6 (on factor analysis for polytomous items) and 9 (empiricalBayes) are more technical. Although the book is intended for both students and researchers, it isless of a textbook than introductory books on IRT such as that of Embretson and Reise (2000).To return to my own research that was inspired by chapter 7, the authors claim that usinghighest density regions (HDR) may be a good alternative to using statistical distributions to flagunexpected score combinations within an examinee. That is an interesting idea. However, they useHDR integrating over the complete proficiency distribution, which means that a score combinationis unexpected given the sum score combinations for all examinees in the population. In someapplications, this may be useful; however, in general a researcher is interested whether an examinee’sitem scores are unexpected given his or her proficiency level. Therefore, it is often more convenientR. R. MEIJERBOOK REVIEW 77to determine a distribution conditional on the estimated proficiency level. In conclusion, this is aninspiring book that should be on the shelf of every researcher in psychological and educationaltesting.Rob R. MeijerUniversity of Twente, the NetherlandsReferencesEmbretson, S. E., & Reise, S. P. (2000) Item responsetheory for psychologists. Mahwah, NJ: LawrenceErlbaum.Meijer, R. R. (2002). Using patterns of summed scoresin paper-and-pencil tests and CAT to detect misfit-ting item score patterns. Unpublished manuscript.Author’s AddressSend requests for reprints or further informationto Rob R. Meijer, University of Twente, Depart-ment of Educational Measurement and Data-analysis,P.O. Box 217, 7500 AE Enschede, the Netherlands.E-mail: meijer@edte.utwente.nl.

Book review: Test scoring

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information

NARCIS