On the efficiency of IRT models when applied to different sampling designs

The problem of obtaining designs that result in the greatest precision of the parameter estimates is encountered in at least two situations in which item response theory (IRT) models are used. In so-called two-stage testing procedures, certain designs may be specified that match difficulty levels of test items with abilities of examinees. The advantage of such designs is that the variance of the estimated parameters can be controlled. In situations in which IRT models are applied to different groups, efficient multiple-matrix sampling designs are applicable. The choice of matrix sampling designs will also influence the variance of the estimated parameters. Heuristic arguments are given here to formulate the efficiency of a design in terms of an asymptotic generalized variance criterion, and a comparison is made of the efficiencies of several designs. It is shown that some designs may be found to be most efficient for the one- and two- parameter model, but not necessarily for the three-parameter model. Index terms: efficiency, generalized variance, item response theory, optimal design

A general approach to algoritmic design of fixed-form tests, adaptive tests, and testlets

The selection of items from a calibrated item bank for fixed-form tests is an optimal test design problem; this problem has been handled in the literature by mathematical programming models. A similar problem, however, arises when items are selected for an adaptive test or for testlets. This paper focuses on the similarities of optimal design of fixed-form tests, adaptive tests, and testlets within the framework of the general theory of optimal designs. A sequential design procedure is proposed that uses these similarities. This procedure not only enables optimal design of fixed-form tests, adaptive tests, and testlets, but is also very flexible. The procedure is easy to apply, and consistent estimates for the trait level distribution are obtained. Index terms: adaptive tests, consistency, efficiency, optimal test design, sequential procedure, test design, testlets

Empirical comparison between factor analysis and item response models

Many multidimensional item response theory (IRT) models have been proposed. A comparison is made between the so-called full information models and the models that use only pairwise information. Three multidimensional models described are: (1) the compensatory model of R. D. Bock and M. Aitken (1981) using the computer program TESTFACT; (2) a model based on R. P. McDonald's (1985) harmonic analysis using the program NOHARM; and (3) the computer program MAXLOG of R. L. McKinley and M. D. Reckase (1983). Five factor analysis procedures for dichotomous items are discussed. A simulation study was conducted to compare the various methods. The item parameters of four different sets of items were used with numbers of subjects set at 250, 500, and 1,000. Ten replications were generated for each set of item parameters and each sample size. All models were compared with respect to estimates of IRT and factor analysis parameters using six criteria in terms of mean squared differences between the known and estimated item parameters. The most striking result of the simulation study was that common factor analysis programs outperformed the more complex programs TESTFACT, MAXLOG, and NOHARM. It was apparent that a common factor analysis in the matrix of tetrachoric correlations yielded the best estimates. A procedure based on the mean squared residuals of the correlation matrix was also presented for assessing the dimensionality of the model. Nine tables present the data from the simulation study

D-optimal sequential sampling designs for item response theory models

The selection of optimal designs in IRT models encounters at least two problems. The first problem is that Fisher’s information matrix is generally not independent of the values of the IRT parameters, and the second problem is that the design points are unknown parameters and have to be estimated together with the other parameters. In this study, these two problems are taken care of by a sequential design procedure. This procedure is a modification of a D-optimality procedure proposed by Wynn (1970). The results show that this algorithm leads to consistent estimates and that errors in selecting the abilities generally do not affect optimality very much

Optimal item discrimination and maximum information for logistic IRT models

Items with the highest discrimination parameter values in a logistic item response theory model do not necessarily give maximum information. This paper derives discrimination parameter values, as functions of the guessing parameter and distances between person parameters and item difficulty, that yield maximum information for the three-parameter logistic item response theory model. An upper bound for information as a function of these parameters is also derived. An algorithm is suggested for the maximum information item selection criterion for adaptive testing and is compared with a full bank search algorithm

On the assessment of dimensionality in multidimensional item response theory models

The assessment of dimensionality of data is important to item response theory (IRT) modelling and other multidimensional data analysis techniques. The fact that the parameters from the factor analysis formulation for dichotomous data can be expressed in terms of the parameters in the multidimensional IRT model suggests that the assessment of the dimensionality of the latent trait space can also be approached from the factor analytical viewpoint. Some problems connected with the assessment of the dimensionality of the latent space are discussed, and the conclusions are supported by simulated results for sample sizes of 250 and 500 on a 15-item test. Five tables contain data from the simulation; and 48 graphs illustrate eigenvalues and plotted mean residuals

A simple and fast item selection procedure for adaptive testing

Items with the highest discrimination parameter values in a logistic item response theory (IRT) model do not necessarily give maximum information. This paper shows which discrimination parameter values (as a function of the guessing parameter and the distance between person ability and item difficulty) give maximum information for the three-parameter logistic IRT model. The optimal discrimination parameter value is shown to be inversely related to the distance between item difficulty and person ability. An upper bound for the information as a function of these parameters is derived; and this upper bound is used to formulate a fast item selection algorithm for adaptive testing. In a small simulation study this algorithm was one and one half to six times as fast as an algorithm in which the information of all items in an item bank is calculated

Some new item selection criteria for adaptive testing

In this study some alternative item selection criteria for adaptive testing are proposed. These criteria take into account the uncertainty of the ability estimates. A general weighted information criterion is suggested of which the usual maximum information criterion and the suggested alternative criteria are special cases. A simulation study was conducted to compare the different criteria. The results showed that the likelihood weighted mean information criterion was a good alternative to the maximum information criterion. Another good alternative was a maximum information criterion with the maximum likelihood estimate of ability replaced by the Bayesian EAP estimate. An appendix discusses the interval information criterion for the two- and three-parameter logistic item response theory model

A review of selection methods for optimal test design

The designing of tests has been a source of concern for test developers over the past decade. Various kinds of test forms have been applied. Among these are the fixed-form test, the adaptive test, and the testlet. Each of these forms has its own design. In this paper, the construction of test forms is placed within the general framework of optimal design theory. A review of various objective functions and methods for the designing of different test forms is given. The advantages of using these methods are discussed, and an illustration of an optimal test design is provided