Psychometrics in Practice at RCEC

A broad range of topics is dealt with in this volume: from combining the psychometric generalizability and item response theories to the ideas for an integrated formative use of data-driven decision making, assessment for learning and diagnostic testing. A number of chapters pay attention to computerized (adaptive) and classification testing. Other chapters treat the quality of testing in a general sense, but for topics like maintaining standards or the testing of writing ability, the quality of testing is dealt with more specifically.\ud All authors are connected to RCEC as researchers. They present one of their current research topics and provide some insight into the focus of RCEC. The selection of the topics and the editing intends that the book should be of special interest to educational researchers, psychometricians and practitioners in educational assessment

Combining decision trees and stochastic curtailment for assessment length reduction of test batteries used for classification.

For classification problems in psychology (e.g., clinical diagnosis), batteries of tests are often administered. However, not every test or item may be necessary for accurate classification. In the current article, a combination of classification and regression trees (CART) and stochastic curtailment (SC) is introduced to reduce assessment length of questionnaire batteries. First, the CART algorithm provides relevant subscales and cutoffs needed for accurate classification, in the form of a decision tree. Second, for every subscale and cutoff appearing in the decision tree, SC reduces the number of items needed for accurate classification. This procedure is illustrated by post hoc simulation on a data set of 3,579 patients, to whom the Mood and Anxiety Symptoms Questionnaire (MASQ) was administered. Subscales of the MASQ are used for predicting diagnoses of depression. Results show that CART-SC provided an assessment length reduction of 56%, without loss of accuracy, compared with the more traditional prediction method of performing linear discriminant analysis on subscale scores. CART-SC appears to be an efficient and accurate algorithm for shortening test batteries. © The Author(s) 2013

Computerized adaptive test and decision trees: A unifying approach

In the last few years, several articles have proposed decision trees (DTs) as an alternative to computerized adapted tests (CATs). These works have focused on showing the differences between the two methods with the aim of identifying the advantages of each of them and thus determining when it is preferable to use one method or another. In this article, Tree-CAT, a new technique for building CATs is presented. Unlike the existing work, Tree-CAT exploits the similarities between CATs and DTs. This technique allows the creation of CATs that minimise the mean square error in the estimation of the examinee’s ability level, and controls the item’s exposure rate. The decision tree is sequentially built by means of an innovative algorithmic procedure that selects the items associated with each of the tree branches by solving a linear program. In addition, our work presents further advantages over alternative item selection techniques with exposure control, such as instant item selection or simultaneous administration of the test to an unlimited number of participants. These advantages allow accurate on-line CATs to be implemented even when the item selection method is computationally costly.Numerical experiments were conducted in Uranus, a supercomputer cluster located at Universidad Carlos III de Madrid and jointly funded by EU-FEDER funds and by the Spanish Government via the National Projects No. UNC313-4E-2361, No. ENE2009-12213- C03-03, No. ENE2012-33219, No. ENE2012-31753 and No. ENE2015-68265-P

cat.dt: An R package for fast construction of accurate Computerized Adaptive Tests using Decision Trees

This article introduces the cat.dt package for the creation of Computerized Adaptive Tests (CATs). Unlike existing packages, the cat.dt package represents the CAT in a Decision Tree (DT) structure. This allows building the test before its administration, ensuring that the creation time of the test is independent of the number of participants. Moreover, to accelerate the construction of the tree, the package controls its growth by joining nodes with similar estimations or distributions of the ability level and uses techniques such as message passing and pre-calculations. The constructed tree, as well as the estimation procedure, can be visualized using the graphical tools included in the package. An experiment designed to evaluate its performance shows that the cat.dt package drastically reduces computational time in the creation of CATs without compromising accuracy.This article has been funded by the Spanish National Project No. RTI2018-101857-B-I00

Merged Tree-CAT: A fast method for building precise computerized adaptive tests based on decision trees

Over the last few years, there has been an increasing interest in the creation of Computerized Adaptive Tests (CATs) based on Decision Trees (DTs). Among the available methods, the Tree-CAT method has been able to demonstrate a mathematical equivalence between both techniques. However, this method has the inconvenience of requiring a high performance cluster while taking a few days to perform its computations. This article presents the Merged Tree-CAT method, which extends the Tree-CAT technique, to create CATs based on DTs in just a few seconds in a personal computer. In order to do so, the Merged Tree-CAT method controls the growth of the tree by merging those branches in which both the distribution and the estimation of the latent level are similar. The performed experiments show that the proposed method obtains estimations of the latent level which are comparable to the obtained by the state-of-the-art techniques, while drastically reducing the computational time.Numerical experiments were conducted in Uranus, a supercomputer cluster located at Universidad Carlos III de Madrid and jointly funded by EU-FEDER funds and by the Spanish Government via the National Projects nos. UNC313-4E-2361, ENE2009-12213- C03-03, ENE2012-33219, ENE2012-31753 and ENE2015-68265-P. This article was also funded by the Spanish National Project no. RTI2018-101857-B-I00

Recursive partitioning vs computerized adaptive testing to reduce the burden of health assessments in cleft lip and/or palate : comparative simulation study

Background: Computerized adaptive testing (CAT) has been shown to deliver short, accurate, and personalized versions of the CLEFT-Q patient-reported outcome measure for children and young adults born with a cleft lip and/or palate. Decision trees may integrate clinician-reported data (eg, age, gender, cleft type, and planned treatments) to make these assessments even shorter and more accurate. Objective: We aimed to create decision tree models incorporating clinician-reported data into adaptive CLEFT-Q assessments and compare their accuracy to traditional CAT models. Methods: We used relevant clinician-reported data and patient-reported item responses from the CLEFT-Q field test to train and test decision tree models using recursive partitioning. We compared the prediction accuracy of decision trees to CAT assessments of similar length. Participant scores from the full-length questionnaire were used as ground truth. Accuracy was assessed through Pearson’s correlation coefficient of predicted and ground truth scores, mean absolute error, root mean squared error, and a two-tailed Wilcoxon signed-rank test comparing squared error. Results: Decision trees demonstrated poorer accuracy than CAT comparators and generally made data splits based on item responses rather than clinician-reported data. Conclusions: When predicting CLEFT-Q scores, individual item responses are generally more informative than clinician-reported data. Decision trees that make binary splits are at risk of underfitting polytomous patient-reported outcome measure data and demonstrated poorer performance than CATs in this study

Adaptive Assessment and Guessing Detection Implementation

Computerized adaptive testing (CAT) is a context-based adaptive assessment. However, the assessment result may not be valid because the examinee might cheat or guess the answers. Although there are many guessing detection methods, there are not many discussions about their implementation into CAT. Therefore, this paper presents an example of a modification of an existing software so the newly modified software can detect guessed answers and be able to select questions adaptively. The system can detect assuming behavior by recording the examinee’s answer time. Also, the designed system can like questions adaptively by connecting Fuzzy logic, which calculates what level the question should select for the next iteration. The system is responded well by elementary and college students. A total of 56.6% felt the system was straightforward to use. The detection methods can detect guessing behavior of about 73%. However, the system’s sensitivity is low if the method is forced to classify answers which answered in a long response time / general guessing. Nevertheless, when we limit the data classified within 10s response time (rapid-guessing), the method’s sensitivity rises to 68.78%