Psychometric changes on item difficulty due to item review by examinees

If good measurement depends in part on the estimation of accurate item characteristics, it is essential that test developers become aware of discrepancies that may exist on the item parameters before and after item review. The purpose of this study was to examine the answer changing patterns of students while taking paper-and-pencil multiple choice exams, and to examine how these changes affect the estimation of item difficulty parameters. The results of this study have shown that item review by examinees does produce some changes to the examinee ability estimates and to the item difficulty parameters. In addition, these effects are more pronounced in shorter tests than in longer tests. In turn, these small changes produce larger effects when estimating the changes in the information values of each student’s test score. Accessed 4,021 times on https://pareonline.net from January 07, 2015 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right

Evaluating the Usefulness and Properties of a Subjective Assessment of Brazilian Portuguese

This is the published version. Copyright 2002 Johns Hopkins University Press.This is an evaluation of a Brazilian self-assessment test. All questions on the test guide the students to evaluate themselves on their linguistic and socio-cultural skills in Brazilian Portuguese. The main points discussed in this article are (1) an item analysis, (2) the reliability of the test to determine the consistency of the results obtained by the instrument, (3) the construct validity, and (4) the creation of cutscores. Our analysis of the BP self-assessment was supported by the use of descriptive statistics, by a factorial analysis and by a reliability test to determine the psychometrics characteristics of the test. The major claim in this study is that this test is useful, valid and reliable, if used appropriately, especially with an audience of motivated students such as students going abroad, instead of students who take language classes only as a requirement in their school program

Rearrangement procedure A 'REARRANGEMENT PROCEDURE' FOR SCORING ADAPTIVE TESTS WITH REVIEW OPTIONS Running head: Rearrangement procedure Rearrangement procedure

Abstract: Due to the increased popularity of computerized adaptive testing (CAT), many admissions tests, as well as certification and licensure exams have been transformed from their paper-and-pencil versions to computerized adaptive versions. A major difference between paper-and-pencil tests and CAT, from an examinee's point of view, is that in many cases examinees are not allowed to revise their answers on CAT. Examinees prefer item review since they can correct misread or miskeyed items, while some researchers, are afraid that examinees might try to use item review to cheat on the test. The purpose of this study is to test the efficiency of a 'rearrangement procedure' that rearranges and skips certain items in order to better estimate the examinees' abilities, without allowing them to cheat on the test. This was examined through a simulation study. The results show that the rearrangement procedure is effective in reducing the bias of the ability estimates. The reliability slightly decreased after the procedure, but this decrease was negligible

Coding schemes as measurement instruments? an attempt to assess the psychometric properties of a coding scheme

CSCL researchers rely heavily on coding schemes to analyze collaborative discourse. As Rourke and Anderson (2004) explain, examples of qualitative content analysis (QCA) research in which a coding scheme is developed methodically and validated systematically are rare. In this study, we attempted to conduct a validation of a coding scheme using established quantitative methods; this was an effort to foster the replicability and validity of a coding scheme designed to measure online collaborative learning. Exploratory Factor Analysis suggested a different factor structure than the one conceptualized in the coding scheme. The authors raise questions related to what are the most appropriate and acceptable procedures in establishing the validity of coding schemes used in QC