A power study of a compromised item detection procedure based on item response theory under different scenarios of subjects’ latent trait

This thesis explores whether or not the changing of students’ latent trait influences the power and the lag performance of a detection procedure based on Item Response Theory for successfully identifying a compromised item in computerized adaptive testing. A simulation study was conducted under three scenarios. The first scenario is the regular scenario, where the students’ latent trait follows the standard normal distribution. In the other two scenarios, the mean of true ability of student population can change in different pattern but not due to the item compromising. Therefore, this simulation mimics two more difficult scenarios, where one shows the ability with linear growth, and the other one has the ability with periodical variation. The simulation experiment yielded five main findings. (1) The mean and median of the distribution of the ability with linear growth scenario were larger than that under the other two scenarios, and the dispersion level of the distribution of the ability with periodical variation scenario is wider than that under the other two conditions; (2) The critical value c_0.01 is always higher than c_0.05, and the value of the moving sample size hardly affects the critical value c_α when moving sample size is greater than 20; (3) Nearly all of the items in the item pool are monitored under all three scenarios conducted in the simulation; (4) The detection procedure always holds a high quality of power (almost stays at 1 all the time); that is, it would not be affected by the changing of students’ latent traits in terms of the power index; (5) The critical values would produce a little bit longer lag under the setting of ability with linear growth scenario than the regular scenario, and there is no difference between the regular scenario and the ability with periodical variation scenario; 6)There is no significant difference of the value of power between α=0.01 and α=0.05, and also for lag

The effect of reading engagement on scientific literacy – an analysis based on the XGBoost method

Scientific literacy is a key factor of personal competitiveness, and reading is the most common activity in daily learning life, and playing the influence of reading on individuals day by day is the most convenient way to improve the level of scientific literacy of all people. Reading engagement is one of the important student characteristics related to reading literacy, which is highly malleable and is jointly reflected by behavioral, cognitive, and affective engagement, and it is of theoretical and practical significance to explore the relationship between reading engagement and scientific literacy using reading engagement as an entry point. In this study, we used PISA2018 data from China to explore the relationship between reading engagement and scientific literacy with a sample of 15-year-old students in mainland China. 36 variables related to reading engagement and background variables (gender, grade, and socioeconomic and cultural status of the family) were selected from the questionnaire as the independent variables, and the score of the Scientific Literacy Assessment (SLA) was taken as the outcome variable, and supervised machine learning method, the XGBoost algorithm, to construct the model. The dataset is randomly divided into training set and test set to optimize the model, which can verify that the obtained model has good fitting degree and generalization ability. Meanwhile, global and local personalized interpretation is done by introducing the SHAP value, a cutting-edge machine model interpretation method. It is found that among the three major components of reading engagement, cognitive engagement is the more influential factor, and students with high reading cognitive engagement level are more likely to get high scores in scientific literacy assessment, which is relatively dominant in the model of this study. On the other hand, this study verifies the feasibility of the current popular machine learning model, i.e., XGBoost, in a large-scale international education assessment program, with a better model adaptability and conditions for global and local interpretation

A power study of a compromised item detection procedure based on item response theory under different scenarios of subjects’ latent trait

This thesis explores whether or not the changing of students’ latent trait influences the power and the lag performance of a detection procedure based on Item Response Theory for successfully identifying a compromised item in computerized adaptive testing. A simulation study was conducted under three scenarios. The first scenario is the regular scenario, where the students’ latent trait follows the standard normal distribution. In the other two scenarios, the mean of true ability of student population can change in different pattern but not due to the item compromising. Therefore, this simulation mimics two more difficult scenarios, where one shows the ability with linear growth, and the other one has the ability with periodical variation. The simulation experiment yielded five main findings. (1) The mean and median of the distribution of the ability with linear growth scenario were larger than that under the other two scenarios, and the dispersion level of the distribution of the ability with periodical variation scenario is wider than that under the other two conditions; (2) The critical value c_0.01 is always higher than c_0.05, and the value of the moving sample size hardly affects the critical value c_α when moving sample size is greater than 20; (3) Nearly all of the items in the item pool are monitored under all three scenarios conducted in the simulation; (4) The detection procedure always holds a high quality of power (almost stays at 1 all the time); that is, it would not be affected by the changing of students’ latent traits in terms of the power index; (5) The critical values would produce a little bit longer lag under the setting of ability with linear growth scenario than the regular scenario, and there is no difference between the regular scenario and the ability with periodical variation scenario; 6)There is no significant difference of the value of power between α=0.01 and α=0.05, and also for lag

Data_Sheet_1_The effect of reading engagement on scientific literacy – an analysis based on the XGBoost method.docx

Scientific literacy is a key factor of personal competitiveness, and reading is the most common activity in daily learning life, and playing the influence of reading on individuals day by day is the most convenient way to improve the level of scientific literacy of all people. Reading engagement is one of the important student characteristics related to reading literacy, which is highly malleable and is jointly reflected by behavioral, cognitive, and affective engagement, and it is of theoretical and practical significance to explore the relationship between reading engagement and scientific literacy using reading engagement as an entry point. In this study, we used PISA2018 data from China to explore the relationship between reading engagement and scientific literacy with a sample of 15-year-old students in mainland China. 36 variables related to reading engagement and background variables (gender, grade, and socioeconomic and cultural status of the family) were selected from the questionnaire as the independent variables, and the score of the Scientific Literacy Assessment (SLA) was taken as the outcome variable, and supervised machine learning method, the XGBoost algorithm, to construct the model. The dataset is randomly divided into training set and test set to optimize the model, which can verify that the obtained model has good fitting degree and generalization ability. Meanwhile, global and local personalized interpretation is done by introducing the SHAP value, a cutting-edge machine model interpretation method. It is found that among the three major components of reading engagement, cognitive engagement is the more influential factor, and students with high reading cognitive engagement level are more likely to get high scores in scientific literacy assessment, which is relatively dominant in the model of this study. On the other hand, this study verifies the feasibility of the current popular machine learning model, i.e., XGBoost, in a large-scale international education assessment program, with a better model adaptability and conditions for global and local interpretation.</p