Methodological issues in large-scale educational surveys

This thesis focuses on the application of item response theory (IRT) in the context of large scale international educational surveys like PISA 2009 (OECD). Although IRT methodology has been widely used in educational applications such as test construction, norming of examinations, detection of item bias and computerized adaptive testing, large scale surveys present a number of specific problems. A number of these problems are addressed in this thesis using PISA student data of the 2006 and 2009 cycles. The first problem in international comparative educational tests relates to the detection of cultural bias over countries. In this thesis, we targeted a problem know as country-specific Differential Item Functioning (CDIF) or country-by-item-interaction. This problem is tackled by modeling CDIF using country specific item parameters. In Chapter 2 this methodology is applied to the background questionnaires of the PISA 2009 cycle. In Chapter 3, this methodology is outlined and applied to the Reading dataset of the PISA 2006 cycle. The results showed that the impact of CDIF on the ranking of countries is not prominent and becomes almost negligible when the statistical uncertainty regarding the country means is properly taken into account. In Chapter 4, the practical significance of modeling item bias on the background questionnaire scales was studied not only in terms of the ordering of countries on the respective scales but also its impact on the results of regression analyses with latent variables in survey research. The next topic relates to the combination of the results of IRT measurement models with multilevel structural models to relate cognitive outcomes to background variables. According to our findings, results obtained using separately estimated plausible values and latent covariates (as provided in large scale data sets like PISA) were comparable with the results obtained using a concurrent estimation with raw data for the various kinds of IRT measurement models embedded in the structural (regression) model. The last Chapter gives an example of an advanced latent regression model based on plausible value methodology that explores the relation between socio-economic status and reading achievement in PISA

Exploring the relation between socio-economic status and reading achievement in PISA 2009 through an intercepts-and-slopes-as-outcomes paradigm

In some countries more than others, factors like parental socio-economic status (SES) can cause inequalities in educational achievement. Here we show how the mechanisms leading to such inequalities can be scrutinized by involving background variables which impact the relation between SES and achievement. We use the intercepts-and-slopes-as-outcomes paradigm which recognizes that the outcomes of schooling systems are not only characterized by average achievement (the intercept) but also by the achievement–SES regression slope. We show how certain background variables moderate the relationship between SES and achievement. As an illustration, we examine the relationship between reading achievement and SES, and how this is moderated by school funding and curriculum. This is done for several countries that participated in the PISA 2009 cycle

The Impact of Education Policies on Socioeconomic Inequality in Student Achievement: A Review of Comparative Studies

This chapter reviews international comparative studies on the determinants of socioeconomic inequality in student performance. We were interested in studies of explanatory variables that are amenable to educational policy interventions. To identify such publications, we developed a comprehensive search strategy and conducted an electronic search based on six databases. We also manually searched two existing hand-picked reviews. After duplicates were removed, the search resulted in 814 references, of which a total of 35 studies met the eligibility criteria. The included studies investigated diverse topics such as learning environments inside and outside of school, educational expenditure, teacher education, autonomy, accountability, differentiation, and competition from private schools. Most studies are descriptive in nature and their findings are sometimes ambiguous. Despite these limitations, we tentatively conclude that the opportunity of choice reinforces inequality. Measures that target social selection can be effective