Computing Sampling Weights in Large-scale Assessments in Education

Sampling weights are a reflection of sampling design; they allow us to draw valid conclusions about population features from sample data. This paper explains the fundamentals of computing sampling weights for large-scale assessments in educational research. The relationship between the nature of complex samples and best practices in developing a set of weights to enable computation of unbiased population estimates is described. Effects of sampling weights on estimates are shown, as well as potential consequences of not using weights when analysing data from complex samples. Illustrative examples are provided in order to make it easy to understand the rationale behind the mathematical foundations

Bekämpfung der Kraut- und Knollenfäule im Ökokartoffelanbau

In dieser Arbeit wird ein zusammenfassender Überblick über Schadbilder und Infektionsketten sowie indirekte und direkte Gegenmaßnahmen gegen die Kraut- und Knollenfäule von Kartoffeln gegeben. Ein Schwerpunkt liegt in der Vermittlung, dass viele Faktoren beachtet werden müssen, um die Schadwirkung in Grenzen zu halten

Developments and methodological challenges in international large-scale assessments in education: an IEA perspective

Ongoing societal and technological developments in education and changes in the global debate about education continue to promote the value of international large-scale assessments (ILSAs) in education. ILSAs are expanding their sphere of influence, evolving to cover novel target populations and subject domains. Advances in the methods and technology available to collect, scale, and analyze data present continuous methodological challenges, but also foster rapid developments of the methodological research and respective technology. Most ILSAs in education are now enforcing a transition to computer-based assessment. Recent research has suggested new approaches for addressing nonresponse, novel methods to improve measurement invariance evaluation, and explored innovative methodologies for statistical data analyses. This paper reflects on IEA\u27s extensive experience of ILSA research to identify the most important contemporary challenges, contextualized by historical developments. The authors discuss these developments considering their potentials, drawing conclusions and giving recommendations on best practice. (DIPF/Orig.)Die Autorinnen und die Autoren dieses Beitrags formulieren aus der Perspektive der International Association for the Evaluation of Educational Achievement (IEA) aktuelle Entwicklungen und methodische Herausforderungen im Kontext von international vergleichenden Schulleistungsuntersuchungen. (DIPF/Orig.

Hierarchical Modeling with Large‐Scale Assessment Data: Influence of Intra-Class Correlation on Sampling Precision

Most data collected in educational large scale assessments (LSA) is very well suited for multilevel modeling because sampled individuals are usually nested within clusters (e.g., students nested within schools). Hierarchical models allow for the effect of explanatory variables at different clustering levels; parameters can be determined as fixed or random effects depending on the research question. All model parameters are however estimated based on random samples and are therefore subject to sampling error. A Monte Carlo simulation was utilized to explore the connections between the sample sizes at different levels and intra-class correlation coefficients (ICCs) in settings that mimic scenarios typical for LSA. It was observed that varying levels of ICC influence the margins of sampling variance of the estimated model parameters in different amounts and even different directions. Assuming fixed sample sizes, the coefficients of variation (CVs) of the model parameters mean of random intercepts (γ00) and slope of random intercepts (γ01) increased with increasing ICC levels, as expected. However, the inverted relationship was observed for parameters U0 – variance of random intercepts, γ10 – mean of random slopes, and β1 – fixed slope: with increasing ICC, the CVs decreased. The findings can help us to determine sample sizes in LSA when particular hierarchical models are to be investigated.status: publishe

Sample size requirements in HLM: An empirical study

http://www.ierinstitute.org/dissemination-area.htmlnrpages: 172status: publishe

Evaluating the risk of nonresponse bias in educational large-scale assessments with school nonresponse questionnaires: a theoretical study

Abstract Survey participation rates can have a direct impact on the validity of the data collected since nonresponse always holds the risk of bias. Therefore, the International Association for the Evaluation of Educational Achievement (IEA) has set very high standards for minimum survey participation rates. Nonresponse in IEA studies varies between studies and cycles. School participation is at a higher risk relative to within-school participation; school students are more likely to cooperate than adults (i.e., university students or school teachers). Across all studies conducted by the IEA during the last decade, between 7 and 33% of participating countries failed to meet the minimum participation rates at the school level. Quantifying the bias introduced by nonresponse is practically impossible with the currently implemented design. During the last decade social researchers have introduced and developed the concept of nonresponse questionnaires. These are shortened instruments applied to nonrespondents, and aim to capture information that correlates with both: survey’s main outcome variable(s), and respondent’s propensity of participation. We suggest in this paper a method to develop such questionnaires for nonresponding schools in IEA studies. By these means, we investigated school characteristics that are associated with students’ average achievement scores using correlational and multivariate regression analysis in three recent IEA studies. We developed regression models that explain with only 11 school questionnaire variables or less up to 77% of the variance of the school mean achievement score. On average across all countries, the R 2 of these models was 0.24 (PIRLS), 0.34 (TIMSS, grade 4) and 0.36 (TIMSS grade 8), using 6–11 variables. We suggest that data from such questionnaires can help to evaluate bias risks in an effective way. Further, we argue that for countries with low participation rates a change in the approach of computing nonresponse adjustment factors to a system were school´s participation propensity determines the nonresponse adjustment factor should be considered

The relationship between students’ use of ICT for social communication and their computer and information literacy

Abstract This study investigates the relationship between students’ use of information and communication technology (ICT) for social communication and their computer and information literacy (CIL) scores. It also examines whether gender and socioeconomic background moderates this relationship. We utilized student data from IEA’s International Computer and Information Study (ICILS) to build multivariate regression models for answering the research questions, and accounted for the complex sample structure of the data by using weights for all statistical analyses, employing jackknife repeated replication for variance estimation. Students who frequently use the internet for messaging and participation in social networks (i.e., at least once a week) scored on average 44 points higher than those who use ICT for the same purpose only less than once a week or never. The direction of this effect was the same in all 21 participating educational systems, the difference ranging from 19 to 75 points (always statistically significant). We continued the analysis by testing whether the relationship is moderated by gender; as girls use more often ICT for social communication and have higher CIL scores on average. After controlling for the gender effect the CIL scores between the two examined groups decreased only by 2 points on average. Even after including students’ socio-economic background into the model, the difference in CIL between the two groups of interest declined only little—to 32 points on average across all countries. The difference remained to be statistically significant in all countries but one. The results suggest a strong relationship between students’ CIL proficiency level and the frequency of their use of electronic devices for social communication; hence, respective skills needed at schools and later on at the workplace are reflected in their use outside of school and for socializing

Considerations for correlation analysis using clustered data: working with the teacher education and development study in mathematics (TEDS-M) and other international studies

En: Large-scale Assessments in Education, No. 1The Teacher Education and Development Study in Mathematics (TEDS-M) of 2008 focused on how teachers are prepared to teach mathematics in primary and lower-secondary schools in 17 countries. The main results were published in 2012, and the associated public-use database provides a valuable source for secondary analysis of the collected data. The data originate from complex samples and present a hierarchical structure. With future teachers embedded in programs embedded in institutions, various types of cluster effects can be observed. Complex methods, including the use of sampling weights and replication methods for variance estimation, are therefore required for data analysis. This paper focuses on the aspects that need to be considered during any exploration of relationships between variables. Correlation analysis may produce misleading results if attention is not paid to the structure under which the data were collected. We illustrate our points with exemplary analysis of TEDS-M data and propose some guidelines to address the issue

Evaluating German PISA stratification designs: a simulation study

Abstract Stratification is an important design feature of many studies using complex sampling designs and it is often used in large-scale assessment (LSA) studies, such as the Programme for International Student Assessment (PISA), for two main reasons. First, stratification variables that achieve a high between and low within strata variance can improve the efficiency of a survey design. Second, stratification allows one to, explicitly or implicitly, control for sample sizes across subpopulations. It ensures that some parts of a population are in the sample in predetermined proportions. In this study, we determine through simulation which stratification scheme is best for PISA in Germany. For this, we consider the constraints imposed by the international sampling design, the available information about schools, and specific national characteristics of the German educational system. We examine seven different stratification designs selected based on scenarios used in past LSAs in Germany and theoretical considerations for future implementations. The chosen scenarios were compared with two reference scenarios: (1) an unstratified design and (2) a synthetic optimal stratification design. The simulation study reveals that the stratification design currently applied in PISA produces satisfactory results regarding sampling precision. The present stratification design is based on Germany's federal states and school types. However, this approach leads to small strata, which has been problematic for estimating sampling variance in previous cycles. Therefore, alternative stratification scenarios were considered and, in addition to overcoming the small-strata problem, also led to smaller standard errors for estimates of student mean performance in mathematics, science, and reading. As a result of this study, we recommend considering three different stratification designs for Germany in future cycles of PISA. These recommendations aim to: (1) improve the sampling efficiency while keeping the sample size constant, (2) follow a sound methodological approach, and (3) make conservative and cautious changes while maintaining a reflection of the structure of the German federal school system with different school types. These suggestions include a reinvented stratification of grouped German federal states and designs with school types as explicit stratifiers and federal states as implicit stratifiers