Evaluating the risk of nonresponse bias in educational large-scale assessments with school nonresponse questionnaires: a theoretical study

Abstract Survey participation rates can have a direct impact on the validity of the data collected since nonresponse always holds the risk of bias. Therefore, the International Association for the Evaluation of Educational Achievement (IEA) has set very high standards for minimum survey participation rates. Nonresponse in IEA studies varies between studies and cycles. School participation is at a higher risk relative to within-school participation; school students are more likely to cooperate than adults (i.e., university students or school teachers). Across all studies conducted by the IEA during the last decade, between 7 and 33% of participating countries failed to meet the minimum participation rates at the school level. Quantifying the bias introduced by nonresponse is practically impossible with the currently implemented design. During the last decade social researchers have introduced and developed the concept of nonresponse questionnaires. These are shortened instruments applied to nonrespondents, and aim to capture information that correlates with both: survey’s main outcome variable(s), and respondent’s propensity of participation. We suggest in this paper a method to develop such questionnaires for nonresponding schools in IEA studies. By these means, we investigated school characteristics that are associated with students’ average achievement scores using correlational and multivariate regression analysis in three recent IEA studies. We developed regression models that explain with only 11 school questionnaire variables or less up to 77% of the variance of the school mean achievement score. On average across all countries, the R 2 of these models was 0.24 (PIRLS), 0.34 (TIMSS, grade 4) and 0.36 (TIMSS grade 8), using 6–11 variables. We suggest that data from such questionnaires can help to evaluate bias risks in an effective way. Further, we argue that for countries with low participation rates a change in the approach of computing nonresponse adjustment factors to a system were school´s participation propensity determines the nonresponse adjustment factor should be considered

Implementing ICT in classroom practice: what else matters besides the ICT infrastructure?

Abstract Background The large-scale International Computer and Information Literacy Study (2018) has an interesting finding concerning Luxembourg teachers. Luxembourg has one of the highest reported level of technology-related resources for teaching and learning, but a relatively lower reported use of ICT in classroom practice. Methods ICT innovation requires a high initial level of financial investment in technology, and Luxembourg has achieved this since 2015. Once the necessary financial investment in ICT technology has been made, the key question is what else matters to increase the use of ICT in teaching. To identify the relevant factors, we used the “Four in Balance” model, aimed explicitly at monitoring the implementation of ICT in schools. Results Using data for 420 teachers in Luxembourg, we identify that within such a technology-driven approach to digitalization, teachers’ vision of ICT use in teaching, level of expertise, and the use of digital learning materials in class are significant support factors. Leadership and collaboration, in the form of an explicit vision of setting ICT as a priority for teaching in the school, also prove to be important. Conclusions Through these findings, we show that the initial investment in school infrastructure for ICT needs to be associated in its implementation with teachers’ ICT-related beliefs, attitudes, and ICT expertise