Data from the International Adult Literacy Survey are used to illustrate how league tables can be obtained from summary data, consisting of percentages and their standard errors, using random effects models estimated by nonparametric maximum likelihood

Eilers, Paul

Einbeck, Jochen

Hoad, Dominique

Sofroniou, Nick

English

Durham Research Online

League tables for literacy survey data basedon random effect modelsNick Sofroniou1, Dominique Hoad2 and Jochen Einbeck21 Educational Research Centre, St Patrick’s College, Dublin, Ireland,nick sofroniou@yahoo.co.uk2 Durham University, Department of Mathematical Sciences, Durham, UKAbstract: Data from the International Adult Literacy Survey are used to il-lustrate how league tables can be obtained from summary data, consisting ofpercentages and their standard errors, using random effects models estimated bynonparametric maximum likelihood.Keywords: Random effect models; Mixture models, Nonparametric maximumlikelihood; Effective sample sizes.1 IntroductionIn the years 1994–95 twelve countries participated in the InternationalAdult Literacy Survey (IALS). Literacy is defined as using printed and writ-ten information to function in society, to achieve one’s goals and to developone’s knowledge and potential. The IALS developed a rigorous framework,building on work done in the 1985 Young Adult Literacy Survey (YALS)which consisted of three scales: prose, document and quantitative. It wasfelt that these three scales were the most significant for measuring literacyand sufficiently practical, with speaking and listening being too costly tomeasure. We concentrate on the measurement of prose in this paper. Thedata were reported by giving the percentage of individuals achieving proselevel 1, 2, . . . , 5, with level 1 being worst. One way of analyzing these datais to dichotomize the data around the lowest cutpoint (i.e., the thresholdbetween level 1 and level 2) to give percentages of adults in each countrywho could/could not reach a basic level of literacy. This is of particular in-terest to educationalists and policy makers concerned with social inclusionand its educational and economic implications. For the prose measure, thedata can then be summarized in the form of Table 1.2 Methodology for league table construction2.1 Effective sample sizesThe IALS used complex sample designs that varied with each country andwhich involved both stratification by factors such as region or school size,and clustering of pupils within schools. This complicates the issue of the2 League tablesTABLE 1. Percentage of adults not reaching at least Level 2 for 12 countries(Switzerland was split into two parts according to language and is treated asif it were two separate countries. Canada was treated as English-speaking andBelgium (Flanders) as Dutch). SE denotes the standard error of this percentageand n the sample size.Male FemaleCountry n % Level 1 SE n % Level 1 SESweden 1289 7.31 0.80 1355 7.18 1.03Netherlands 1358 10.39 1.08 1479 10.49 0.98Germany 938 14.31 1.89 1124 13.31 1.85Australia 3767 18.33 0.85 4437 15.69 0.78Canada 1979 18.76 2.03 2521 14.44 2.04New Zealand 1821 19.94 1.28 2402 16.52 1.46Belgium (Flanders) 1066 15.55 1.69 1180 21.61 2.29Switzerland (French) 682 17.46 1.88 751 19.44 1.70Switzerland (German) 659 18.30 1.51 733 20.66 1.66United Kingdom 1730 21.38 1.26 2081 21.60 1.82Ireland 1050 24.21 2.91 1319 20.93 1.32United States 1416 23.00 1.65 1577 18.76 1.45Poland 1431 43.72 0.91 1569 41.74 1.74denominator to be used in mixed binomial models as the effective samplesizes will tend be considerably less than the actual number of students ineach country, due to the intra-cluster correlations.Cochran (1977) states that under simple random sampling the sample pro-portion p = a/n is an unbiased estimate of the population proportionP = A/N and that an unbiased estimate of the variance of p obtainedfrom the sample isv(p) = s2p =N − n(n − 1)Npqwhich simplifies even further assuming large N, and hence a negligible finitepopulation correction, tov(p) =pqn − 1.By rearranging these expressions one can obtain the corresponding samplesize n under simple random sampling, e.g.,n =N(pq + v(p))pq + Nv(p)for the former expression. Thus, it is possible to use the summary infor-mation in Table 1, consisting of percentages and their standard errors, tocalculate an effective sample size corresponding to the number of inde-pendent observations in a theoretical simple random sample. This allowsthe use of standard mixed binomial modelling software with the effectiveN. Sofroniou et al. 3sample size as the binomial denominator, reflecting the uncertainty in thepercentages of the original table.2.2 Random effect modelsProbabilities are commonly either modelled through a binomial logit or aPoisson log model, with the latter one being less adequate in this exampleas we have relatively large probabilities and small sample sizes involved.The variability of the upper-level units, here countries i, can be taken intoaccount by adding a random intercept zi with unspecified distribution g(·)to the linear predictor, so that the the binomial random effect model takesthe formlogpij1 − pij= β′xij + γ′si + zi (1)where xij contains the lower-level covariates (here only gender), si containsthe upper-level covariates (here only the factor for language is considered),γ is a non-random parameter and β is a fixed or random parameter. Suchvariance component models with unspecified random effect distributioncan be conveniently fitted using the method of nonparametric maximumlikelihood (Aitkin, Hinde & Francis, 2005, p. 440ff), which is implementedin the R package npmlreg (Einbeck, Hinde & Darnell, 2007). In short,the density g(·) is approximated by a discrete distribution with K masspoints, the locations zk and masses πk, k = 1, . . . , K, of which are estimatedthrough the EM algorithm. Thereby, the E-step corresponds to updatingof the probabilities wik = P (unit i comes from mass point k), and the M-step to a weighted generalized linear model fit with weights wik. Fromthe set of weights after the final EM iteration, one computes posteriorintercepts zi =∑k wikzk which represent the cluster-level contribution tothe response, adjusted by the covariates. As this posterior intercept “sticks”to the cluster for all its lower-level units, it forms a characteristic of thecluster (country). Sofroniou, Einbeck & Hinde (2006) used the posteriorintercept for the construction of league tables in the absence of upper-levelcovariates.3 Results and conclusions for the literacy surveyWe considered several additive logistic random effect models of type (1).To keep the model parsimonious (with only 26 observations available), themodels considered exclude a language.gender interaction term and ran-dom coefficients. Fitting gender as a covariate and no language factor re-quires 5 masspoints for the random intercept distribution and has a dispar-ity of −2 logL = 229.0 with df = 16. Table 2 gives posterior probabilitiesof the membership of each country to a given component. It suggests thatthere are two main groups of countries, two countries who performed con-siderably better (Sweden and the Netherlands), and one low scoring outlier4 League tablesTABLE 2. Posterior probabilities for the IALS data.Posteriorinterept MasspointsInterept  2.602  2.156  1.599  1.379  0.330Proportion 0.077 0.093 0.434 0.319 0.077Sweden  2.60 1.00Netherlands  2.16 1.00Germany  1.72 0.21 0.79Australia  1.60 1.00Canada  1.59 0.97 0.03New Zealand  1.58 0.92 0.08Belgium (Flanders)  1.58 0.89 0.11Switzerland (Frenh)  1.54 0.72 0.28Switzerland (German)  1.45 0.34 0.66Ireland  1.38 1.00United Kingdom  1.38 1.00United States  1.38 0.01 0.99Poland  0.33 1.00Posterior probabilities: p  0:95, 0:90  p < 0:95, p < 0:90.(Poland). Adding language dichotomized into English/non-English speak-ing required 5-masspoints and reduced the disparity to 223.3 with df = 15.This was further improved by using all 6 levels of language, with a dis-parity of 210.3, df = 15, and 3 masspoints. However, several categoriesare based on only a single country and so their performance levels be-come confounded with language spoken. Therefore, we experimented withadding the fitted upper-level contribution to the posterior intercept, yield-ing a similar league table to the one presented above, but further researchon issues such as representing the uncertainty corresponding to each valueis required. These last two models provide some evidence in favour of thesuggestion that one contribution to the observed differences in performancemay be that of the language of testing.ReferencesAitkin, M., Francis, B. and Hinde, J. (2005). Statistical Modelling in GLIM4 (2nd edn.). Oxford, UK.Cochran, W.G. (1977). Sampling Techniques (3rd edn.). Wiley, New York.Einbeck, J., Hinde, J. and Darnell, R. (2007). A new package for fittingrandom effect models – The npmlreg package. R News, 7, 26–30.Sofroniou, N., Einbeck, J. and Hinde, J. (2006). Analyzing Irish Suicide Rateswith Mixture Models. In Proceedings of the 21st International Work-shop on Statistical Modelling, Galway, Ireland, 474–481.

League tables for literacy survey data based on random effect models

https://durham-repository.worktribe.com/file/1157849/1/Accepted%20Conference%20Proceeding

League tables for literacy survey data based on random effect models

Abstract

Similar works

Full text

Available Versions

Durham Research Online