Essays on the psychometric and statistical properties of the Programme for International Student Assessment

The Organisation for Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) has become one of the key studies for evidence-based education policymaking across the globe. Yet its psychometric and statistical properties – and therefore its limitations and implications – are often poorly understood by researchers and stakeholders. This thesis aims to shed new light on the impact that psychometric and statistical issues can have on key statistics and cross-country comparisons. In the first study, I investigate the ‘conditioning model’, where background variables are used in the derivation of student achievement scores. Thereby, I systematically vary the background variables used within the conditioning model and analyse the impact upon country rankings. My key finding is that the exact specification of the conditioning model matters; cross-country comparisons of PISA scores can change substantially depending upon the statistical methodology used. The second study is an extension of the first, as I conduct a simulation study to further investigate research questions and additionally examine other key aspects influencing the achievement score computation, more specifically the preparation of variables and the test and background questionnaire design. This study confirms that the specification of the conditioning model matters, though bias in the results can be reduced by asking students questions in all subjects. In contrast, variable preparation and the background questionnaire design have negligible impact. The third study aims to provide a broader and more comprehensive review of different psychometric and statistical properties in PISA. By using a case study, six different properties that can potentially affect the validity of parental education group comparisons in Germany are evaluated. This study highlights how diverse sources of bias can be in PISA, and that they indeed impact the results. While I did not find that all the investigated issues introduce bias, several properties substantially impacted the results

The internal validity of the school-level comparative interrupted time series design: evidence from four new within-study comparisons

Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias

The Pennsylvanian of the Ruhr Basin, Germany : peat deposition and characterisation of dispersed kerogen, vitrinite, and coal at different levels of maturation

This thesis deals with the depositional conditions during peat formation of Pennsylanian coals from the paralic German Ruhr Basin as well as the chemical and structural changes occurring in these coals, associated kerogen from clastic rocks, and vitrinite upon thermal maturation. For this purpose vitrinites of a natural maturity sequence of Pennsylvanian coals from the Ruhr Basin, consisting of eleven coal seams from subbituminous to semi-anthracite ranks (0.5 to 2.9% vitrinite reflectance), were handpicked, characterised concerning their elemental composition and analysed using Rock-Eval pyrolysis, attenuated total reflectance infrared spectroscopy (ATR FT-IR), and Curie Point-pyrolysis gas chromatography-mass spectroscopy (CP-Py-GC-MS) at two pyrolysis temperatures. Micro-FT-IR was used in order to assess variations in the proportions of functional groups in megaspores from five oil mature coal samples. Infrared spectra of vitrinites show a clear increase in aromaticity with increasing thermal maturity. Spectra of megaspores reveal a more aliphatic character than the vitrinites and the degree of aromaticity increases more slowly compared to that of the vitrinites with increasing maturity. Both types of macerals show a pronounced decrease in carboxyl/carbonyl functional groups. Vitrinites pyrolysed at 590 °C yield high amounts of aliphatic hydrocarbons, while those pyrolysed at 764 °C produce more aromatic compounds, including phenols, naphthalenes and phenanthrenes as well as sulphur-containing aromatics. The aromatic fraction relative to the aliphatic fraction increases upon maturation of the pyrolysates. The loss of phenolic moieties in favour of benzenes upon maturation evidences the loss of oxygen-containing functional groups and is in line with the findings obtained from the FT-IR measurements. Increasing yields of polyaromatic structures in the pyrolysed vitrinites show the evolution towards more condensed aromatic clusters as maturation proceeds. These general trends are also found for vitrinites of a narrower maturity window ranging from subbituminous to high volatile bituminous coals, as revealed by the analysis of coals and associated siliciclastic rocks. These coals of Bolsovian age were also characterised regarding the environmental conditions prevailing during the deposition of the original peat mires. Eleven coal seams from the German Ruhr Coal Basin were analysed along depth profiles, on the one hand microscopically and on the other hand in terms of their ash yield, carbon and sulphur content as well as their Rock-Eval parameters. Established maceral indices and the newly introduced A/I ratio were employed to evaluate the depo-settings. Results show that the same basic depositional pathways of terrestrialisation and paludification exist as reported before for older (more mature) Duckmantian coal seams from the Ruhr Basin. The presence of coals formed as ombrotrophic peat further indicates a wet climate until the Upper Bolsovian. Very high sulphur values of seam Parsifal II indicate a marine influence at the peat stage. A marine ingression event was not reported before for this stratigraphical level within the Ruhr Basin. The influence of marine water on seam Parsifal II is recognisable on a molecular level as revealed by the results obtained by ATR FT-IR and CP-Py-GC-MS analyses on the Bolsovian samples. The same analytical methods as for the Pennsylvanian maturity series were applied in order to detect the influence on depositional environment and precursor maceral on the structural properties of coals, kerogen from adjacent clastic rocks, and pure vitrinite, while the maturity window was narrow (0.55 to 0.73% VRr). Higher Tmax values of siliciclastic rocks associated with the Bolsovian coals are explained by the mineral matrix effect, while the relatively higher oxygen index values of the siliciclastics and kerogen isolated from them are surprisingly not related to the abundance of carbonyl/carboxyl groups deduced from ATR FT-IR spectroscopy. FT-IR results imply a lower degree of aromaticity and at the same time higher degree of condensation for vitrinites originated from the sedimentary layers, as compared to vitrinites hand-picked from the coal seams. Results from CP-Py-GC-MS at 590 °C show that the vitrinites originating from the siliciclastic rocks generate higher amounts of long-chained n-alkanes and -alkenes than the bulk samples or the vitrinites originating from the coals. In the narrow maturity range represented by the Bolsovian samples, only vitrinites of the coal seams show some correlation with depth. As for the natural maturity sequence of the Ruhr Basin, the relative abundance of phenols is decreasing with increasing maturity, as well as the ratio of benzenes over higher condensed aromatic moieties. The loss of oxygen-containing functional groups seems to be the most pronounced from the subbituminous to the high volatile bituminous coal ranks