Full versus incomplete cross-validation: measuring the impact of imperfect separation between training and test sets in prediction error estimation

In practical applications of supervised statistical learning the separation of the training and test data is often violated through performing one or several analysis steps prior to estimating the prediction error by cross-validation (CV) procedures. We refer to such practices as incomplete CV. For the special case of preliminary variable selection in high-dimensional microarray data the corresponding error estimate is well known to be strongly downwardly biased, resulting in over-optimistic conclusions regarding prediction accuracy of the fitted models. However, while other data preparation steps may also be affected by these types of problems, their impact on error estimation is far less acknowledged in the literature. In this paper we shed light on these issues. We present a new measure quantifying the impact of incomplete CV that is based on the ratio between the errors estimated by incomplete CV and by a formally correct "full CV." The new measure is illustrated through applications to several low- and high-dimensional biomedical data sets and various data preparation steps including preliminary variable selection, choice of tuning parameters, normalization of gene expression microarray data, and imputation of missing values. It may be used in biometrical applications to determine whether specific data preparation steps can be safely performed as preliminary steps before running the CV procedure, or if they should be repeatedly trained in each CV iteration

General and Specific Contributions of RAN to Reading and Arithmetic Fluency in First Graders: A Longitudinal Latent Variable Approach

In the present study, we opted for a longitudinal design and examined rapid automatized naming (RAN) performance from two perspectives. In a first step, we examined the structure of RAN performance from a general cognitive perspective. We investigated whether rapid naming measures (e.g., digit RAN and color RAN) reflect a mainly domain-general factor or domain-specific factors. In a second step, we examined how the best fitting RAN model was related to reading and arithmetic outcomes, assessed several months later. Finally in a third step we took a clinical perspective and investigated specific contributions of RAN measures to reading and arithmetic outcomes. While RAN has emerged as a promising predictor of reading, the relationship between RAN and arithmetic has been less examined in the past. Hundred and twenty-two first graders completed seven RAN tasks, each comprising visually familiar stimuli such as digits, vowels, consonants, dice, finger-numeral configurations, objects, and colors. Four months later the same children completed a range of reading and arithmetic tasks. From a general descriptive perspective, structural equation modeling supports a one-dimensional RAN factor in 6- to -7-year-old children. However, from a clinical perspective, our findings emphasize the specific contributions of RANs. Interestingly, alphanumeric RANs (i.e., vowel RAN) were most promising when predicting reading skills and number-specific RANs (i.e., finger-numeral configuration RAN) were most promising when predicting arithmetic fluency. The implications for clinical and educational practices will be discussed

Project NUMTEST: Assessing basic number competence without language

Although numerical skills are essential in modern societies, 5- 7 % of the population suffer from mathematical learning disabilities. Due to the hierarchical nature of mathematical knowledge, screening during the earliest stages of learning is essential to intervene efficiently. While different screening tools exist, they rely on verbal instructions and task content, hampering their usefulness in linguistically heterogeneous young school populations. We developed a computerized task that requires subjects to encode both auditory and visual numerical information to successfully respond to the task’s demands. For task instruction, participants were presented a video showing a person correctly solving three easy items of the task, before moving on to a practice session of three different items. If any item was solved incorrectly, the entire practice session was repeated for the participant. We administered the task to a sample of first grade students and collected participant’s performance in standardized addition, subtraction and number comparison (1 & 2-digit) tasks. A multivariate analysis with practice repetition as between-subject factor on the four standardized control measures revealed that participants that repeated the practice session scored significantly lower in three out of four dependent measures. These results suggest that the immediate understanding (i.e. practice items solved correctly on the first try) of our task, requiring deductive reasoning and an abstract, format-independent representation of quantity, is able to differentiate between high and low performers on standardized measures of basic math competence non-verbally at an early stage of learning. Implications and limitations will be discussed

Project NUMTEST: Assessing basic number competence without language

While numerical skills are fundamental in modern societies, some estimated 5-7% of children suffer from a mathematical learning disorder, called developmental dyscalculia (DD). Nevertheless, universally valid diagnostic instruments are still lacking, as all current DD test batteries are based on language instructions. Consequently, their measurements are tightly linked to the specific language context of test administration and thus their results cannot easily be compared across countries. Here we are showing results of the first pilot study of a research project that aims to develop a test for basic math abilities that does not rely on language instruction and minimizes language use. To this aim, video and animation based instructions were implemented on touchscreen devices. A first version of the tasks has been tested with two samples of first grade children in Luxembourg’s fundamental schools, of which half completed the same tasks with traditional verbal instructions. Our results indicate that performance in the experimental group was similar or better than the control group using verbal instructions. Relationships between linguistic background and the sample’s performance on one hand and qualitative usability aspects of nonverbal task instruction and tablet-pc use with young children will be discussed

Predicting first-grade mathematics achievement: the contributions of domain-general cognitive abilities, nonverbal number sense, and early number competence

Early number competence, grounded in number-specific and domain-general abilities, is supposed to lay the foundation for later math achievement. Few longitudinal studies tested a comprehensive model for early math development. Using structural equation modeling and mediation analyses, we studied the influence of kindergarteners’ basic cognitive abilities (i.e., nonverbal number sense, working memory, fluid intelligence, and receptive vocabulary) and their early number competence (i.e., symbolic number skills) on first grade math achievement (arithmetic, shape and space, and number line estimation) assessed one year later. Latent regression models revealed that nonverbal number sense and working memory are central building blocks for developing early number competence in kindergarten and that early number competence is key for first grade math achievement. Fluid intelligence significantly predicted arithmetic and number line estimation while receptive vocabulary significantly predicted shape and space after controlling for early number competence. In sum we suggest that early math achievement draws on different constellations of number-specific and domain-general mechanisms

Can language skills in Luxembourgish be a stepping stone to German? Evidence against the transfer assumption

peer reviewedIn Luxembourg, children usually learn to write and read in German, a language, which most of them do not speak at home or learn in early childhood education settings. Many had assumed that the promotion of Luxembourgish would be enough to prepare (Kühn, 2008) the children for the German language; that Luxembourgish language skills could be transferred into German language skills. We examine the validity of this assumption by comparing the listening comprehension of first grade students in German and Luxembourgish between different home language groups. For this, we used quantitative data from the 2022 Épreuves Standardisées, the national monitoring programme (Hornung et al., 2023). While for children who only speak Luxembourgish at home, the German test seemed easier, bilingual children or children with other different home languages struggled more with German than with Luxembourgish listening comprehension – indicating that a (direct) transfer from one to the other language does not seem plausible for all children.4. Quality educatio

Early Childhood Education and Care in Luxembourg - Is attendance influenced by immigration background and socioeconomic status?

For decades, researchers have been raising awareness of the issue of educational inequalities in the multilingual Luxemburgish school system. Especially children from families with a migration background or a lower socio-economic status show large deficits in their language and mathematics competences in comparison to their peers. The same applies to children who do not speak Luxemburgish or German as their first language (Hornung et al., 2021; Sonnleitner et al., 2021). One way to reduce such educational inequalities might be an early and extensive participation in early childhood education and care (ECEC). Indeed, participation in ECEC was found to be positively connected to language and cognitive development in other countries, especially for children from disadvantaged families (Bennett, 2012). However, these children attend ECEC less often (Vandenbroeck & Lazzari, 2014). There are indications that lower parental costs might go hand in hand with a greater attendance of ECEC in general (for a Luxembourgish study, see Bousselin, 2019) and in particular by disadvantaged families (Busse & Gathmann, 2020). The aim of this study is to spotlight the attendance of ECEC in Luxembourg during the implementation of the ECEC reform after 2017 which increased free ECEC hours for all families from 3 to 20 hours a week. We draw on a large dataset of about 35.000 children from the Épreuves Standardisées (ÉpStan, the Luxemburg school monitoring programme) from 2015 to 2021 and investigate which children attend any kind of regulated ECEC service (public, private or family daycare) in which intensity, taking socio-economic and cultural family factors into account. The findings might help to understand in which contexts ECEC attendance should be further encouraged. Implications for future policy decisions are discussed with the goal of further promoting equal educational opportunities for all children

Wissenschaftliche Politikberatung in Krisenzeiten in der Schweiz: Eine Analyse der Finanzkrise, des Fukushima-Unfalls und der Covid-19-Pandemie. Study commissioned by the Swiss Science Council

Diese Studie wurde zwischen September 2021 und Juni 2022 von den Universitäten Bern, St. Gallen und Lausanne im Auftrag des Schweizerischen Wissenschaftsrats (SWR) durchgeführt. Die Studie befasst sich mit der Frage der wissenschaftlichen Politikberatung in der Schweizer Bundesverwaltung und Politik in Krisensituationen. Sie beantwortet anhand einer vergleichenden Analyse der wissenschaftlichen Politikberatungssysteme in drei vergangenen Krisen − der Finanzkrise 2008−2009, dem Atomunfall in Fukushima 2011 und der Covid-19-Pandemie 2020−2022 − folgende übergeordnete Fragestellung (Kapitel 1): Welche Organisationsformen und Systeme der wissenschaftlichen Politikberatung wurden in vergangenen Krisen eingesetzt und wie haben sich diese bewährt? Die drei Krisen wurden aufgrund ihrer unterschiedlichen Merkmale sowie der unterschiedlichen wissenschaftlichen Politikberatungssysteme, die jeweils zum Einsatz kamen, ausgewählt. Diese Unterschiede ermöglichen es, verschiedene Science-for-Policy-Mechanismen zu erforschen. Die Studie wurde von einem interdisziplinären Team durchgeführt, welches einen historischen und politikwissenschaftlichen Ansatz kombinierte. Das Mandat ist Teil der verschiedenen Initiativen, die im Rahmen der Aufarbeitung der Pandemiebewältigung darauf abzielen, Lehren für die Zukunft zu ziehen

Taking Language out of the Equation: The Assessment of Basic Math Competence Without Language

While numerical skills are fundamental in modern societies, some estimated 5–7% of children suffer from mathematical learning difficulties (MLD) that need to be assessed early to ensure successful remediation. Universally employable diagnostic tools are yet lacking, as current test batteries for basic mathematics assessment are based on verbal instructions. However, prior research has shown that performance in mathematics assessment is often dependent on the testee's proficiency in the language of instruction which might lead to unfair bias in test scores. Furthermore, language-dependent assessment tools produce results that are not easily comparable across countries. Here we present results of a study that aims to develop tasks allowing to test for basic math competence without relying on verbal instructions or task content. We implemented video and animation-based task instructions on touchscreen devices that require no verbal explanation. We administered these experimental tasks to two samples of children attending the first grade of primary school. One group completed the tasks with verbal instructions while another group received video instructions showing a person successfully completing the task. We assessed task comprehension and usability aspects both directly and indirectly. Our results suggest that the non-verbal instructions were generally well understood as the absence of explicit verbal instructions did not influence task performance. Thus we found that it is possible to assess basic math competence without verbal instructions. It also appeared that in some cases a single word in a verbal instruction can lead to the failure of a task that is successfully completed with non-verbal instruction. However, special care must be taken during task design because on rare occasions non-verbal video instructions fail to convey task instructions as clearly as spoken language and thus the latter do not provide a panacea to non-verbal assessment. Nevertheless, our findings provide an encouraging proof of concept for the further development of non-verbal assessment tools for basic math competence