A Brief Note on the Standard Error of the Pearson Correlation

The product-moment correlation is a central statistic in psychological research including meta-analysis. Unfortunately, it has a rather complex sampling distribution which leads to sample correlations that are biased indicators of the respective population correlations. Moreover, there seems to be some uncertainty on how to properly calculate the standard error of these correlations. Because no simple analytical solution exists, several approximations have been previously introduced. This note aims to briefly summarize 10 different ways to calculate the standard error of the Pearson correlation. Moreover, a simulation study on the accuracy of these estimators compared their relative percentage biases for different population correlations and sample sizes. The results showed that all estimators were largely unbiased for sample sizes of at least 40. For smaller samples, a simple approximation by Bonett (2008) led to the least biased results. Based on these results, it is recommended to use the expression (1-r²) / √N-3 for the calculation of the standard error of the Pearson correlation

The Web-Based Assessment of Mental Speed : An Experimental Study of Testing Mode Effects for the Trail-Making Test

Although web-based cognitive assessments have gained increasing attention in recent decades, it is still debated whether unstandardized test settings allow for comparable measurements as compared to proctored testing, particularly for speeded cognitive tests. Therefore, two within-subject experiments (N = 73 and N = 72) compared differences in means, criterion correlations with measures of intelligence, and subjective test quality perceptions of a trail-making test between a proctored paper-based, a proctored computerized, and an unproctored web-based administration mode. The results in both samples showed equivalent means between the two computerized modes, equivalent criterion correlations between the three modes, and no differential item functioning. However, the web-based tests were rated as having an inferior measurement quality as compared to the proctored assessments. Thus, web-based testing allows for comparable measurements of mental speed as compared to traditional computerized tests, although it is still regarded as a lower quality medium by test takers

The Longitudinal Measurement of Reasoning Abilities in Students With Special Educational Needs

Students with special educational needs in the area of learning (SEN-L) have learning disabilities that can lead to academic difficulties in regular schools. In Germany, these students are frequently enrolled in special schools providing specific training and support for these students. Because of their cognitive difficulties, it is unclear whether standard achievement tests that are typically administered in educational large-scale assessments (LSA) are suitable of students with SEN-L. The present study evaluated the psychometric properties of a short instrument for the assessment of reasoning abilities that was administered as part of a longitudinal LSA to German students from special schools (N = 324) and basic secondary schools (N = 338) twice within 6 years. Item response modeling demonstrated an essentially unidimensional scale for both school types. Few items exhibited systematic differential item functioning (DIF) between students with and without SEN-L, allowing for valid cross-group comparisons. However, change analyses across the two time points needed to account for longitudinal DIF among students with SEN-L. Overall, the cognitive test allowed for a valid measurement of reasoning abilities in students with SEN-L and comparative analyses regarding students without SEN-L. These results demonstrate the feasibility of incorporating students with SEN-L into educational LSAs

Bidirektionale Effekte zwischen Lese- und Mathematikentwicklung in der Sekundarstufe

Reading and mathematical competencies are important cognitive prerequisites for children’s educational achievement and later success in society. An ongoing debate pertains to potential transfer effects between both domains and whether reading and mathematics influence each other over time. Therefore, the present study on N = 5185 students from the German National Educational Panel Study examined cross-lagged effects between reading and mathematics from Grades 5 to 12. The results revealed, depending on the chosen causal estimand, negligible to small bidirectional effects. Adopting a between-person perspective, students with higher mathematics scores at one point exhibited somewhat higher reading scores at the subsequent measurement. In contrast, when adopting a within-person perspective, both skills predicted longitudinal increases of the other skill in the lower grades but reversed effects in higher grades. Taken together, these findings not only demonstrate that transfer effects between reading and mathematics in secondary education tend to be small but also suggest different patterns of effects depending on the modeling choice

Remote Testing of Reading Comprehension in 8-Year-Old Children : Mode and Setting Effects

Proctored remote testing of cognitive abilities in the private homes of test-takers is becoming an increasingly popular alternative to standard psychological assessments in test centers or classrooms. Because these tests are administered under less standardized conditions, differences in computer devices or situational contexts might contribute to measurement biases that impede fair comparisons between test-takers. Because it is unclear whether cognitive remote testing might be a feasible assessment approach for young children, the present study (N = 1,590) evaluated a test of reading comprehension administered to children at the age of 8 years. To disentangle mode from setting effects, the children finished the test either in the classroom on paper or computer or remotely on tablets or laptops. Analyses of differential response functioning found notable differences between assessment conditions for selected items. However, biases in test scores were largely negligible. Only for children with below-average reading comprehension small setting effects between on-site and remote testing were observed. Moreover, response effort was higher in the three computerized test versions, among which, reading on tablets most strongly resembled the paper condition. Overall, these results suggest that, on average, even for young children remote testing introduces little measurement bias

Accuracy and precision of fixed and random effects in meta‐analyses of randomized control trials for continuous outcomes

Meta-analyses of treatment effects in randomized control trials are often faced with the problem of missing information required to calculate effect sizes and their sampling variances. Particularly, correlations between pre- and posttest scores are frequently not available. As an ad-hoc solution, researchers impute a constant value for the missing correlation. As an alternative, we propose adopting a multivariate meta-regression approach that models independent group effect sizes and accounts for the dependency structure using robust variance estimation or three-level modeling. A comprehensive simulation study mimicking realistic conditions of meta-analyses in clinical and educational psychology suggested that imputing a fixed correlation 0.8 or adopting a multivariate meta-regression with robust variance estimation work well for estimating the pooled effect but lead to slightly distorted between-study heterogeneity estimates. In contrast, three-level meta-regressions resulted in largely unbiased fixed effects but more inconsistent prediction intervals. Based on these results recommendations for meta-analytic practice and future meta-analytic developments are provided

The longitudinal measurement of reasoning abilities in students with special educational needs

Students with special educational needs in the area of learning (SEN-L) have learning disabilities that can lead to academic difficulties in regular schools. In Germany, these students are frequently enrolled in special schools providing specific training and support for these students. Because of their cognitive difficulties, it is unclear whether standard achievement tests that are typically administered in educational large-scale assessments (LSA) are suitable of students with SEN-L. The present study evaluated the psychometric properties of a short instrument for the assessment of reasoning abilities that was administered as part of a longitudinal LSA to German students from special schools (N = 324) and basic secondary schools (N = 338) twice within 6 years. Item response modeling demonstrated an essentially unidimensional scale for both school types. Few items exhibited systematic differential item functioning (DIF) between students with and without SEN-L, allowing for valid cross-group comparisons. However, change analyses across the two time points needed to account for longitudinal DIF among students with SEN-L. Overall, the cognitive test allowed for a valid measurement of reasoning abilities in students with SEN-L and comparative analyses regarding students without SEN-L. These results demonstrate the feasibility of incorporating students with SEN-L into educational LSAs. (DIPF/Orig.

Out-of-Level Cognitive Testing of Children with Special Educational Needs

Children with special educational needs in the area of learning (SEN-L) have severe learning disabilities and often exhibit substantial cognitive impairments. Therefore, standard assessment instruments of basic cognitive abilities designed for regular school children are frequently too complex for them and, thus, unable to provide reliable proficiency estimates. The present study evaluated whether out-of-level testing with the German version of the Cognitive Abilities Test using test versions developed for younger age groups might suit the needs of these children. Therefore, N = 511 children with SEN-L and N = 573 low achieving children without SEN-L attending fifth grades in Germany were administered four tests measuring reasoning and verbal comprehension that were designed for fourth graders. The results showed that children with SEN-L exhibited significantly more missing responses than children without SEN-L. Moreover, three of the four tests were still too difficult for them. Importantly, no substantial differential response functioning was found for children with and without SEN-L. Thus, out-of-level testing might represent a feasible strategy to assess basic cognitive functioning in children with SEN-L. However, comparative interpretations would require additional norms or linked test versions that place results from out-of-level tests on a common metric

The impact of gender-stereotypical text contents on reading competence in women and men

Societies have socially shared assumptions about what constitutes typically male or female attributes. Language can contribute to gender inequality by transmitting gender stereotypes. This study examines whether gender-stereotypical connotations in stimulus texts within a reading competence test might serve as a nuisance factor distorting reading competence measurements. In addition to a general factor for reading competence, we expected gender-stereotypical texts to give rise to gender-specific factors regarding the text content. The research was based on a sample of 813 adults from a pilot study of the German National Educational Panel Study (NEPS). A bifactor model confirmed a general factor for reading competence. However, the two gender-specific factors were not found; consequently, no substantial gender differences in reading competence for gender-stereotypical text content were observed. These findings indicate that there is no substantial impact of gender-stereotypical text connotations on the measurement of women’s and men’s reading competence.Es gibt sozial geteilte Annahmen darüber, was typisch weibliche oder männliche Eigenschaften sind oder sein sollten. Über Sprache werden Stereotype vermittelt, beispielsweise über geschlechtsstereotype Textinhalte, wodurch Ungleichheiten zwischen Frauen und Männern aufrechterhalten werden. Die vorliegende Studie untersucht, ob geschlechtsstereotype Konnotationen von Texten, die zur Messung der Lesekompetenz verwendet werden, einen differenziellen Einfluss für Frauen und Männer auf das Ergebnis der Messung haben. Neben einem generellen Faktor der Lesekompetenz wurde erwartet, dass sich geschlechterspezifische Faktoren identifizieren lassen. Die Stichprobe umfasste 813 Erwachsene aus einer Entwicklungsstudie des Nationalen Bildungspanels (NEPS). Im Bi-Faktormodell konnte der allgemeine Faktor der Lesekompetenz bestätigt werden. Darüber hinaus konnten die beiden geschlechtsspezifischen Faktoren jedoch nicht ermittelt werden. Es zeigten sich keine Unterschiede in der Lesekompetenz hinsichtlich geschlechtsstereotyper Textinhalte. Das Ergebnis deutet darauf hin, dass eine geschlechtsstereotype Textkonnotation keinen substantiellen Einfluss auf die Lesekompetenzmessung von Frauen und Männern hat

Meta-Analytic Structural Equation Modeling With Fallible Measurements

Meta-analytic structural equation modeling (MASEM) combines the strengths of meta-analysis with the flexibility of path models to address multivariate research questions using summary statistics. Because many research questions refer to latent constructs, measurement error can distort effect estimates in MASEMs if the unreliability of study variables is not properly acknowledged. Therefore, a comprehensive Monte Carlo simulation evaluated the impact of measurement error on MASEM results for different mediation models. These analyses showed that point estimates in MASEM were distorted by up to a third of the true effect, while confidence intervals exhibited undercoverage that were less than 10% in some situations. However, the use of adjustments for attenuation facilitated recovering largely undistorted point and interval estimates in MASEMs. These findings emphasize that MASEMs with fallible measurements can often yield highly distorted results. We encourage applied researchers to regularly adopt adjustment methods that account for attenuation in MASEMs