Educational Research with Real-World Data: Reducing Selection Bias with Propensity Score Analysis

Often it is infeasible or unethical to use random assignment in educational settings to study important constructs and questions. Hence, educational research often uses observational data, such as large-scale secondary data sets and state and school district data, and quasi-experimental designs. One method of reducing selection bias in estimations of treatment effects is propensity score analysis. This method reduces a large number of pretreatment covariates to a single scalar function and allows researchers to compare subjects with similar probability to receive the treatment. This article provides an introduction to propensity score analysis and stratification, an example illustrating its use, and suggestions for using propensity score analysis in educational research. Accessed 6,397 times on https://pareonline.net from December 18, 2013 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right

Exploring the Limitations of Measures of Students` Socioeconomic Status (SES)

This study uses a nationally representative student dataset to explore the limitations of commonly used measures of socioeconomic status (SES). Among the identified limitations are patterns of missing data that conflate the traditional conceptualization of SES with differences in family structure that have emerged in recent years and a lack of theoretically-based guidance for how the components of SES should be combined. Using kindergarten achievement data, the study illustrates how both the observed relation between SES and achievement and the observed interaction between SES and kindergarten program would be impacted by the use of different measures of SES. This study also explores the measurement of SES within a structural equation modeling (SEM) framework, highlighting both the relevant conceptual and measurement issues. Accessed 9,241 times on https://pareonline.net from May 15, 2014 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right

A randomized trial to evaluate the course of effects of a program to prevent adolescent depressive symptoms over 12 months.

Although few prevention studies have been designed to investigate the course of prevention effects over time, it seems that the effects on depressive symptoms increase from post-intervention to 6-month follow-up but then decrease with longer lags to follow-up. Furthermore, previous prevention studies have found differential intervention effects for boys and girls without testing possible explanations for this effect. The present randomized control group study with 301 8th-grade students examined the effects of a depression prevention program from baseline until 12-month follow-up. As expected, while positive intervention effects were found on girls’ depressive symptoms, no such effects were found on boys’ depressive symptoms. Further, the positive intervention effects on girls’ depressive symptoms increased to the 6-month follow-up and remained stable through the 12-month follow-up, while depression symptoms in control-group girls increased from 6-month to 12-month follow-up. Further exploratory analyses revealed that neither baseline conduct problems nor cognitive or social knowledge of the prevention program at 12-month follow-up alone explained the sex effect. However, some limited evidence was found indicating that total knowledge (cognitive and social) might partially explain the effect but there was significant variability remaining to be explained

Self-Efficacy, Intrinsic Motivation, and Academic Outcomes Among Latino Middle School Students Participating in an After-School Program

This longitudinal study examined how academic self-efficacy, intrinsic motivation, and participation in an after-school program contributed to the academic achievement of Latino middle school students over the course of one school year. Participants were 47 Latino students in sixth through eighth grades who attended two public middle schools in which an after-school program was held that was specifically for Latino students. Results from ordinary least squares regression revealed that intrinsic motivation was positively associated with students GPAs, self-efficacy was a positive predictor of students’ school attendance and standardized math achievement scores, and attendance at the after-school program also contributed positively to students’ math achievement. Results from multilevel growth modeling showed that students’ self-efficacy and intrinsic motivation remained stable across the school year and were not related to students’ degree of participation in the after-school program. Several avenues for future research within the Latino student population are discussed

Associations between perceived teaching behaviors and affect in upper elementary school students.

We explored the associations between student-perceived teaching behaviors and negative affect (NA) and positive affect (PA) in upper elementary age students, both before and after controlling for perceived parenting behaviors. The Teaching Behavior Questionnaire (TBQ), the Alabama Parenting Questionnaire (APQ), and the Positive and Negative Affect Schedule for Children (PANAS-C) were completed by 777 third to fifth graders in nine elementary schools. Using two-level hierarchical linear model analyses, we found that (a) perceived instructional teaching behavior was negatively associated with NA and positively associated with PA; (b) perceived organizational behavior was not associated with either; (c) perceived socio-emotional teaching behavior was positively associated with both; (d) perceived negative teaching behavior was positively associated with NA but not associated with PA. When parenting behaviors were controlled for, the associations with NA but not with PA held up. We discuss implications of the findings for education and mental health personnel

Newborn Hearing Screenings for Babies Born at Home: Report from an Initiative in Michigan

Objective: Babies born in an out-of-hospital setting (e.g., homebirth) often do not receive a universal newborn hearing screening (UNHS). The purpose of this study was to evaluate the effect of providing training and equipment for newborn hearing screening to midwives who attend homebirths. Study Design: Midwives from around the state of Michigan were invited to participate in a two-part UNHS training. Hearing screening data from all midwives who attended homebirths (N=112) during the 2015 and 2016 calendar years were analyzed using a two-level multilevel model. Estimated odds of babies being screened were calculated based on midwife group. Results: Having a midwife who hosted an AABR machine at her practice increased the likelihood of receiving a screening by 39.37 times. Having a midwife who had access to an AABR machine increased the likelihood of receiving a screening by 8.57 times. Having a midwife who received focused education about the importance of newborn hearing screening increased the likelihood of receiving a screening by 10.82 times. Conclusion: Providing UNHS equipment to midwives significantly increases the likelihood that babies born at home will receive a hearing screening at birth. This is evidence for the continued outreach and inclusion of midwives in UNHS programs

Examining associations between classroom environment and processes and early mathematics performance from pre-kindergarten to kindergarten.

One benefit of the No Child Left Behind legislation (2001) has been the increasing attention on the importance of the skills learned in the pre-kindergarten period for later academic achievement. There is a growing awareness that mathematics skills in kindergarten and beyond are influenced by the formal and informal mathematics skills acquired in the pre-kindergarten classroom. In recent years, a body of research has emerged pointing to the contributions to children’s learning from pre-kindergarten program quality as indexed by structure and process elements in the classroom. Results from this study point to three major findings. First, the growth of mathematics skills from pre-kindergarten to kindergarten for the full sample varied between classes but was not significantly associated with the elements of the classroom environment selected for study. Second, classroom elements were differentially related to growth of mathematics scores depending on children’s scores at pre-kindergarten entry. Third, overall differences between high- and low-performing children at pre-kindergarten entry are evident in their growth through kindergarten

Examining the effects of gifted programming in mathematics and reading using the ECLS-K

Researchers and educators have debated the benefits and possible drawbacks of gifted programming for decades. Unfortunately, little empirical evidence, particularly with a national sample, supports gifted programming effects on achievement or academic attitudes for either gifted or nongifted students. This type of research has been problematic because we cannot assign students randomly to receive gifted programming or schools randomly to provide it. Using multilevel propensity score stratification to analyze data from the Early Childhood Longitudinal Study, Kindergarten Cohort (ECLS-K), this study explored the effects of school personnel-reported gifted programming in reading and mathematics in upper elementary grades on overall school achievement, on gifted students\u27 achievement and academic attitudes, and on nongifted students\u27 achievement and academic attitudes. The data and results represent a broad, national look at school personnel-reported gifted programming, without distinction as to type, length, or degree of programming. ^ The results indicate that, on average, gifted programming (as identified by administrators and teachers) provides no effects on reading or mathematics achievement or attitudes at the overall school level or for gifted or nongifted students (Cohen\u27s d for each analysis \u3c .10). No detrimental effects of providing gifted programming were found. The results suggest that if a school without a gifted program changed policies and provided one, nongifted students\u27 achievement and academic attitudes would not change. However, the results also indicate that school personnel-reported gifted programs in reading and mathematics, on average, have no effect on gifted students\u27 achievement or academic attitudes. ^ In light of inconsistent policies and programming practices nationwide, these results indicate that school personnel who report that they are providing a gifted program in mathematics or reading may not be providing an effective program in terms of achievement or academic attitudes. Therefore, stakeholders must consider whether research-based programs, instructional practices, and curricula are being implemented to serve gifted students. Future research must be conducted to determine which programs and curricula are effective at improving gifted students\u27 achievement and academic attitudes. Furthermore, school personnel and parents should identify research-based practices and curricula to use with gifted children to increase their reading and mathematics achievement.

Examining the effects of gifted programming in mathematics and reading using the ECLS-K

Researchers and educators have debated the benefits and possible drawbacks of gifted programming for decades. Unfortunately, little empirical evidence, particularly with a national sample, supports gifted programming effects on achievement or academic attitudes for either gifted or nongifted students. This type of research has been problematic because we cannot assign students randomly to receive gifted programming or schools randomly to provide it. Using multilevel propensity score stratification to analyze data from the Early Childhood Longitudinal Study, Kindergarten Cohort (ECLS-K), this study explored the effects of school personnel-reported gifted programming in reading and mathematics in upper elementary grades on overall school achievement, on gifted students\u27 achievement and academic attitudes, and on nongifted students\u27 achievement and academic attitudes. The data and results represent a broad, national look at school personnel-reported gifted programming, without distinction as to type, length, or degree of programming. ^ The results indicate that, on average, gifted programming (as identified by administrators and teachers) provides no effects on reading or mathematics achievement or attitudes at the overall school level or for gifted or nongifted students (Cohen\u27s d for each analysis \u3c .10). No detrimental effects of providing gifted programming were found. The results suggest that if a school without a gifted program changed policies and provided one, nongifted students\u27 achievement and academic attitudes would not change. However, the results also indicate that school personnel-reported gifted programs in reading and mathematics, on average, have no effect on gifted students\u27 achievement or academic attitudes. ^ In light of inconsistent policies and programming practices nationwide, these results indicate that school personnel who report that they are providing a gifted program in mathematics or reading may not be providing an effective program in terms of achievement or academic attitudes. Therefore, stakeholders must consider whether research-based programs, instructional practices, and curricula are being implemented to serve gifted students. Future research must be conducted to determine which programs and curricula are effective at improving gifted students\u27 achievement and academic attitudes. Furthermore, school personnel and parents should identify research-based practices and curricula to use with gifted children to increase their reading and mathematics achievement.