Understanding the reasons for better‐than‐expected performance across Kentucky\u27s 173 school districts, taking into account student outcomes, backgrounds, and school district characteristics. Building on the previous work with school districts and using school-level data, this paper discusses the estimated expected level of school-level performance using district-level fixed effects. From this broad range of student outcomes, family and community backgrounds, and school characteristics, we identify schools that have performed better than expected—which we refer to as “bright spots.

Childress, Michael T.

English

University of Kentucky

Follow this and additional works at: https://uknowledge.uky.edu/cber_researchreports  Part of the Economics Commons, and the Education Commons University of Kentucky UKnowledge CBER Research Report Center for Business and Economic Research 9-2-2020 Kentucky Public Schools as Educational Bright Spots (September 2020) Michael T. Childress University of Kentucky, michael.childress@uky.edu Digital Object Identifier: https://doi.org/10.13023/cber.rr.2020.01 Right click to open a feedback form in a new tab to let us know how this document benefits you.   Center for Business and Economic Research Department of Economics | 244 Gatton College of Business and Economics Building    University of Kentucky | Lexington, KY 40506-0034 | (859) 257-9524 | E-mail: cber@uky.edu               Kentucky Public Schools as Educational Bright Spots   Prepared by Center for Business and Economic Research Gatton College of Business and Economics University of Kentucky   September 2, 2020    Michael T. Childress (michael.childress@uky.edu)              This page is intentionally blank.  Page | 1   Kentucky Public Schools as Educational Bright Spots  ach academic year a select group of Kentucky’s public schools perform better than expected on measures of educational achievement. These measures include things like the percentage of elementary students who achieve proficiency or distinguished in reading, or the proportion of less-advantaged middle school students who show a similar level of competency on the math assessment. Understanding the reasons for better-than-ex-pected performance is fundamentally important. While our analysis does not fully address the question of why stu-dents perform better than expected, our results can be used to inform further inquiry on that question. Our work is best viewed as a statistical sieve designed to narrow the list of candidate schools worthy of closer examination. By subjecting a school to closer scrutiny, one can gain a sense of confidence about identifying the constellation of fac-tors facilitating exceptional performance.  Organized within 173 school districts, there are wide differences in the learning environments, sizes, finances, and student outcomes among and within Kentucky’s 1,466 schools.1 Since schools are likely to reflect the underlying economic conditions of their surrounding communities, the socioeconomic characteristics of Kentucky’s schools are as diverse as the state itself. This is evidenced by the percentages of less-advantaged students in two Fayette County School District elementary schools—Arlington and SCAPA, which are, respectively, 94 and 9 percent. Likewise, the average per pupil expenditures in the top quartile of school districts statewide is one-third higher than those in the bottom quartile—$13,380 compared to $10,140. And the funding extremes appear to be even more pronounced at the school level.  Student outcomes, of course, are the bottom lines for schools and districts, and there is a similarly wide distri-bution of outcomes across the state’s public schools. For example, the percentage of students at Arlington Elemen-tary and SCAPA at Bluegrass achieving a proficient or distinguished score on the 2018-2019 Kentucky Performance Rating for Educational Progress (K-PREP) in reading is vastly different, 29.5 percent at the former and 95.3 at the latter. While the size of this difference might be surprising—roughly 66 percentage points—its existence is not. Stu-dents at SCAPA, on average, have important and substantive advantages that students at Arlington do not, such as more experienced teachers and lower hurdles to learning created by poverty.   From this broad range of student outcomes, family and community backgrounds, and school characteristics, we identify schools that have performed better than expected—which we refer to as “bright spots.” For example, Knox County Middle School and South Laurel Middle School in Laurel County performed similarly on the 2018-2019 K-PREP middle school mathematics assessment, demonstrated by 50.9 and 51.1 percent of their students scoring pro-ficient or distinguished, respectively. Yet, once we consider student, school, district, and community factors, only one of these schools performs “better than expect”—Knox County Middle School. While South Laurel Middle School performs at a level we expect, Knox County Middle School performs much better than we expect; in fact, it performs 20 percentage points higher than we expect. In the sections that follow, we provide additional information on our method, approach, and results.  METHOD sing a school-level database that includes, but is not limited to, data from the Kentucky Department of Educa-tion (KDE), the Kentucky Center for Statistics (KYstats), and the U.S. Census Bureau, we analyze data covering eight academic years—2011-12 to 2018-19.2 We estimate an expected level of school-level performance using dis-trict-level fixed effects panel regression analysis—a statistical method for estimating, expressing, and understanding the relationships between variables—and then compare it to the actual performance. The difference between actual performance and model-based expected performance is the residual. If the size of the residual is sufficiently large  1 There are 38 Preschools, 655 Elementary (preschool-6th grade), 84 Elementary/middle (any combination of P-6 and grades 6-8), 203 Middle (grades 6-8), 170 Middle/high (any combination of grades 6-8 and 9-12), 257 High (grades 9-12), and 59 P-12 (any combination of P-6, grades 6-8 and 9-12). Refer to the Kentucky Department of Education website <https://education.ky.gov/comm/edfacts/Pages/default.aspx>. 2 For the Kentucky Department of Education School Report Card data, these years correspond to the 2011-2012 to the 2018-2019 academic years. The Census data, on the other hand, is based on a calendar year. E U Page | 2   and positive, we consider it a “bright spot candidate.”3 The development and creation of our statistical models is informed by Prichard Committee personnel, the scholarly literature on factors affecting student outcomes, data availability, and technical considerations regarding variable selection and model construction.  Outcome Variables The 35 educational outcome variables include:  1. K-PREP Reading, Elementary School (combined grades and 3rd grade alone) and Middle School (combined grades and 8th grade alone), percentage reaching proficient/distinguished for three groups: total students, those participating in the National School Lunch Program (NSLP), and disabled students with an individual-ized educational plan (IEP), (12 outcome variables). 2. K-PREP Mathematics, Elementary School (combined grades and 3rd grade alone) and Middle School (com-bined grades and 8th grade alone), percentage reaching proficient/distinguished for three groups: total students, those participating in the National School Lunch Program (NSLP), and disabled students with an individualized educational plan (IEP), (12 outcome variables). 3. ACT Grade 11 Average Score (overall composite) as well as percentages reaching college readiness bench-marks for reading and math, for three groups: total students, those participating in the National School Lunch Program (NSLP), and disabled students with an individualized educational plan (IEP), (9 outcome var-iables). 4. In- and out-of-state college going rates (combined) derived from the KDE Report Card data (i.e., Transition to Adult Life after Graduation) and the in-state college going rate from the Kentucky High School Feedback Reports (2 outcome variables).4  Predictor Variables During the initial phase of this project, which focused on school districts, we examined several statistical models, which included different combinations of independent or predictor variables. These variables include socioeconomic factors (e.g., percentage qualifying for free and reduced lunch), demographic characteristics (e.g., race), teacher characteristics (i.e., experience), student population factors (e.g., ELL), geographic measures (e.g., urbanity), and community characteristics (e.g., educational attainment of adults, children living in nonfamily households). Ulti-mately, we use the same variables in our models for both the district and school level analyses, which are listed below:   1. School districts are used for the school-level fixed effects analysis (173 districts) 2. Less-advantaged students, specified as the percentage of the students in a school participating in the Na-tional School Lunch Program (NSLP). In 2017, the Kentucky average is 60.4 percent (56.4% free, 4.4% re-duced).5 3. Children under 18 living in single parent or nonfamily households, specified at the county level. Around 34 percent of children in Kentucky had this living arrangement in 2017. 4. Minority children, specified as the percentage of non-White (not Hispanic) children enrolled in the school (i.e., African American, Hispanic, Asian, American Indian or Alaska Native, Native Hawaiian or Other Pacific Islander, or multiple races). The statewide percentage of non-White (not Hispanic) students was 22.6 per-cent in 2017. 5. Teaching experience, specified as the average years of teaching experience in the school. The statewide average in 2017 was 11.9 years. 6. School student enrollment. The average school sizes by type of school during the 2018-2019 academic year are: 430 students for elementary schools, 493 for middle schools, and 753 for high schools.  3 The threshold we use for determining whether a positive residual is a “significant” is when the studentized resid-ual is 2 or higher.  4 The Kentucky High School Feedback Report data on in-state college going rates are only available for 7 years, the academic years 2011-12 to 2017-18. 5 This is the statewide average.  Page | 3   7. Year, specified as the year of the panel data, 2011 to 2018.   For the models used to assess the college going rates, we included the variables listed above along with two additional variables:   8. Bachelor’s degree or higher, specified as the percentage of the population 25 and over in the county with at least a BA degree. The statewide percentage in 2017 was 24 percent. 9. ACT Composite, specified as the district level overall ACT composite score. The statewide percentage was 19.8 in 2017.   The summary statistics for the database are presented in Table 1 (in the appendix, PAGE 8).6  Approach  For each of the 35 educational outcome measures, we use a district-level fixed effects panel regression model; the seven to nine independent variables used in the models are listed in the previous section (Predictor Variables). There are two conditions that a school must meet in order to satisfy our definition as a “bright spot.” First, we evaluate all students on an outcome measure, such as K-PREP elementary mathematics outcomes, to assess whether a school exhibits better-than-expected performance at least once from 2011 to 2018; in other words, we are looking for significant studentized residuals.7 Second, while focusing on the same educational outcome meas-ure, but for at-risk students,8 we analyze the model residuals to assess whether a school exhibits a significant im-provement in performance relative to expectations over the time period; in this case, we regress the residuals on year, and if year is positive and statistically significant, then it is improving relative to expectations over the time period. Any school that satisfies both of these conditions on an educational outcome is deemed a “bright spot.”   While recognizing that there are many ways to define a bright spot, we describe below how Sublimity Elemen-tary School, which is part of the Laurel County School District, exemplifies our criteria. Sublimity Elementary School has ranged in size from 307 to 361 students from 2011 to 2018. On average, about 170 students take the K-PREP mathematics assessment each academic year. In the 2011-2012 academic year, about half of the students scored proficient or distinguished, but this increased to 83.4 percent by the 2018-2019 academic year (see Figure 1). In the earlier years, the school’s proficient-distinguished percentage fell below the level estimated by our model; this is shown in Figure 1 with the green bars (actual percentage) below the blue line (expected percentage). By the 2018-2019 academic year, however, the percentage of students at Sublimity Elementary scoring proficient or dis-tinguished on the K-PREP mathematics assessment was 83.4 percent, far exceeding the expected percentage of 63.9 percent. The actual performance exceeded the expected performance by 19.5 percentage points—a signifi-cant positive residual that meets our criteria as a “bright spot.”   6 The values in Table 1 include data for schools, students, and their surrounding communities for eight years (aca-demic years 2011-2012 to 2018-2019).  7 As previously noted, our threshold is a studentized residual equal to or greater than 2. 8 We define at-risk students as those participating in the National School Lunch Program (NSLP) and/or disabled students with an individualized educational plan (IEP).  Page | 4      While it is necessary for all students, on average, to perform significantly above the expected level to be con-sidered a bright spot, this, by itself, is not sufficient. Our second condition is that at-risk students must show signifi-cant improvement relative to expectations over the time period studied. We evaluate students participating in the National School Lunch Program as well as disabled students with an individualized educational plan to determine the level of progress. If one or both of these at-risk groups shows significant progress, then the condition is met.   Our first step is to analyze each group’s performance using the same district-level fixed effects panel regres-sion model used for the total student body, or all students. Then, we regress the residuals on year to determine if the parameter is positive and statistically significant. In the case of Sublimity Elementary School, both at-risk groups (i.e., NSLP and IEP students) show significant improvement in K-PREP mathematics performance relative to expectations (see Figures 2 and 3). During the 2018-2019 academic year, 69 percent of the students at Sublimity Elementary School participated in the National Free Lunch Program, and, on average during the study period, about 105 students took the K-PREP mathematics assessment each year. A much smaller number of students with an individualized education plan, about 26 each year, were tested. Nonetheless, both groups of at-risk students demonstrated statistically significant improvement over this time period. The residuals are negative at the begin-ning of the time period for both at-risk groups, indicating that performance was lower than expected, but then the residuals gradually become positive and larger, indicating that performance was higher than expected. While both groups of at-risk students at Sublimity Elementary School showed significant improvement relative to expecta-tions, only one at-risk group needs to show improvement to be considered a bright spot candidate.   Page | 5      BRIGHT SPOTS RESULTS he information provided below in Table 2 shows the 47 “bright spot” schools meeting our two conditions de-scribed in the Approach section. There are 28 elementary schools, 4 middle schools, and 15 high schools that can be viewed as bright spots by virtue of student performance on K-PREP or ACT assessments, or successfully transitioning to college; and some schools qualify as bright spots in more than one category. Since we do not have data on college going for at-risk students—only the total graduating class—we used different criteria. For these two outcome measures, we assess the change over time for the total group instead of at-risk groups.  From left to right, the columns in Table 2 show the district and school identifier assigned to a school by the Kentucky Department of Education (Sch_Cd), the school district where the school is located, the school name, the educational outcome category, and the number of years from 2011 to 2018 where all students performed better than expected. The three columns on the right indicate whether groups of students exhibited significant improve-ment relative to expectations over the time period, either all students (TST), those qualifying for the National School Lunch Program (LUP), and those with an individualized educational plan (ACD); the numbers shown are the T Page | 6   t-values of the bivariate regression slope, where residuals are regressed on year. For example, the first row shows Oakview Elementary School is a bright spot for 3rd grade reading. It demonstrated better-than-expected perfor-mance for all students in one year, and students participating in the NSLP (LUP residual) evidenced significant im-provement during the time period with a t-value of 3.3; IEP students (ACD residual) did not show significant posi-tive improvement, which is reflected by the “not sig.”         Page | 7   CONCLUSIONS he 47 “bright spot” schools that performed better than expected from 2011 to 2018 are located in all regions of the state and 30 different counties, as illustrated in the county-level map below; these are diverse settings—urban-rural, east-west, distressed areas as well as prosperous ones.      Our analysis confirms what research has long revealed—that less-advantaged and minority students can face difficult obstacles in the pursuit of academic success. Of the 35 educational outcome models we tested, the predictor variables of less-advantaged students (i.e., % NSLP participants) and minority students (i.e., % nonwhite) were sta-tistically significant and negative in 34 and 30 models, respectively. Additionally, teacher experience—the average number of years teaching—was statistically significant and positive in 20 of the 35 models; the impact of experienced teachers was mostly concentrated in the elementary level KPREP reading, 8th grade KREP reading, and in each of the nine ACT models.   Understanding the reasons for better-than-expected performance is fundamentally important. These results of this analysis can be sorted, selected, and combined with other pieces of information, if desired, to identify educa-tional bright spots worthy of closer examination. With closer qualitative examination, it is possible to identify the critical factors leading to better-than-expected educational outcomes. Given the wide geographic distribution of educational bright spots, there are many candidates available across the Commonwealth for further study and ex-amination.    Michael Childress Center for Business and Economic Research University of Kentucky 859.257.2912 Michael.childress@uky.edu  (Revised September 2, 2020)    T Page | 8    APPENDIX 

Kentucky Public Schools as Educational Bright Spots (September 2020)

https://uknowledge.uky.edu/cgi/viewcontent.cgi?article=1027&amp;context=cber_researchreports

Kentucky Public Schools as Educational Bright Spots (September 2020)

Abstract

Similar works

Full text

Available Versions

University of Kentucky