Empirical Support for Establishing Common Assumptions in Cost Research in Education

The economic evaluation of educational policies and programs employing the ingredients method for cost, cost-effectiveness, or benefit-cost analysis is no exception to the critique that economic models require an untenable number of assumptions. Educational economists must make assumptions due to two sources of uncertainty: model uncertainty, as in the well-documented debate over the selection of the appropriate social discount rate to calculate present value and empirical uncertainty due to the infeasibility of gathering sufficiently detailed data on all resources. This paper highlights the frequency of empirical assumptions made in the education literature and proposes a set of harmonized assumptions to address empirical uncertainty that can be used to increase comparability of economic evaluation across programs and across studies. By building consensus on a set of reasonable, empirically derived assumptions that are selected so as to minimally distort the results of evaluations, differences in costs, cost effectiveness, and benefit-cost ratios can be more confidently ascribed to meaningful differences in resource use, program implementation, and program effectiveness, as opposed to differences in choices made by the analyst

NEPC Review: The System-Level Effects of Denver’s Portfolio District Strategy: Technical Report (Center for Education Policy Analysis, University of Colorado Denver, December 2021)

This report analyzes changes in academic performance as measured by test scores and graduation rates in the Denver Public Schools versus comparable schools in Colorado over 11 years of the district&rsquo;s experimenting with the &ldquo;portfolio&rdquo; approach to school district management. This approach includes central-office oversight of different school types (such as charter schools, innovation schools, and district-run schools), with widespread parental choice under a single enrollment system. The reported academic gains are dramatic and worth drawing attention to, but attributing them specifically to the portfolio reforms seems premature and thus not useful in showing how other districts could replicate that success.</p

All Together Now: The Impact of Team-Based Problem-Solving on Teacher Learning and Effectiveness

Schools face a great challenge in recruiting and retaining quality teachers, given the documented importance of, variability in, and difficulty observing and predicting teacher quality. One option schools have is to identify what more effective teachers do and use that information to train less effective teachers to get better. Unfortunately, there is little empirical support for much traditional teacher training, as measured by gains in student test scores. Models of collaborative, team-based learning – such as Professional Learning Communities and Japanese lesson study – have been widely touted, and there is some evidence that they may be effective in certain contexts. Economic theory suggests this could be because of peer monitoring, peer pressure, specialization, knowledge-sharing, or market failure in pre-service training, particularly if learning to teach is primarily experiential. However, not all collaboration is good due to concerns about free-riding and substituting for more productive individual activity, so unbridled enthusiasm for collaborative professional development may need to be tempered. This dissertation examines the effectiveness of a specific form of teacher collaboration in the form of inquiry teams, groups of teachers and administrators jointly engaged in action research projects with the aim of uncovering innovative instructional strategies and sharing effective approaches. It takes advantage of the phase-in of teams, eventually to all teachers in a large, urban school district in the northeastern United States from 2007-2010 to estimate the results of three natural experiments using difference-in-differences and instrumental variables approaches. The effects of teamwork on teacher value-added, teacher retention, and student test scores are small and sensitive to year, specification, and outcome, although results are mostly positive and occasionally statistically significant, suggesting that overall effects are potentially positive but modest at best. Further examination of heterogeneity and four qualitative case studies of teams suggest that small average effects mask considerable differences in team processes, and that under certain conditions, inquiry team work may be far more effective. A cost analysis reveals that, although it is costly to do inquiry work well, given the low-intensity of average treatment and the large number of students affected, the benefits of inquiry work could exceed the costs if the policy were more targeted. Overall, the policy recommendation is to temper unqualified enthusiasm about teacher collaboration, as without appropriate structures and supports it has little measurable effect on the outcomes examined here. As a policy lever, a universal mandate to participate on collaborative inquiry teams is unlikely to be effective or pass a cost-benefit test. Nonetheless, smaller-scale, higher intensity forms of collaboration that allow for more active leadership support and participation may be more promising, and more cost-effective than alternative forms of professional development, particularly for some sub-groups of teachers such as those in their first year of teaching

REACH Cost Analysis

In this report, we present a cost analysis of Raising Educational Achievement Coalition of Harlem (REACH), a partnership between Teachers College, Columbia University, and five high-needs schools in Harlem, New York City. A rigorous cost analysis can help illuminate the resources used to implement its theory of action, in addition to contextualizing the size of measured effects in a broader implementation framework and helping decision-makers select among alternative uses of scarce resources. REACH entails deep collaboration between schools and program staff in five key areas: Leadership, Teaching and Learning, Expanded Learning Opportunities, Physical and Mental Health, and Family and Community Engagement. The program supports schools in achieving their goals for student learning by utilizing university and community resources, including research from faculty, and graduate student assistants working as interns or volunteers in exchange for hands-on learning experiences. We used the ingredients method for cost analysis, documenting all resources utilized to operationalize the program’s theory of action regardless of whether each resource has a monetary cost or who pays for or provides the resource, in order to fully capture the economic or opportunity cost of the program. We obtained data on ingredients from program documentation, a detailed report on program implementation, and interviews and personal communications with program staff. In 2016-17, REACH cost $2,732,960, or$1,560 per student, with substantial variation by school site, domain of REACH, type of ingredient, and source of ingredient and associated funding. We supplement this analysis with a case study of the Teachers College Community School and sensitivity analysis. While the costs of REACH are substantial, the program itself is comprehensive and wide-reaching; further study should compare the costs of REACH to measured effects in a variety of areas, including student test scores, and behavioral, health, and socioemotional learning outcomes

Estimating a Price Tag for School Vouchers

School vouchers, a school choice policy that allows students and families to use public funds to fully or partially pay the cost of attending private schools, became a major area of policy debate once again during Betsy DeVos&rsquo;s tenure as United States Secretary of Education. Recent evaluations have found negative impacts of vouchers on academic outcomes among students using them, particularly on academic test scores, although older research has found mixed results. However, the input side has received less research attention; we know relatively little about the costs of implementing voucher systems. This policy brief provides an overview of the literature on the effects and costs of vouchers and applies recent empirical evidence on policy effects, behavioral responses, and contextual factors to determine administrative costs of a universal voucher system. It concludes with recommendations for further research and policy. </div

The Economic Value of Social and Emotional Learning

There is a growing body of research emphasizing the advantages of teaching students social and emotional (SE) skills in school. Here we examine the economic value of these skills within a benefit-cost analysis (BCA) framework. Our examination has three parts. First, we describe how the current method of BCA must be expanded to adequately evaluate SE skills, and we identify important decisions analysts must make. Second, we review the evidence on the benefits of SE skills, again noting key methodological issues with respect to shadow pricing. Finally, we perform BCA of four selected social and emotional learning (SEL) interventions: 4Rs; Second Step, Life Skills Training; and Responsive Classroom. These analyses illustrate both methodological and empirical challenges in estimating net present values for these interventions. Even with these challenges, we find that the benefits of these interventions substantially outweigh the costs. We highlight promising areas of research for improving the application of BCA to SEL

A Benefit-Cost Analysis of City Connects

Schools have historically and increasingly played an important role in providing services to meet students’ social and emotional, family, health, and academic needs. Coordinating these services in a way that is strategically aligned with a school’s academic mission and that efficiently addresses the needs of all students is often challenging and costly. This study is an initial investigation of Boston College’s City Connects program, which supports students and schools by evaluating the needs of all students in a school and connecting them to services that are largely provided by community partner organizations. The program aims to help students by connecting them with an individualized set of services to address their academic, social/emotional, family, and health needs. The program also aims to assist schools by connecting them with community agencies and service providers, and streamlining student support referral and management to make the process of providing comprehensive approaches to supporting student learning more strategic and efficient. Prior research has shown evidence of effectiveness of City Connects in terms of increased achievement and educational attainment relative to similar schools that have not implemented the program (City Connects Progress Report, 2014; Walsh, et al., 2014a; 2014b). These positive effects must be weighed against the program’s costs in a benefit-cost analysis to determine whether the program is a worthwhile social investment. This report shows that City Connects provides a whole-school comprehensive service at relatively low cost to the schools—schools themselves only bear about 10% of the core costs of the program. However, the methodological complexity of this work is entailed in the estimation of the total cost when considering the partnerships with community organizations. The results show that the total cost of six years of participation in City Connects from kindergarten through fifth grade (the dosage under which effects were measured) is $4,570 per student, which includes a portion of the costs of the community partner services received by the students in City Connects schools. Depending on what share of the community partner services are considered to be above and beyond the baseline level, the total cost estimate can range from$1,540 to $9,320 per student. Under the model that is most plausible based on implementation data, the benefit-cost ratio is 3.0 and the net benefits are$9,280 per student. This result implies that providing the program to a cohort of 100 students over six years would cost society $457,000 but yield$1,385,000 in social benefits, for a net benefit of $928,000. Even under the most conservative assumptions regarding costs and benefits, the program’s benefits exceed its costs. Sensitivity tests show that the benefit-cost ratio lies somewhere between 1 and 11.8, with a best estimate of$3.00 in benefits per dollar of cost. Further research can investigate the relationship between the program, schools, and community partners and how services provided by partners compare in treatment versus comparison schools

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research

We review the value of cost-effectiveness analysis for evaluation and decision making with respect to educational programs and discuss its application to early reading interventions. We describe the conditions for a rigorous cost-effectiveness analysis and illustrate the challenges of applying the method in practice, providing examples of programs for which we have estimated costs, but find effectiveness data lacking in comparability. We provide a demonstration of how cost-effectiveness analysis can be applied to two early reading programs: the Wilson Reading System and Corrective Reading. We use existing effectiveness data from an experimental evaluation in which the programs were implemented under similar conditions and the use of common outcomes measures for both programs yielded data that are comparable. We combine these data with cost data we collected using the ingredients method to calculate cost-effectiveness ratios for the alphabetics domain. A complete picture of the relative cost-effectiveness of each program could be provided if effectiveness metrics were available for fluency, vocabulary, and comprehension. We highlight the obstacles to applying cost-effectiveness analysis more frequently and recommend strategies for improving the availability of the requisite data

A Benefit-Cost Analysis of a Long-Term Intervention on Social and Emotional Learning in Compulsory School

There is growing evidence that social and emotional skills can be taught to students in school and teaching these skills can have a positive effect on later outcomes, such as better mental health and less drug use. This paper presents a benefit-cost analysis of a longitudinal social and emotional learning intervention in Sweden, using data for 663 students participating in the evaluation. Intervention costs are compared against treatment impact on self-reported drug use. Pre-test and post-test data are available. Since follow-up data for the participants´ drug use as adults is not available, informed projections have been made. Net present monetary values are calculated for the general public and society. The results show that students in the treatment group report decreasing use of drugs over the five year long intervention, the value of which easily outweighs the intervention costs