Evaluating Digital Math Tools in the Field

Many school districts have adopted digital tools to supplement or replace teacher-led instruction, usually based on the premise that these tools can provide more personalized or individualized experiences for students and at lower cost. Rigorously evaluating whether such initiatives promote better student outcomes in the field is difficult as most schools and teachers are unwilling to enforce rigorous study designs such as randomized control trials. We used study designs that were feasible in practice to assess whether two digital math tools, eSpark and IXL, were associated with improvements in 3rd – 6th grade student test scores in math. We also investigated the resource requirements and costs of implementing eSpark and IXL to assess whether these tools represent a valuable use of resources. We find that while IXL is substantially less costly to implement than eSpark, its use is not significantly associated with students’ math performance

Using Cost-effectiveness Analysis to Evaluate School of One

This paper applies cost-effectiveness analysis to a blended learning math intervention for middle school students, School of One (recently renamed Teach to One: Math), to assess whether it is a productive use of education funds

Resource Requirements and Costs of Developing and Delivering MOOCs

Given the ongoing alarm regarding uncontrollable costs of higher education, it would be reasonable to expect not only concern about the impact of MOOCs on educational outcomes, but also systematic efforts to document the resources expended on their development and delivery. However, there is little publicly available information on MOOC costs that is based on rigorous analysis. In this article, we first address what institutional resources are required for the development and delivery of MOOCs, based on interviews conducted with 83 administrators, faculty members, researchers, and other actors in the MOOCspace. Subsequently, we use the ingredients method to present cost analyses of MOOC production and delivery at four institutions. We find costs ranging from $38,980 to$325,330 per MOOC, and costs per completer of $74-$272, substantially lower than costs per completer of regular online courses, by merit of scalability. Based on this metric, MOOCs appear more cost-effective than online courses, but we recommend judging MOOCs by impact on learning and caution that they may only be cost-effective for the most self-motivated learners. By demonstrating the methods of cost analysis as applied to MOOCs, we hope that future assessments of the value of MOOCs will combine both cost information and effectiveness data to yield cost-effectiveness ratios that can be compared with the cost-effectiveness of alternative modes of education delivery. Such information will help decision-makers in higher education make rational decisions regarding the most productive use of limited educational resources, to the benefit of both learners and taxpayers

Efficiency of Automated Detectors of Learner Engagement and Affect Compared with Traditional Observation Methods

This report investigates the costs of developing automated detectors of student affect and engagement and applying them at scale to the log files of students using educational software. We compare these costs and the accuracy of the computer-based observations with those of more traditional observation methods for detecting student engagement and affect. We discuss the potential for automated detectors to contribute to the development of adaptive and responsive educational software

MOOCs: Expectations and Reality

This comprehensive study of MOOCs from the perspective of institutions of higher education includes an investigation of definitions and characteristics of MOOCs, their origins, institutional goals for developing and delivering MOOCs, how MOOC data is being used, a review of MOOC resource requirements and costs, and a compilation of ideas from 83 interviewees about MOOCs and the future of higher education. We identify six major goals for MOOC initiatives and assess the evidence regarding whether these goals are being met, or are likely to be in the future

Why do Institutions Offer MOOCs?

By reviewing the literature and interviewing 83 individuals knowledgeable about massive open online courses (MOOCs), we investigate the goals of institutions of higher education that are currently developing and delivering such courses. We identify six major goals for MOOC initiatives: extending reach and access, building and maintaining brand, improving economics by reducing costs or increasing revenues, improving educational outcomes, innovation in teaching and learning, and conducting research on teaching and learning. Comparing these goals with the data being collected about MOOCs, their participants, and educational outcomes, as well as the resource requirements and cost drivers of the development and delivery process, we assess whether these goals are being met, or are likely to be in the future. While quantification of success in achieving these goals is for the most part lacking, we conclude that institutions are experiencing at least partial success in achieving each of these goals except for improving economics. We identify obstacles to fuller achievement of the goals and some potential solutions

Evaluating Digital Math Tools in the Field

Many school districts have adopted digital tools to supplement or replace teacher-led instruction, usually based on the premise that these tools can provide more personalized or individualized experiences for students and at lower cost. Rigorously evaluating whether such initiatives promote better student outcomes in the field is difficult as most schools and teachers are unwilling to enforce rigorous study designs such as randomized control trials. We used study designs that were feasible in practice to assess whether two digital math tools, eSpark and IXL, were associated with improvements in 3rd – 6th grade student test scores in math. We also investigated the resource requirements and costs of implementing eSpark and IXL to assess whether these tools represent a valuable use of resources. We find that while IXL is substantially less costly to implement than eSpark, its use is not significantly associated with students’ math performance

EdTech Decision-making in Higher Education

The study “EdTech Decision-making in Higher Education” investigates the decision-making inputs, processes, and practices around the acquisition of technology to facilitate teaching and learning at colleges and universities. A summary of findings and recommendations, a more detailed report which includes many examples and quotations from study participants, and a repository of resources for EdTech decision-makers is available at: www.edtechdecisionmakinginhighered.org

Providing Comprehensive Educational Opportunity to Low-Income Students. Part 4: What Are the Social and Economic Returns?

Approximately half of all New York City public school students who live in families with incomes less than 185% of the federal poverty level (FPL) do not graduate from high school. These dropouts are much less likely to achieve economic self-sufficiency in adulthood. This creates both a fiscal and social burden. Rothstein, Wilder and Allgood (2011) propose a set of publicly funded comprehensive services to supplement academic services that could effectively raise the educational attainment levels of low income children to levels associated with middle class children. This report, the fourth in a five part feries, calculates the fiscal and social impact of raising the attainment levels of these students in order to assess whether the costs of such services are justified by the benefits. Specifically, the report estimates the economic costs and benefits attributable to a single cohort of 37,000 12th grade students from families with incomes below 185% FPL who are currently finishing up in New York City public schools. It calculates the net fiscal contributions by education level per individual. These contributions are tax revenues, minus government expenditures on healthcare, the criminal justice system, welfare programs, and school/college. The report finds that New York City and State combined spend $82,000 (in lifetime present value terms) on each high school dropout on healthcare, criminal justice, welfare, and education while receiving only$45,000 in tax revenues. This results in a deficit of $37,000 for each dropout. In contrast, for students earning a BA or higher degree, the city and state expenditures amount to$55,000 while tax revenues are $143,000, resulting in a balance of$88,000 in favor of the government. The impact for the federal government is also substantial with high school dropouts costing $1,360 when federal expenditures are netted against federal tax revenues, while students earning a BA or higher degree contribute$347,460 in lifetime present value terms. Evidently dropouts create a greater fiscal burden for the state and city than for the federal government, but the federal government gains dramatically more as education levels increase. The economic analysis provided in this report indicates that provision of all support programs in the comprehensive approach outlined by Rothstein, Wilder, and Allgood (2011) would require a significant outlay by state and local governments above and beyond what is currently expended. However, the resulting estimated 9% rate of return on investment compares favorably with existing forms of government debt, suggesting that a feasible strategy would be to finance the costs of a comprehensive support program for low-income children through the issuance of long-term municipal bonds. Philanthropic foundations could also help fund these social investments by providing grants to service providers that would be replaced by public funds if target outcomes were met

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research

We review the value of cost-effectiveness analysis for evaluation and decision making with respect to educational programs and discuss its application to early reading interventions. We describe the conditions for a rigorous cost-effectiveness analysis and illustrate the challenges of applying the method in practice, providing examples of programs for which we have estimated costs, but find effectiveness data lacking in comparability. We provide a demonstration of how cost-effectiveness analysis can be applied to two early reading programs: the Wilson Reading System and Corrective Reading. We use existing effectiveness data from an experimental evaluation in which the programs were implemented under similar conditions and the use of common outcomes measures for both programs yielded data that are comparable. We combine these data with cost data we collected using the ingredients method to calculate cost-effectiveness ratios for the alphabetics domain. A complete picture of the relative cost-effectiveness of each program could be provided if effectiveness metrics were available for fluency, vocabulary, and comprehension. We highlight the obstacles to applying cost-effectiveness analysis more frequently and recommend strategies for improving the availability of the requisite data