Early Learning Innovation Fund Evaluation Final Report

This is a formative evaluation of the Hewlett Foundation's Early Learning Innovation Fund that began in 2011 as part of the Quality Education in Developing Countries (QEDC) initiative.  The Fund has four overarching objectives, which are to: promote promising approaches to improve children's learning; strengthen the capacity of organizations implementing those approaches; strengthen those organizations' networks and ownership; and grow 20 percent of implementing organizations into significant players in the education sector. The Fund's original design was to create a "pipeline" of innovative approaches to improve learning outcomes, with the assumption that donors and partners would adopt the most successful ones. A defining feature of the Fund was that it delivered assistance through two intermediary support organizations (ISOs), rather than providing funds directly to implementing organizations. Through an open solicitation process, the Hewlett Foundation selected Firelight Foundation and TrustAfrica to manage the Fund. Firelight Foundation, based in California, was founded in 1999 with a mission to channel resources to community-based organizations (CBOs) working to improve the lives of vulnerable children and families in Africa. It supports 12 implementing organizations in Tanzania for the Fund. TrustAfrica, based in Dakar, Senegal, is a convener that seeks to strengthen African-led initiatives addressing some of the continent's most difficult challenges. The Fund was its first experience working specifically with early learning and childhood development organizations. Under the Fund, it supported 16 such organizations: one in Mali and five each in Senegal, Uganda and Kenya. At the end of 2014, the Hewlett Foundation commissioned Management Systems International (MSI) to conduct a mid-term evaluation assessing the implementation of the Fund exploring the extent to which it achieved intended outcomes and any factors that had limited or enabled its achievements. It analyzed the support that the ISOs provided to their implementing organizations, with specific focus on monitoring and evaluation (M&E). The evaluation included an audit of the implementing organizations' M&E systems and a review of the feasibility of compiling data collected to support an impact evaluation. Finally, the Foundation and the ISOs hoped that this evaluation would reveal the most promising innovations and inform planning for Phase II of the Fund. The evaluation findings sought to inform the Hewlett Foundation and other donors interested in supporting intermediary grant-makers, early learning innovations and the expansion of innovations. TrustAfrica and Firelight Foundation provided input to the evaluation's scope of work. Mid-term evaluation reports for each ISO provided findings about their management of the Fund's Phase I and recommendations for Phase II. This final evaluation report will inform donors, ISOs and other implementing organizations about the best approaches to support promising early learning innovations and their expansion. The full report outlines findings common across both ISOs' experience and includes recommendations in four key areas: adequate time; appropriate capacity building; advocacy and scaling up; and evaluating and documenting innovations. Overall, both Firelight Foundation and TrustAfrica supported a number of effective innovations working through committed and largely competent implementing organizations. The program's open-ended nature avoided being prescriptive in its approach, but based on the lessons learned in this evaluation and the broader literature, the Hewlett Foundation and other donors could have offered more guidance to ISOs to avoid the need to continually relearn some lessons. For example, over the evaluation period, it became increasingly evident that the current context demands more focused advance planning to measure impact on beneficiaries and other stakeholders and a more concrete approach to promoting and resourcing potential scale-up. The main findings from the evaluation and recommendations are summarized here

An evaluation of the integration of m-learning in Total Reading Approach for Children Plus (TRAC+): Enhancing literacy of early grade students in Cambodia.

The Total Reading Approach for Children (TRAC) project was first implemented in Cambodia from 2013 to 2014 by World Education, Inc. (WEI) to improve early grade reading outcomes among Grade 1 and Grade 2 students. This was made possible through a grant from All Children Reading: A Grand Challenge for Development (ACR GCD). ACR GCD, which was launched in 2011 by the United States Agency for International Development (USAID), World Vision, and the Australian Government, is an ongoing series of competitions that leverages science and technology to source, test, and disseminate scalable solutions to improve the literacy skills of early grade learners in developing countries. End-of-project assessments of TRAC were encouraging: over 90% of performance indicators were successfully achieved. As a result, WEI was awarded follow-on funding by World Vision International – Cambodia to scale up TRAC. Called TRAC Plus (TRAC+), the scale up rolled out in 13 World Vision area development programs in five provinces in Cambodia in December 2014. In Year 1, TRAC+ ran in 170 schools, and continued to work in 138 of the 170 original target schools in Year 2. By the end of the project in September 2017, TRAC+ had directly reached about 20,000 students. This report presents the findings of an independent evaluation of TRAC+ conducted from February to September 2017 by Dr. Grace Oakley, Dr. Mark Pegrum, and Dr. Thida Kheang—all from the Graduate School of Education, The University of Western Australia—assisted by Cambodian researcher Mr. Krisna Seng. The primary focus of the evaluation was the m-learning component of TRAC+—the use of Aan Khmer, a game-based app developed with funding from ACR GCD to teach Khmer alphabetical principles, vocabulary, and fluency in low resource environments. The evaluation set out to answer the question, “How and to what extent does the integration of m-learning in TRAC+ enhance the literacy of early grade students?” The findings of this study contribute to the body of knowledge on the effectiveness, sustainability, and scalability of m-learning integrated into TRAC+ in the Cambodian primary school context. Equity and efficiency issues were also addressed. This evaluation was conducted under the Digital Learning for Development (DL4D) project of the Foundation for Information Technology Education and Development (FIT-ED) of the Philippines. As part of the Information Networks in Asia and Sub-Saharan Africa (INASSA) program of the International Development Research Centre (IDRC) of Canada and the Department for International Development (DFID) of the United Kingdom, DL4D aims to improve educational systems in developing countries in Asia through testing digital learning innovations and scaling proven ones. Funding for the evaluation was provided jointly by DL4D and ACR GCD.This research was conducted under the Digital Learning for Development (DL4D) project of the Foundation for Information Technology Education and Development (FIT-ED) of the Philippines, jointly funded by the International Development Research Centre (IDRC) of Canada, the Department for International Development (DFID) of the United Kingdom, and All Children Reading: A Grand Challenge for Development (ACR GCD)—a partnership of United States Agency for International Development (USAID), World Vision, and the Australian Government

LITERACY AND EDUCATIONAL QUALITY IMPROVEMENT IN ETHIOPIA: A MIXED METHODS STUDY

This study examines the development of early grade reading skills as a means for quality improvement in global education. Specifically, this study explores the contextual factors that affect the achievement of early reading skills in Ethiopia and investigates the relationship between literacy and educational quality. The sequential explanatory mixed-methods design is employed to answer four research questions: 1. According to the Ethiopia Early Grade Reading Assessment (EGRA) dataset in the Addis Ababa region, what contextual factors affect achievement in basic literacy skills and how are they related? 2. According to qualitative data, how do parents' and teachers' perspectives explain and substantiate the contextual factors identified in the EGRA dataset and do other factors emerge? 3. Given the answers to the first two research questions, what are the factors associated with achievement that are most favorable and most challenging for literacy development? 4. Given the answer to the third research question, how can interventions for literacy development be best implemented in relationship to overall educational quality improvement? The first, quantitative phase of this study shows that a vast majority of students do not perform at expected levels on the Ethiopia EGRA. The results from three multiple regression analysis models for oral reading fluency and reading comprehension outcomes suggest that both in-school and out-of-school variables have a significant influence on student achievement. The second, qualitative phase of this study reveals several important findings above and beyond those identified in Phase I. First, the findings from both Phase I and Phase II demonstrate the importance of out-of-school variables, but the importance of these to both teachers and parents was underestimated in Phase I. School directors, parents, and teachers highlight the home environment as the most important factor in student achievement. This study demonstrates the utility of a mixed-methods approach to investigate more holistically the practice of literacy in Ethiopia and its relationship to the pursuit of educational quality more broadly. This study also provides a responsive, critical, and theoretical grounding for understanding conflicting perspectives, policies, and approaches to improving the quality of education through literacy development

The Early Grade Reading Assessment

The Early Grade Reading Assessment (EGRA) measures students' progress towards reading. EGRA gauges early literacy skills through a 15-minute individual oral assessment of five fundamental reading skills. RTI worked with education experts to develop the EGRA in 2006, and it has been piloted and implemented in more than 40 countries. This volume aims to take stock of the substantial amount of information and experience generated through the use of EGRA, and to share this knowledge with practitioners, policymakers, and international donors. Chapters cover not only particular applications of the instrument but also put EGRA in the context of broader issues and developments in literacy and education.Publishe

Smaller, Quicker, Cheaper: Improving Learning Assessments for Developing Countries

More and more children are going to school in developing countries. In the years since the 2000 UN Education for All Summit in Dakar, the poorest nations have made the most gains in achieving improved educational access. This is a major achievement. Such success also comes with a realization that rapid growth in school enrollments is not enough. Schooling must be of good quality for all children, and that has not been the case for too many children to date. The next push for educational development will surely focus on improving learning and educational quality. Learning assessments can play an important role to drive school reform in many countries, but many are not adaptable for developing country needs, or are not financially sustainable. Thus, it is imperative tht we develop the appropriate tools that can provide better ways of measuring learning outcomes that nations wish to achieve

The Early Grade Reading Assessment: Applications and interventions to improve basic literacy

The Early Grade Reading Assessment (EGRA) measures students' progress towards reading. EGRA gauges early literacy skills through a 15-minute individual oral assessment of five fundamental reading skills. RTI worked with education experts to develop the EGRA in 2006, and it has been piloted and implemented in more than 40 countries. This volume aims to take stock of the substantial amount of information and experience generated through the use of EGRA, and to share this knowledge with practitioners, policymakers, and international donors. Chapters cover not only particular applications of the instrument but also put EGRA in the context of broader issues and developments in literacy and education

Enabling teachers through action research: A case study of building social and emotional skills for the Kenya Competency-Based Curriculum

This exploratory case study was conducted in Grades 1-3 of one government primary school in Kwale county, Kenya, between June and October 2019. It used mixed methods within a transformative paradigm, informed by action research approaches. The research aimed to understand better how best to support teachers in their teaching and assessment of the Kenya Competency-Based Curriculum (KCBC). The Basic Education Curriculum Framework (BECF), launched in 2017, promotes inclusive, holistic, quality education. It reflects the Kenyan government’s recognition of the importance of social and emotional competencies, or ‘life skills’, for coming generations in Kenya, and its commitment to working towards Sustainable Development Goal (SDG) 4. The research process and its findings provide a space for the voices of teachers: their commitment, discoveries, and relationships with learners and their families. Teachers experienced that a deeper understanding of the competencies of the KCBC positively influenced their practice, noting the inter-connectedness of competencies as they supported their pupils to acquire them. Changes in teaching practice were observed, including increased teacher sensitivity and self-reflection. Teachers were found to be role models for strong relationship skills throughout the school day, facilitating awareness of the new curriculum in the community. A teacher-led social and emotional competency rating scale was adapted and tested. The updated tool provides five questions per competency, for four of the seven KCBC competencies. Use of the tool at two timepoints showed significant progress in Grade 1-3 learners’ demonstration of behaviours that build the competencies. Significant correlations were found between results for individual competencies, and for Grade 3 pupils, between reading and mathematics scores and the competencies of Self-efficacy, Communication and collaboration and Critical thinking and problem solving. The study offers important insights for teacher support and professional development nationally in Kenya, which are relevant to the growing number of competency-based curricula worldwide

Can an Accelerated Intervention Close the School Readiness Gap for Disadvantaged Children? An Evaluation of the Effects of the LEARN Project’s Summer Pre-Primary Program on Literacy Outcomes in Northern Lao PDR

Developed against the backdrop of Sustainable Development Goal 4, as well as a global trend towards rigorous assessment of early childhood programs, this thesis answers questions about the effects of an accelerated school readiness intervention for non-Lao children in disadvantaged communities of Lao People’s Democratic Republic. Through a longitudinal, cluster randomized control trial, the study employs multi-level regression with an analytical sample of 391 children to examine the outcomes of a summer pre-primary program piloted from 2015-2018 by the Lao government with support from Plan International and Save the Children International in the Dubai-Cares funded Lao Educational Access, Research, and Networking (LEARN) Project. Research questions are investigated through a design in which the same panel of children are assessed against a control group at three intervals using the Measurement of Development and Early Learning. The thesis identifies significant associations between receiving the treatment and achieving higher gain scores on several emergent literacy tasks between baseline and midline, with effects roughly in line with similar interventions in other contexts. At the same time, the thesis finds that those effects had largely faded by endline. An interaction between treatment and ethnicity was only evident in a few instances, suggesting that the intervention may have boosted school readiness for Khmu children more by the start of grade 1 and for Hmong children more during grade 1. The thesis raises important recommendations about how to improve the fit between the ultimate objectives of accelerated interventions, the evaluations they undergo, and the needs of the broader education system. New contributions to knowledge are also found by interrogating a global assessment paradigm through a comparative linguistic lens, so that forthcoming evaluations benefit from the lessons learned based on LEARN’s attempt to fit a square peg into a unique alpha-syllabic, tonal Southeast Asian language

Using State-of-the-Art Speech Models to Evaluate Oral Reading Fluency in Ghana

This paper reports on a set of three recent experiments utilizing large-scale speech models to evaluate the oral reading fluency (ORF) of students in Ghana. While ORF is a well-established measure of foundational literacy, assessing it typically requires one-on-one sessions between a student and a trained evaluator, a process that is time-consuming and costly. Automating the evaluation of ORF could support better literacy instruction, particularly in education contexts where formative assessment is uncommon due to large class sizes and limited resources. To our knowledge, this research is among the first to examine the use of the most recent versions of large-scale speech models (Whisper V2 wav2vec2.0) for ORF assessment in the Global South. We find that Whisper V2 produces transcriptions of Ghanaian students reading aloud with a Word Error Rate of 13.5. This is close to the model's average WER on adult speech (12.8) and would have been considered state-of-the-art for children's speech transcription only a few years ago. We also find that when these transcriptions are used to produce fully automated ORF scores, they closely align with scores generated by expert human graders, with a correlation coefficient of 0.96. Importantly, these results were achieved on a representative dataset (i.e., students with regional accents, recordings taken in actual classrooms), using a free and publicly available speech model out of the box (i.e., no fine-tuning). This suggests that using large-scale speech models to assess ORF may be feasible to implement and scale in lower-resource, linguistically diverse educational contexts

Our Children Have a Right to Read! Increasing Literacy Skills in the Early Primary Grades in Developing Contexts: A Case Study of a Rights-Based Initiative in Cross River State, Nigeria.

This thesis adds to the sparse body of knowledge concerning how to increase early grade literacy levels in the Global South through evaluating how, if at all, a rights-based intervention has helped to do so in Cross River State, Nigeria. It presents a critical realist case study, containing a broad range of both quantitative and qualitative data, including insider-participant observations. Overall, it was found that this particular rights-based intervention produced significant and often large increases in pupils’ literacy skills, but that the extent of the impact varied according to the contextual conditions, again showing that context mattered. The key finding presented is that teachers have been fundamental to determining the impact, mainly because of how frequently they have been choosing to implement the intervention teaching method. The thesis postulates that the frequency of teachers’ implementation was determined by incentive mechanisms, a social reciprocity mechanism and an informal social control mechanism. Through identifying these social mechanisms, and through the mixed methods, insider-researcher methodology, the thesis is able to provide a deep understanding of incentives, motivations and relationships, and so how and why context mattered, adding to discussions on providing a “good-fit”. In doing so, the thesis highlights how the specific rights-based approach needs to now provide a greater merging of an outcomes and a processes approach. The thesis also contributes to scholarly debates concerning whether rights-based actors should be more processes or outcomes focused, whether they should be pushing for systematic reforms or working within systems and also whether a principal-agent approach will provide a good fit. It also makes important contributions to knowledge concerning the benefits and challenges associated with insider research, as well as how critical realist philosophical assumptions can help to generate the depth that is needed to truly understand how interventions are and are not effective