Bennett, Ellen

Dayson, Christopher

Pearson, Sarah

English

Sheffield Hallam University Research Archive

Evaluation of the Early Action Neighbourhood Fund: Learning Summary 1 - Data, Evidence and ImpactDAYSON, Christopher <http://orcid.org/0000-0003-2402-1183>, PEARSON, Sarah <http://orcid.org/0000-0001-5049-5396> and BENNETT, Ellen <http://orcid.org/0000-0003-3271-8757>Available from Sheffield Hallam University Research Archive (SHURA) at:http://shura.shu.ac.uk/25776/This document is the author deposited version.  You are advised to consult the publisher's version if you wish to cite from it.Published versionDAYSON, Christopher, PEARSON, Sarah and BENNETT, Ellen (2019). Evaluation of the Early Action Neighbourhood Fund: Learning Summary 1 - Data, Evidence and Impact. Project Report. Centre for Regional Economic and Social Research, Sheffield Hallam University. Copyright and re-use policySee http://shura.shu.ac.uk/information.htmlSheffield Hallam University Research Archivehttp://shura.shu.ac.ukEvaluation of the Early Action Neighbourhood Fund: Learning Summary 1Data, Evidence and Impactwww.shu.ac.uk/research/cresrNovember 2019Chris Dayson, Sarah Pearson and Ellen Bennett CRESR, Sheffield Hallam UniversitySummaryThis short report outlines what we have learnt from the Evaluation of the Early Action Neighbourhood Fund (EANF) in terms data, evidence and impact. The key points are as follows: » Funding for a designated evaluation lead enabled the projects to embed an evidence-based and data-led approach to evaluation and learning in their work from the start and throughout. » All three projects have faced challenges accessing data held by public bodies to help demonstrate the impact of their work on public sector priorities. When this data has been ‘unlocked’ it has required senior officials within the relevant public service to authorise and prioritise its release. » When the projects have been able to access data held by public bodies it has proved a powerful accompaniment to project level data for demonstrating a project’s impact on public sector priorities. » Good quality data and evidence, from a range of sources, provides projects with credibility when advocating on behalf of communities with public sector stakeholders. » Attempting to directly attribute impacts to projects is unwise when operating within complex service systems. It is better to focus on developing plausible evidence and data informed accounts of a project’s contribution to outcomes instead. » Approaches to data, evidence and impact should be responsive and adapt to the priorities and needs of a project and its key stakeholders on an ongoing basis. » This learning will be applicable to other early action projects and their funders, and ought to inform how similar projects approach evaluation in the future. 2IntroductionAs part of our Evaluation of the Early Action Neighbourhood Fund (EANF) we have worked closely with each of the three funded projects to support the development and implementation of local evaluation plans and help them think through how they approach data, evidence and impact. This short report draws on as series of interviews and workshops undertaken with the three projects over the past four years to discuss the key learning that has emerged from this work. We present this learning under the following themes:1. The importance of data and evidence for early action projects.2. The challenge of attributing outcomes and impact to early action projects.3. The need to review and adjust early action project’s evaluation expectations and focus on an ongoing basis.We also discuss the main implications of this learning for other early action projects and their funders.Learning Theme 1: The importance of data and evidence An evidence review undertaken at the beginning of the evaluation and a learning report on Data and Evidence published at the end of the first year both highlighted the importance of data on and need to build a more robust evidence base about the outcomes and impact of early action. All three of the projects have embedded evaluation in their work and some key lessons have emerged from their experience.Having a designated evaluation leadTwo of the three EANF projects (Norwich and Hartlepool) have had a dedicated evaluation lead from beginning of the programme whilst the third project (Coventry) sought additional funding to recruit one mid-way through. The focus was of these roles was generating data and evidence to demonstrate the impact of the work and support learning and development within the project over a five-year period. Both the Norwich and Hartlepool projects have highlighted the importance of having a dedicated post funded through the programme in enabling them to maintain a focus on data and evidence alongside project delivery. In fact, in Norwich they are trying to embed this approach across the organisation, factoring in the costs when developing new projects.The importance of data held by public sector bodiesAll three projects had identified certain specific data held by public sector bodies as central to their evaluation plans (Table 1): it was through these data that they intended to demonstrate the impact of their work on public sector priorities, including preventable need and savings to the public purse. However, in practice each project has faced considerable challenges accessing the public sector data they require. In the few examples where it has been possible to access these data it has followed protracted negotiations and only been made possible after achieving ‘buy-in’ from a senior official within the relevant public body with the authority to prioritise the release of data.Table 1: Project level examples of public sector data requirements3When projects have been able to access these data they have been able to use it to present powerful evidence about the impact of their work. For example, in Norwich they have been able access pupil level data on attendance, behaviour and attainment to demonstrate improvements for pupils accessing direct interventions such as counselling. Similarly, in Coventry the project has been able to access data from children’s services which highlight an increase in the numbers of families accessing early help in the communities in which the project is working. This is being used to open-up conversations with service providers about the impact of the early action approach in these areas and what the lessons might be for mainstream provision. Although these data didn’t necessarily meet the highest ‘standards’ of evidence – for example there were no readily available comparison groups - the fact that it came from within the ‘system’ provided a degree of perceived validity and importance from the perspective of public sector stakeholders compared with other data collected at a project level.The importance of data informed discussions with key stakeholdersHaving a data lead, and significant amounts of data and analysis about the project, has enabled the projects to engage in data informed discussions with key stakeholders about needs and outcomes. This knowledge and understanding of individuals and communities provided the projects with added credibility, even legitimacy, when advocating on behalf of their clients and broader communities.Learning Theme 2: The challenge of attributing impact - graphAll three projects started-out with well thought through but ambitious plans about how they would evidence the impact of their work. This included the development of comparison groups of people with similar characteristics but not directly involved with the projects. However, in practice the projects have struggled to collect comparative data, and this has made it difficult for them to directly attribute long term outcomes, including reductions in preventable need and associated savings to the public purse, to their work.Box 1: Challenges collecting comparative dataDuring a learning session in 2017 we explored the potential of theory-based ‘small n’ approaches such as contribution analysis (box 2) as an alternative way to understand and attribute impact. The discus-sion concluded that these types of approach, and the principles that underpin them, could be helpful for the evaluation of early action projects when: » it is not possible collect large amounts of primary quantitative data; » comparator data is not available; » an intervention is being delivered in a complex setting or system; » evaluation resources are limited.It was concluded that a key advantage of small n approaches was their focus on developing plausible theories about how and why an approach works, and to use this as the basis for telling an evi-dence-based ‘story’ about the role of specific inter-ventions and activities in bringing about the desired change. » Accessing public service data on non-service users: Information Governance guidelines and GDPR regulations typically require explicit consent from individuals to access and share sensitive data. Obtaining the consent required from non-service users is rarely in the gift of early action projects. » Collecting primary data on non-service users: early action projects will struggle to identify non-service users from whom they can collect primary data. Even if this is possible it is unlikely that that early action projects will have the resources necessary to undertake extensive data collection.4Learning Theme 3: Reviewing and adjusting evaluation expectations and focus - clipboard/person thinkingDespite having made detailed plans at beginning of the programme about the outcomes they would measure and their approaches to evidencing these, each of the projects has regularly reviewed and adjusted their approach as they have progressed. These changes have been made in response to a number of factors including the priorities of the pro-ject and its stakeholders at a particular point in time, the availability of data, and opportunities to engage with wider research projects.Importantly, each of the projects has revised their expectations about what they will be able to evi-dence in terms of reducing preventable need. For example, the Coventry and Hartlepool projects had both initially identified reducing problematic looked after children (LAC) numbers in their area as a potential impact of their work but have since reflected that this may not be achievable, particular-ly in the short term, and revised their expectations accordingly. This is because a range of factors which are beyond the reach and scope of individual early action projects affect an area’s LAC numbers. For example, changes in Local Authority Children’s Services approach to assessing the risk of Children In Need, and cuts to preventative services such as Children’s Centres, which means families are una-ble to access much needed support. The learning from the EANF projects suggest that evaluation measures that focus on changes to cultures, values and practices within local public services which with which an early action project engages may be more relevant and realistic indica-tors of change in the short to medium term, not least because these are directly responsive to the work of the projects. However, exploring the relationships between these system-level changes, and those which improve outcomes for beneficiaries is a more challenging as the causal linkages and pathways are often complex and unclear.Implications for other early action projects - resultsThe learning from the EANF evaluation about data, evidence and impact will be applicable other early action projects and ought to inform how they ap-proach evaluation in the future. Each of the lessons presented in this report ought to resonate with frontline organisations delivering early action and funders wishing to support work in this field, but we highlight the following points as being most impor-tant.1.  Allocating sufficient resources for evaluation: doing evaluation well is a resource-intensive undertaking that requires specific data collection and analysis skills. If early action Box 2: A brief overview of amall n approaches and contribution analysisFor more information see our earlier learning report on this topic.5projects want to evaluate their processes and impacts effectively then they and their funders must ensure that a dedicated and skilled resource is in place to do so.2. Ensuring good quality project level data collection: early action projects and their funders may not be able to control the availability of or access to public service data but they can ensure that good quality project level data is collected. This should include quantitative data on who has accessed the project and what the benefits are alongside qualitative data on how and why these changes have occurred. This type of evidence will provide a sound base upon which to build an early action project’s contribution story. 3.  Using evidence for learning and development: early action projects should ensure that their evaluation evidence is used to inform practice on an ongoing basis. Internally (within projects), it can help teams reflect on and revise their practice by understanding what works, and what doesn’t, in different contexts. Externally (with stakeholders), it can help tell a story about the contribution early action makes to different public service priorities, which can enable project level learning to be transferred to other parts of public service systems.4. Flexible, adaptable and responsive evaluation strategies: early action projects should adopt an approach to evaluation that is able to change in response to the ever-shifting requirements and contexts of the public service systems in which they are operating, and funders should support and promote this way of working.

Evaluation of the Early Action Neighbourhood Fund: Learning Summary 1 - Data, Evidence and Impact

http://shura.shu.ac.uk/25776/1/EANF-report-2019-data-evidence-impact.pdf

Evaluation of the Early Action Neighbourhood Fund: Learning Summary 1 - Data, Evidence and Impact

Abstract

Similar works

Full text

Available Versions

Sheffield Hallam University Research Archive