Evaluation of outreach interventions for under 16 year olds. Tools and guidance for higher education providers

During 2017-18, OFFA commissioned research that aimed to understand the nature of outreach activities for under 16 year olds (which were funded through access and participation investment) and how these were evaluated. This document, developed from the research, is intended to act as a resource for pre-16 outreach practitioners and evaluators, drawing both on the data collected by this project and the wider literature around evaluation and outreach. It seeks to recognise the complexity of pre-16 outreach work and eschews a prescriptive approach in favour of establishing important principles and actions that are likely to underpin good practice. Our discussion is broadly positioned within a ‘social realist’ worldview (Archer, 2008; Pawson, 2013) that seeks to understand the fuzzy nature of the cause-and-effect relationships that exist within complex social fields, where individuals construct their own realities in reference to those around them. There is a particular focus on epistemology – the pathways to creating dependable, if contingent, knowledge – as a vehicle for making meaning from data that is usually incomplete, compromised or mediated through young people’s emergent constructions of their worlds. Fundamentally, outreach is predicated on the ability of practitioners to influence young people in a planned way, albeit that the plan will not always work for every young person in every cohort. An important element in this epistemology is that it is not concerned with finding single ‘solutions’ that exist outside time and context. Rather, it is concerned with understanding how young people are influenced by their life experiences – not ‘what works’, but what works in a given context and, importantly, why. It is only through understanding the latter element that practices can become robustly effective in the long-term and potentially transferable to other contexts. This is particularly appropriate to pre-16 outreach work due to the lengthy time lag between activity and application to higher education (HE).Office for Students (OfS

A framework for autonomic web service selection

Web services are a form of distributed computing. As applications accessible over standard internet protocols, web services allow access to disparate computational resources. Recently, with an increased commoditization of web services, there has been a greater interest in the problem of selection. If a web service client can be configured to use one of a number of different web services, which should it select? In this thesis, an approach based on examining the past quality of service (QoS) parameters of similar clients is presented. Standard web service clients are augmented to report their experiences, and can reason over both these and the experiences of others using a number of formal techniques, thereby arriving at an informed decision

Understanding the evaluation of access and participation outreach interventions for under 16 year olds

The project team was asked to address the following six research questions and these were used to guide the project: 1. What are the intended outcomes for current outreach interventions directed at under 16 year olds from disadvantaged backgrounds where the long-term aim is to widen access to higher education (HE)? 2. What types of outreach intervention activity or activities are institutions using in relation to intended outcomes? 3. What evaluation tools, methods and metrics are being used to measure the intended outcomes? 4. What are the perceived and actual challenges and barriers for different stakeholders to effective evaluation of long-term outreach? 5. What do different stakeholders consider most effective evaluation practice and why? 6. How valid and suitable are the evaluation tools, methods and metrics (identified through the research) that are commonly used? The project was constructed around six interlinked work packages: 1. A quantitative analysis of what higher education providers (HEPs) say about their pre-16 outreach activities (and their evaluation) in their 2017-18 access agreements (as the most recent available). 2. An online survey of HEPs to gather information about the pre-16 outreach activities delivered during the 2016-17 academic year and their evaluation, as well as the structure of their evaluation resources and challenges faced. 3. Case studies of four HEPs identified as demonstrating elements of good practice through their access agreements and the online survey, derived from telephone interviews with key staff and documentary analysis. 4. Telephone interviews with 11 third sector organisations (TSOs) to explore their practices and the evaluation of their activities, providing a counterpoint to the data collected from higher education institutions (HEIs). 5. A synthesis of the four preceding work packages to explore elements of good practice, determine a basis for assessing the quality of evaluations and highlight challenges for the sector and OFFA. 6. An invited participatory workshop for evaluators from HEPs and TSOs identified as demonstrating elements of good practice through the online survey and telephone interviews, to act as a sounding board for the emerging conclusions and recommendations.Office for Students (OfS