This briefing document is based on a review of 376 examples of widening participation research published in peer reviewed journals. The document sets out the ‘state of the art’ of this literature in terms of trends and methodological focus. The document then highlights concerns amongst the research community in respect of gaps in knowledge and limitations in current widening participation and outreach research

Dytham, Siobhan

Hughes, Christina

Warwick Research Archives Portal Repository

     warwick.ac.uk/lib-publications      Original citation: Dytham, Siobhan and Hughes, Christina (2017) Widening participation research and evaluation : where are we now? Discussion Paper. Coventry, UK: University of Warwick.  Permanent WRAP URL: http://wrap.warwick.ac.uk/88772       Copyright and reuse: The Warwick Research Archive Portal (WRAP) makes this work by researchers of the University of Warwick available open access under the following conditions.  Copyright © and all moral rights to the version of the paper presented here belong to the individual author(s) and/or other copyright owners.  To the extent reasonable and practicable the material made available in WRAP has been checked for eligibility before being made available.  Copies of full items can be used for personal research or study, educational, or not-for-profit purposes without prior permission or charge.  Provided that the authors, title and full bibliographic details are credited, a hyperlink and/or URL is given for the original metadata page and the content is not changed in any way.  A note on versions: The version presented here is a working paper or pre-print that may be later published elsewhere.  If a published version is known of, the above WRAP url will contain details on finding it.  For more information, please contact the WRAP Team at: wrap@warwick.ac.uk    Widening Participation Research and Evaluation Where are we now?    Briefing Document for the OFFA-Warwick Symposium Evaluating Outreach Dr Siobhan Dytham and Professor Christina Hughes June 2017   Introduction Reay et al. (2005: 106) argue that widening participation is both ‘under-researched and under-theorized’. This is a common assertion in the literature and Kettley (2007: 343) claims that as a result ‘little debate has occurred about the content and qualities of a powerful approach to widening participation’. In consequence, it is commonly noted that a lack of systematic and rigorous research affects the quality of knowledge produced as well as the potential to influence policy. In terms of widening participation, concerns are also raised in relation to interpreting policy and creating appropriate initiatives, as goals are interpreted and prioritised in many different ways (Doyle and Griffin, 2012). Indeed, ‘the whole area needs to be understood more fully, defined more clearly and for policy and interventions to be more effective they need to be informed by a range of high quality research involving an inclusive methodological vision and a rich theoretical tapestry’ (Doyle and Griffin, 2012: 86).   Widening Participation Literature Review This briefing document is based on a review of 376 examples of widening participation research published in peer reviewed journals. A search was conducted using the keyword ‘Widening Participation’ in the following databases - Education Research Complete, Scopus, Sociological Abstracts, Social Science Citation Index, Web of Science, EconLit, Ingenta Online Journals and the University of Warwick search tool ‘Encore’. After removing all duplicates and any irrelevant literature, 376 articles remained. These were then coded in Nvivo for analysis.  This briefing document sets out the ‘state of the art’ of this literature in terms of trends and methodological focus. The document then highlights concerns amongst the research community in respect to gaps in knowledge and limitations in current widening participation and outreach research.    Key Foci of Widening Participation Research Variables: The most common variables drawn on in research on widening participation are, in rank order: 1. Education (e.g. type of qualifications, or type of previous education student received).  2. Socio-economic status.  3. Age (which mostly focused on mature students).  4. Gender 5. Class 6. Ethnicity  7. Teaching/Pedagogy 8. Finance/funding   Educational Stage: The articles were coded in terms of which ‘educational stage’ the research focused on i.e. either research carried out with student before they enter university, during their time at university, after university, or research focusing on access to something other than university, namely further education or work based learning/training.   o 49% of the research focused on ‘Before University’  o 24% focused on ‘During University’  o 9% of the literature focussed on access to further education, work based learning/training or barriers and inequalities experienced during GCSEs.  o 6% of the research focused on ‘After University’   Research Methods: The articles were also coded in terms of the research methods used – qualitative, quantitative, or mixed methods. It was found that almost half of the articles used qualitative methods.   o 49% qualitative methods; o 29% used mixed methods;  o 22% quantitative methods.   Key Concerns and Gaps in Knowledge From reading literature discussing widening participation and outreach research, it is clear that a number of concerns are regularly highlighted. These are:  Lack of longitudinal research, meaning that it is difficult to know the long-term impact of outreach initiatives, or to fully appreciate widening participation as a broader, longer term problem. Research focused mostly on the stages closer to university can miss potential issues in primary school, or even earlier. There is a call in the literature for a more ‘holistic’ approach, as there has been significant focus on entry into higher education but less consideration of students moving through and out of HE (Kettley, 2007). As such, more longitudinal and mixed methods approaches are required to address this problem.   Much of the ‘Programme Evaluation’ research has been considered to be of poor quality, context specific or lacking in generalisability. Doyle and Griffin (2012: 85) indicate that ‘there is a need for research instruments that allow for analysis of single interventions where they can be isolated and traced, but that also facilitate comparison, and see such interventions as cumulative and contributions’.  Methodological divides: In the widening participation research more generally, there seems to be some tension and separation between different approaches. For example, Gorard and Smith (2006) are critical of the potential contribution of qualitative research. Equally, Baker et al. (2006) describe  the widening participation literature as falling into two ‘camps’ and are critical of what they term the ‘Empowerment Literature’.   Lack of comparative approaches: Kettley (2007) argues that many of the studies which make claims about certain groups in relation to widening participation, do so without considering or comparing them to alternative groups. As such claims are made based on, for example, the discourses and experiences of working class students about the barriers they experience in terms of HE without checking whether these discourses or experiences are also found when speaking to middle-class students or those from ‘elite’ backgrounds. Without comparison groups it is impossible to know which experiences and barriers are specific to certain groups and which may be broader issues which affect multiple groups of students (Gorard and Smith, 2006).   Overly strong focus on barriers: Kettley (2007: 343) also argues that ‘widening participation research must move away from the underdeveloped concept of barriers’, arguing that consideration should also be given to bridges, and that these should not be considered as discrete entities.   Focus on Socio-Economic variable: Research is predominantly focused on social class and socio-economic status, potentially creating a lack of focus on other aspects or intersectional approaches (Shaw, 2009).   Lack of conceptual clarity on diversity: Multiple perspectives and understandings of ‘diversity’ exist and ideas about what diversity is to be achieved can vary greatly in the literature, and as such it can be difficult to make cross study comparisons (Shaw, 2009). Kettley (2007) argues that future research must establish its intellectual context, as this has been omitted or under-developed in previous research which has been a weakness of the field.   Gap in evaluating outreach: Doyle and Griffin (2012), when evaluating AimHigher, highlighted a number of limitations. In general, it is acknowledged in the literature that longer term tracking of students has been ineffectual or absent, meaning that the outcomes for students are unknown and therefore the actual impact of these types of initiatives cannot be accurately assessed.   Effectiveness of initiatives/interventions: An important issue highlighted in the literature is the difficulty in assessing whether certain initiatives have been affective. As there are a very large number of factors involved in any student participating or not participating, it is difficult and potentially impossible to attribute cause and effect to specific interventions (Doyle and Griffin, 2012). An additional problem is that much of the data collected in relation to these types of initiatives and programmes are local and fairly context specific, so it is not clear how robust, reliable or generalizable this research is. This means that knowledge about the success or impact of initiatives is often partial and it is unclear whether findings from these studies can offer insights for future initiatives in different areas or contexts.   Targeting: Doyle and Griffin (2012) argue that in the case of AimHigher, initiatives were too often targeted incorrectly. Some groups, such as certain ethnicity groups, had low levels of involvement with the schemes. Also, it is suggested that in general the initiatives often focused on students who  would have gone to HE anyway. However, one area where targeting has been more successful is in relation to disabled students, where residential visits can be particularly valuable.   Area based approaches: Harrison and Hatt (2010) highlight the difficulties and problems in using area-based approaches for targeting, arguing that these approaches often include many students who were not intended for targeting (and as such, including them the initiative is giving already advantaged students further advantage) and missing many very disadvantaged students who live in other areas. Equally, aside from these issues, they raise the point of whether targeting the most extreme cases of deprivation for short term projects is the best use of resources, as short term projects in these areas are likely to only have little or a short term impact, whereas the same project could have greater or longer term impacts in areas which are not as deprived. As an example of this, Allen (2010) targeted the most deprived communities and as such ‘the profile of students indicated that they were less HE ready, had less previous education and at lower levels, were more likely to be out of paid work for some time or were from ethnic minority groups. Despite the provision of additional support, this was demonstrated in slightly lower completion and achievement rates’ (Allen, 2010: 150-1). However, as Harrison and Hatt (2010) suggest, despite this, there are social justice reasons for continuing to target these areas. Allen (2010: 152) states that ‘the Open University has made a long-term commitment to work in these communities and accepts that the work is inevitably resource intensive. It remains to be seen if such a programme, driven by a social justice mission yet shaped by the funding context, can survive in an increasingly austere financial climate’.  Evaluation: there is usually more focus on methods rather than theory in relation to evaluation (Smith, 2012). Programme evaluation is dominated by a pragmatic perspective meaning that programme evaluation theory currently lags behind programme evaluation practice (Stufflebeam and Coryn, 2014). As programme evaluation practice lacks unified theory, each variety of evaluation is based upon separately developed evaluation theory (Stufflebeam and Coryn, 2014).    Evaluation Theory There are a number of different theoretical approaches which underpin evaluation frameworks. Here we give a brief summary of some of the most common approaches underpinning different types of widening participation and outreach evaluation.   Experimental Perspective: Regarded as the ‘classical’ form of evaluation (Pawson and Tilley, 1997). This perspective emphasises quantitative measurement and the objective determination of the gap between pre-determined aims and actual outcomes (Scriven, 2008). However, although claiming objectivity, these approaches still rely on the subjective judgement of evaluators to judge what outcomes are desired and what measurements count as ‘successful’.   Pragmatic Approaches: Based upon integrating quantitative and qualitative measurement. As such, this approach is more inclusive of objective and more subjective evaluation (Alkin, 1990). However,  pragmatic evaluation is still in the main concerned with using methods that enable the evaluator to understand a detached reality in some objective manner.  Realist Approaches: Ontologically, realist evaluation adopts the perspective that reality exists independently of the actions and perceptions of the observer. Realist approaches generally have an explanatory focus i.e. ‘what works for whom, in what circumstances and why?’ These approaches focus on causality whereby mechanisms, mediated by context, explain outcomes (Pawson and Manzano-Santaella, 2012). Realist approaches can adopt mixed methods, although quantitative methods are generally primary. As Pawson and Manzano-Santaella (2012: 182) suggest, from this perspective, ‘mining mechanisms requires qualitative evidence, observing outcomes is quantitative’.  Critique of constructionism and postmodernism: Whilst recognising that a wide range of approaches are used in contemporary evaluation, Weiss (1998a) states that she regards ‘constructionism’ and ‘post-modernism’ as threats to the essential function of evaluation. Equally, Stufflebeam and Coryn (2014) argue that only systematic and rigorous evaluation can provide sufficient validity and reliability and despite the expansion of methods and approaches in evaluation, the key remains being evidence based.  Critical Realist Approaches: Jennings (2013) argues that a critical realist approach is superior in appreciating and reflecting the influence of multiple mechanisms on outputs and multiple influences on, or interconnections within, mechanisms. With a general focus on causality, realist approaches focus on does X produce Y. Critical realism concerns its self with the process by which X does or does not produce Y.   This approach does not question the existence of reality independent of an observer, but acknowledges that understanding reality is not independent of an observer (Delaney, 1999). As Maxwell (2009: 108) explains, ‘the essential characteristic of critical realism is that it combines ontological realism with epistemological constructivism in a productive, if apparently inconsistent ‘constellation’ of positions’. Therefore, this approach removes the tension between objectivism and subjectivism in experimental, pragmatic, and realist approaches. Critical realism also embraces methodological pluralism meaning that it is well suited to mixed methods rather than single method approaches.   Critical realism’s weaknesses concern difficulties in amalgamation in meta-analysis, difficulties in defining widely-accepted criteria to judge quality, implementation (because of a lack of clear empirical methodology in light of the centrality of methodological pluralism), and understanding/acceptance by participants (Jennings, 2013).       Bibliography Alkin, M.C. (1990) Debates on Evaluation, Thousand Oaks, USA: Sage Publications.   Allen, M. (2010) ‘In defence of targeting? The Open University’s Community Partnerships Programme’, Widening Participation and Lifelong Learning, 12(Special Issue), 138-53.  Baker, S., Brown, B. and Fazey, J.A. (2006) ‘Individualization in the widening participation debate’, London Review of Education, 4(2), 169-82.  Delaney, C.F. (1999) Critical Realism, In: Audi, R. (ed.) The Cambridge Dictionary of Philosophy, (2nd ed.), New York: Cambridge University Press.  Doyle, M. and Griffin, M. (2012) ‘Raised aspirations and attainment? A review of the impact of AimHigher (2004-2011) on widening participation in higher education in England’, London Review of Education, 10(1), 75-88.  Gorard, S. and Smith, E. (2006) ‘Beyond the ‘learning society’: what have we learnt from widening participation research?’, International Journal of Lifelong Education, 25(6), 575-94.  Harrison, N. and Hatt, S. (2010) ‘‘Disadvantaged Learners’: Who Are We Targeting? Understanding the Targeting of Widening Participation Activity in the United Kingdom Using Geo-Demographic Data From Southwest England’, Higher Education Quarterly, 64(1), 65-88.  Jennings, P.L. (2013) Hidden Treasures: The contribution of critical realism to evaluation. Available at: www.aston.ac.uk/EasySiteWeb/GatewayLink.aspx?alId=171885 [Accessed March 2016].   Kettley, N. (2007) ‘The Past the present and the future of widening participation research’, British Journal of Sociology of Education, 28(3), 333-47.  Maxwell, J.A. (2009) Evidence: A Critical Realist Perspective for Qualitative Research, In Denzin, N.K. and Giardina M D (eds) Qualitative Enquiry and Social Justice, USA: Left Coast Press.  Pawson, R. and Manzano-Santaella, A. (2012) ‘A Realist Diagnostic Workshop’, Evaluation, 18(2), 176-191.  Pawson, R. and Tilley, N. (1997) Realistic Evaluation, London: Sage Publications Ltd.  Reay, D., David, M.D. and Ball, S.J. (2005) Degrees of choice: Social class, race and gender in higher education, Stoke on Trent: Trentham Books.  Scriven, M. (2008) ‘A Summative Evaluation of RCT Methodology: An Alternative Approach to Causal Research’, Journal of Multi-disciplinary Evaluation, (5)9, 11-24.   Shaw, J. (2009) ‘The diversity paradox: does student diversity enhance or challenge excellence?’, Journal of Further and Higher Education, 33(4), 321-31  Smith, N.L. (2012) Foreword, In: Cousins, J.B. and Chouinard, J.A. (eds.) Participatory Evaluation Close Up: An Integration of Research-based Knowledge, USA: IAP-Information Age Publishing Inc.  Stufflebeam, D.L. and Coryn, C.L. (2014) Evaluation Theory, Models, and Applications: Research Methods for the Social Sciences, (2nd ed.), USA: Jossey Bass.  Weiss, C.H. (1998a) ‘Have We Learned Anything New About the Use of Evaluation?’, American Journal of Evaluation, 19(1), 21-33.  Weiss, C.H. (1998b) Evaluation (2nd ed.), USA: Prenice Hall.  

Widening participation research and evaluation : where are we now?

http://wrap.warwick.ac.uk/88772/8/WRAP-widening-participation-research-evaluation-Hughes-2017.pdf

Widening participation research and evaluation : where are we now?

Abstract

Similar works

Full text

Available Versions

Warwick Research Archives Portal Repository