Cameron, Amy

Floyd, Ali

Hardee, Erin

Mackintosh, Ailsa

Stanley-Wall, Nicola

English

In 2018, we, the School of Life Sciences Public Engagement team, worked with Evaluation Support Scotland to create an evaluation strategy aligned to our then recently developed Public Engagement with Research Strategy. The result was ‘OurEvaluation Story’ - an evaluation framework .Over the past 5 years, we used the framework to evaluate public engagement work. Starting in 2018, we produced an annual report to look in detail at what we were doing and if it helped the School of Life Sciences community achieve our ‘indicators of success’. Five years on, we have a strong collection of evidence to show how we have achieved some of our aims and objectives and where more work is needed.We revisited the evaluation strategy, with three key questions in mind:• What changes need to be made following its use and review?• Do the outcomes reflect the revised Public Engagement with Research and Scholarship Strategy?• Is the evaluation strategy the best it can be to help measure the impact of the Public Engagement with Research and Scholarship Strategy?This document contains our new evaluation framework and recording forms

University of Dundee Online Publications

                                                                    University of DundeePublic Engagement with Research and Scholarship Evaluation Framework 2023-2028Cameron, Amy; Floyd, Ali; Hardee, Erin; Mackintosh, Ailsa; Stanley-Wall, NicolaPublication date:2023Document VersionPublisher's PDF, also known as Version of recordLink to publication in Discovery Research PortalCitation for published version (APA):Cameron, A., Floyd, A., Hardee, E., Mackintosh, A., & Stanley-Wall, N. (2023). Public Engagement withResearch and Scholarship Evaluation Framework 2023-2028. University of Dundee.General rightsCopyright and moral rights for the publications made accessible in Discovery Research Portal are retained by the authors and/or othercopyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated withthese rights. • Users may download and print one copy of any publication from Discovery Research Portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain. • You may freely distribute the URL identifying the publication in the public portal.Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Download date: 12. Oct. 20231   School of Life Sciences University of Dundee  Public Engagement with Research and Scholarship Evaluation Framework  2023-2028     Amy Cameron, Ali Floyd, Erin Hardee, Alisa Mackintosh, and Nicola Stanley-Wall         Contact us: SLS_PublicEngagement@dundee.ac.uk   2  Contents  Forward .............................................................................................. 3 A New Logic Model for 2023-2028 ................................................. 3 New outcomes and indicators ........................................................ 3 Conclusion......................................................................................... 5 Appendix 1: Logic Model for 2023-2028 ........................................ 6 Appendix 2: New Outcomes and Indicators .................................. 7 Appendix 3: Evaluation Report Template ...................................... 9 References ...................................................................................... 13   Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  3  Forward  In 2018, we, the School of Life Sciences Public Engagement team, worked with Evaluation Support Scotland to create an evaluation strategy aligned to our then recently developed Public Engagement with Research Strategy1. The result was ‘Our Evaluation Story’ - an evaluation framework 2.  Over the past 5 years, we used the framework to evaluate public engagement work. Starting in 2018, we produced an annual report to look in detail at what we were doing and if it helped the School of Life Sciences community achieve our ‘indicators of success’ 3,4,5,6. Five years on, we have a strong collection of evidence to show how we have achieved some of our aims and objectives and where more work is needed7. We revisited the evaluation strategy, with three key questions in mind: • What changes need to be made following its use and review? • Do the outcomes reflect the revised Public Engagement with Research and Scholarship Strategy8? • Is the evaluation strategy the best it can be to help measure the impact of the Public Engagement with Research and Scholarship Strategy?   This document contains our new evaluation framework and recording forms.  Logic Model for 2023-2028 We separated out the three groups for whom there were outcomes. These are: • the public, including teachers and pupils • our staff and students • the School of Life Sciences public engagement team Each group has different priorities for taking part in engagement. We updated the outputs and outcomes along with the assumptions we made for each group. This made it easier to see the contributions of each group to the evaluation framework. Each group’s individual outcomes then fed into our long-term outcomes that align with the five main aims of our School of Life Sciences Public Engagement with Research and Scholarship Strategy8. The revised logic model is provided in Appendix 1.  New outcomes and indicators As stated, some of our existing outcomes and indicators no longer aligned with our updated aims, and so they were updated.  Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  4  Outcomes for public We wanted to reflect our growing focus on public and patient involvement, so added a new outcome, ‘Greater involvement in science’, and several indicators. This includes 'I feel this activity reflects my needs/the needs of my community' and 'I feel my voice has been heard'. This will help us remember to work with our audiences in the development and delivery of our activities and to ensure that we are listening to them throughout. We removed the outcomes around new skills and pupil attainment which, while worthy aims, require more resources and staffing than we are likely to be able to provide going forward. This doesn't mean we won't be contributing to activities that up-skill teachers and students, helping them with their science learning, but it means we can use what resources we have more effectively. Outcomes for staff and students We added an outcome around increased embedding within career pathways. This has always been an important part of our work and we realised we had been collecting this information for years, but it wasn't reflected in our evaluation strategy. Indicators placed under this outcome include the number of staff and students getting nominations and awards for their public engagement work, and the number of public engagement grants our researchers apply for and achieve in the course of their work. We updated the outcome ‘New perspective on topic in question’ to capture the mutual benefits of public engagement by including the publics’ impact on our staff and students. We changed the outcome to ‘Impact of the public upon research’ to reflect this and added an indicator ‘The public have influenced my research process’ which also considers our expanding public and patient involvement work.  We removed the outcome about increased access to useful resources, as we realised this is more appropriate in the public engagement team section of the evaluation strategy. We will still be measuring it, but the focus will be on what we as a team are providing for our life sciences community. For the same reason, we also moved an indicator, ‘Number of people accessing training and resources’ from the wider participation in PE activities outcome within the PE team section to the more appropriate ‘our staff and students have increased PE skills’ outcome. Outcomes for the School of Life Sciences public engagement team We added a section around the internal audiences we want to reach and the increased embedding of public engagement in research culture. We included indicators around the demographics of staff and students doing public engagement, the number of requests for our expertise, and the appropriate inclusion of public engagement in operational processes. This gives us a push to monitor how we are feeding into research culture and to assess our effectiveness in supporting researchers and students in their development.  We also specified what our aims for external audiences were, removing some indicators that were not fit for purpose and adding ‘I have reflected on the best way of reaching my desired audience’ to reinforce that our audience needs come first. The revised list is provided in Appendix 2.  Methods of measuring We revised our methods of measuring to reflect our increased online engagement work. These include and are not restricted to: Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  5  • Questionnaires • Observation • Interactive Feedback • Interview • Internal Records • External Endorsement • Online Analytics Collecting Evaluation  We updated the evaluation report template to reflect the changes to our outcomes and indicators. We also added additional points such as adding entries for public engagement events and research outputs into Discovery, the University of Dundee's online research repository, and photo file links to ensure these aspects are completed as part of the evaluation report process. The revised report template is provided in Appendix 3. Conclusion Our revised evaluation strategy is clearly aligned with our new Public Engagement with Scholarship and Research Strategy8. This will ensure we can measure both the work we currently are doing and the work we want to do over the next five years. This will also help us document and assess our progress effectively, ensuring the aims of our Public Engagement with Scholarship and Research Strategy8 are met while continuing to embed public engagement within the School of Life Sciences and the wider university for years to come. Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  6  Appendix 1: Logic Model for 2023-2028   Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  7  Appendix 2: New Outcomes and Indicators  Outcomes for public Greater understanding of science • Number of aims and objectives adequately achieved • “I can explain [activity]” • “I understand the world of science” • “I understand the methodology of science” Greater enthusiasm for science • “I feel enthusiastic about science” • Number of events that individuals come to • “I share what I have learned” Greater involvement in science • “I felt involved in the development of [activity]” • “I feel this activity reflects my needs/the needs of my community” • “I feel my voice has been heard” • Involvement in science activities/pathways  Outcomes for staff and students Staff and students have increased public engagement skills • “I can run a public engagement event safely” • Number of post-event evaluations and reflections • Number of people accessing training and resources • “I can explain my ideas in simple language” Staff and students have greater enthusiasm for public engagement • “I feel confident to do more of this” • Number of staff participating/repeat participation • Variety of activities and novel approaches Impact of the public upon research • “I understand the public perspective on my topic” • “The public have influenced my research process” • Number of new collaborations Increased embedding within career pathways • Number of staff and students getting nominations and awards • Number of grants applied for/achieved • “Doing public engagement is part of my responsibilities as a member of the School of Life Sciences”   Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  8  Outcomes for School of Life Sciences public engagement team More meaningful public engagement activities • “I feel this activity reflects the research taking place within the University” • Activities have explicit aims and objectives • Number of uses of each resource, idea, or activity • Consultation and development with audience or partner groups More productive partnerships • Length of time for partnership and number of projects/activities produced • Partnership agreement exists • “I feel all partner goals are being worked towards and there is good communication in place” • Instances of sharing outcomes externally External audiences reached • Demographics of public engaged with • “I can see myself reflected in those participating” • “I have reflected on the best way of reaching my desired audience” Internal audiences and university culture • Demographics of staff and students doing public engagement • Number of requests for public engagement expertise • Staff aware of support and resources available • Appropriate inclusion of public engagement in university operational processes   Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  9  Appendix 3: Evaluation Report Template [Project name and date] Evaluation Report  Summary sentence: [a single sentence about the project] Number of public participants/demographics: [Who took part? Could include ages, gender balance, where they’re from, etc] Number of staff/demographics: [Departments, numbers, career stages – as much detail as possible] Type of activity:  List of partners involved: Planned Outcomes:  These can include things like learning, behavioural, and attitudinal outcomes. They should also reference whether they are outcomes for public (inc. teachers and pupils), our staff & students, and/or the PE team. They should be clearly defined as they will be what’s used to set up the feedback system. There are lots of ways of defining these; Bloom’s taxonomy is one useful tool for thinking about what you really want people to do – learn, consider, list, state, etc.  Evaluation methods used: Questionnaires, observations, post-its – list all  Links to Discovery entry: • Activity: • Research Outputs:  Link to images folder: • Photo consents stored:    Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  10  Outcomes Evidence Learning/Action Points For public (including teachers & pupils)   Greater understanding of science  Number of aims and objectives adequately achieved • “I can explain [activity]” • “I understand the world of science” • “I understand the methodology of science”   Greater enthusiasm for science  • “I feel enthusiastic about science”   • No. of events that individuals come to  • “I share what I have learned”     Greater involvement in science “I felt involved in the development of [activity]” • “I feel this activity reflects my needs/the needs of my community” • “I feel my voice has been heard” • Involvement in science activities/pathways          Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  11  Outcomes Evidence Learning/Action Points For staff and students   Staff and students have increased public engagement skills   • “I can run a public engagement event safely” • Number of post-event evaluations and reflections • Number of people accessing training and resources • “I can explain my ideas in simple language”     Staff and students have greater enthusiasm for PE   • “I feel confident to do more of this” • Number of staff participating/repeat participation • Variety of activities and novel approaches     New perspective on topic in question   • “I understand the public perspective on my topic”  • No. of new collaborations/partnerships    Increased embedding within career pathways • Number of staff and students getting nominations and awards • Number of grants applied for/achieved • “Doing PE is part of my responsibilities as a member of SLS”      Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  12  Outcomes Evidence Learning/Action Points For PE team   More meaningful PE activities   • “I feel this activity reflects the research taking place within the University”   • Activities have explicit learning outcomes   • Number of uses of each resource, idea or activity  • Consultation and development with audience or partner groups     More productive partnerships   • Length of time for partnership and number of projects/activities produced  • Partnership agreement exists  • “I feel all partner goals are being worked towards and there is good communication in place”  • Instances of sharing outcomes externally     External audiences reached • Demographics of public engaged with • “I can see myself reflected in those participating” • Number of events individuals attend • “I have reflected on the best way of reaching my desired audience”   Internal audiences and university culture • Demographics of staff and students doing PE • Number of requests for PE expertise • Staff aware of support and resources available • Appropriate inclusion of PE in university operation processes   Further Information [Capture any other evidence, add here: e.g., social media links, articles]   Public Engagement with Research and Scholarship Evaluation Framework - 2023-2028  13  References  1. Stanley-Wall, N, Cameron, A, Hardee, E & Martin, M 2020, University of Dundee, School of Life Sciences: Public Engagement with Research Strategy. University of Dundee, Dundee. https://doi.org/10.20933/100001156 2. Hardee, E, Cameron, A, Floyd, A & Kennedy, D 2020, University of Dundee, School of Life Sciences: Our Evaluation Story. University of Dundee, Dundee. https://doi.org/10.20933/100001155  3. Hardee, E, Cameron, A, Floyd, A, Stanley-Wall, N, Post, J & Stephenson, E 2019, University of Dundee, School of Life Sciences, Annual Report 2019: Public Engagement. University of Dundee, University of Dundee. https://doi.org/10.20933/100001134  4. Stanley-Wall, N, Hardee, E, Cameron, A, Floyd, A, Mackintosh, A & MacGowan, D 2020, University of Dundee, School of Life Sciences, Annual Report 2020: Public Engagement. University of Dundee, Dundee. 5. Mackintosh, A, Floyd, A, Cameron, A, Hardee, E, Stanley-Wall, N & Quinn, E 2021, University of Dundee, School of Life Sciences, Annual Report 2021: Public Engagement. University of Dundee, Dundee, UK. 6. Mackintosh, A, Floyd, A, Cameron, A, Hardee, E, Stanley-Wall, N & Quinn, E 2022, University of Dundee, School of Life Sciences, Annual Report 2022: Public Engagement. University of Dundee, Dundee, UK. 7. Mackintosh, A, Floyd, A, Cameron, A, Hardee, E, Stanley-Wall, N & Quinn, E School of Life Sciences Using an Evaluation Framework to Direct Public Engagement Work 2017-2022 with Case Studies. DOIs: 10.20933/100001290 8. Hardee, E, Cameron, A & Stanley-Wall, N 2023, Public Engagement with scholarship and research strategy. University of Dundee, Dundee. https://doi.org/10.20933/100001280   

Public Engagement with Research and Scholarship Evaluation Framework 2023-2028

https://discovery.dundee.ac.uk/files/108323783/Public_Engagement_Evaluation_Framwork_2023_FINAL.pdf

Public Engagement with Research and Scholarship Evaluation Framework 2023-2028

Abstract

Similar works

Full text

Available Versions

University of Dundee Online Publications