Eadie, Michael

Grey, Anna

McCutcheon, Valerie

Robson, Sue

Stringer, Shelly Anne

English

Enlighten: Publications

 Note of Association of Research Managers and Administrators  REF Audit Compliance Meeting  
11​th​ April 2019, Glasgow. 
 
ARMA LEADS: Anna Grey; Valerie McCutcheon 
SUPPORT: Michael Eadie; Sue Robson; Shelley Anne Stringer 
Observer from Research England:  Andy Hepburn 
1. Summary 
The aim of the meeting was to examine audit requirements for Research Excellence Framework (REF). Attendees were involved in managing REF and open 
access exceptions on a day to day basis.  The meeting discussed the expectations of audit arising from Guidance on Submissions, Panel Criteria and 
Working Methods and Open Access implementation.  Outcomes of the meeting included key issues and potential solutions identified by the group. 
 
This document summarises the prioritised points rather than every detail of the discussion. 
 
The feedback will be shared with the Research England REF Team.  A further meeting is planned to discuss the new audit guidelines when they are issued. 
 
2. Background/structure 
The workshop arose from discussions in the ARMA Open Access and REF special interest groups. The agenda was split as follows: 
● A short, initial presentation which outlined the 3 stages of audit in 2014 and identified practical aspects of preparations. 
● Breakout groups which focused on Generic REF audit (including impact) and Open Access specific issues. These identified key issues for the group 
and, where possible, identified potential solutions.  
 
3. Overview of audit aspects of REF 
3.1. In 2014 there were 3 parts to audit:  
● A sample audit​ which included: proof that the members of staff were employed; sample of outputs and evidence for impact case studies. 
● Data comparisons/ reconciliations​ which included: HESA submission versus REF submission in terms of income.  
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 ● Panel instigated audit​ which included:  items raised by the panel for example ‘how much did the author contribute to a paper?’  Last time these 
were the most stressful due to the 3 day turnaround time. 
 
3.2. Everything in the REF is open to audit and a good audit trail is required. Area of focus include:  
● Staff eligibility and FTE will need to show were they employed on the census date and were they employed when the output came out? 
● Staff Circumstances: how will this link with EDAP decisions made prior to the REF 2021 submission?  
● Impact and corroborating evidences 
● Publication dates of Outputs:  the change to OA and pre-publication has complicated this.  
● Open Access status 
● Co-authorship claims – this can still be queried even if these details were not specifically requested as part of the  the submission 
● Staff connection to the unit 
3.3. Guidance on Submission and panel criterias are the primary source of information until the REF audit guidelines are issued.  The 2014 ‘grid’ OA guidance 
provides a starting point for Open access but is difficult to find online. 
 
3.4. Best practice advice​: 
● Plan for audit as part of REF preparations and capture evidence as you go along.  
● After the REF there may be staff changes/ team members disband. Build this into your planning, for information sourced from other departments 
in your University. Make sure that you know who will be answering any queries.  
● Find the people in your institution who do the HESA return. Look at how the UOAs match the cost centres that are used in HESA.  Differences may 
occur, be aware of them while preparing your submission. 
 
4. Issue gathering 
Outcomes from the breakout groups are summarised in Table 1. 
Key areas of discussion included: 
● Staff processes outlined in the codes of practice, whether the focus of audit would be process or case driven, and there was a strong preference for 
a process driven approach. 
● Recognition of the difficulties in the embedding of OA processes for changing guidance and a preference for OA audit to be investigative rather 
than punitive. 
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 ● Issues around clarifying and recording OA exceptions – in particular ‘Other Exceptions’. 
● Issues arising from the change in the GDPR landscape and from changes to HR systems – in particular, for information on staff who have left the 
institution. 
 
5. Information links 
Guidance on submissions (2019/01)  
Panel criteria and working methods (2019/02)  
Open access in the post-2014 Research Excellence Framework: information and audit requirements  
 
 
6. For further information contact 
Valerie McCutcheon ​valerie.mccutcheon@glasgow.ac.uk 
 Anna Grey ​anna.grey@york.ac.uk 
 
 
  
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 Table 1: Key issues and potential solutions identified by the breakout groups. 
AREA ISSUES Possible solutions 
Staff - Evidencing the 
Eligibility of Former Staff 
 
What is required to evidence the Eligibility of Former 
Staff? 
 
Guidance around this did not exist at the time of them 
leaving. 
 
Change of policies around GDPR so this information 
may no longer exist. 
 
Impact is a much longer time period 
 
Dealing with changes on internal systems (HR) and 
record keeping 
● Verification though previous REF/RAE submission 
● Verification through external sources such as JeS, 
ResearchFish etc. 
● Allow for system issues/failure 
● Allow if evidence has been destroyed for GDPR 
reasons 
 
Staff - connection to 
UOA 
What is required to evidence Substantial Connection. 
 
● Research England to give examples of acceptable 
evidence e.g a screenshot from websites/ 
publications affiliated with the publication 
 
Staff - employed 
elsewhere  
What evidence do we need to provide of employment 
outside HEI at the time of acceptance? 
● Include detail in FAQs 
Staff - significant 
Responsibility for 
Research 
 
 
What is required to evidence Significant Responsibility 
for Research. 
 
Will the process or decisions made and recorded on 
individuals be audited? 
 
Will a low % staff identified trigger an audit? 
 
 Correlation of HESA data. 
● HEIs to provide details of the decisions of the 
deciding body (minutes or recommendations as 
supplied to the individual). 
 
● If there is an appeal process HEIs to provide 
information on the number and outcome of 
appeals. 
 
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 ● Research England to advise on what evidence 
would demonstrate that HEIs have followed the 
process. 
● Would like audit to focus that process in Code of 
Practice has been followed, rather than queries on 
individual staff (this was an issue for a number of 
areas) 
 
Staff- Evidencing 
Research Independence 
 
What is required to evidence Research Independence? 
What records do we need to keep? 
Will they use RA records on HESA? 
To what extent is ‘academic judgement’ accepted as 
evidence? 
What about colleagues from a practice in research 
background registered for a PhD? 
● HEIs to provide details of the decisions of the 
deciding body (minutes or recommendations as 
supplied to the individual). 
 
● If there is an appeal process HEIs to provide 
information on the number and outcome of 
appeals. 
 
● Research England to advise on what evidence 
would demonstrate that HEIs have followed the 
process 
 
Staff- Decisions on 
circumstances 
 
How will decisions on circumstances 
be audited? 
 
Is there specific guidance for acquiring and retaining 
staff circumstance information 
 
 
● Accessible FAQs 
● Research England to provide example cases and 
advise on acceptable types of evidence  
● The timing of audit should be at the time of EDAP 
decisions 
● Clarify if Panels can raise a query on staff 
circumstances 
 
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 Open Access - dates 
 
How will Open Access dates be audited? 
 
What counts as ‘best efforts’ on acceptance dates? 
What if the author and publisher don’t agree? 
Variety of practices for difficult circumstance (such as 
where there is no acceptance date online/in the public 
domain record). 
What if there is a delay applying the right embargo? 
● Audit to focus on process rather than individual 
dates/cases. 
● HEIs to provide details of the process for when/if 
proxy dates are applied. 
● Research England to provide confirmation around 
difficult cases in audit guidance/ FAQs 
● Community/Research England to lobby for 
publishers to provide this data 
 
Open Access - Exception 
Ambiguity 
 
Open Access exceptions are ambiguous (especially 
‘Other’ and gold OA). 
 
Some form of template or checklist would be very 
helpful. 
 
The dates are complex.  
 
This has added to the level of OA burden . 
● Community/Research England to establish common 
understanding and use cases for exceptions, 
especially for those exceptions such as ‘Other’ and 
gold OA 
● Research England to ensure examples and guidance 
communicated openly to HEIs 
● HEIs to document their process for recording 
compliance and assigning exceptions and ensure 
this is applied consistently 
● HEIs to ensure control measures are included to 
protect against risk e.g. to protect against 
exceptions being applied inconsistently all staff 
assigning exceptions have regular training etc. 
 
Open Access - Gold 
Open Access 
Gold Open Access specific queries 
 
Was an article immediately made gold - not later or 
retrospective? 
If an article is published without a clear licence is that 
acceptable? 
● Help and examples for difficult cases in audit 
guidance/ FAQs 
● Audit the process 
● If the published version of the article is 
immediately, permanently,  free to access on the 
publisher’s website with a licence that permits 
copying and reuse this is compliant.  See clause 239 
in Guidance on Submission.  If the licence is unclear 
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 judgement is required.  If uncertain  it is possible to 
use other options for compliance or exception. 
Outputs - 
pre-print/online early 
Guidance for outputs published as pre-prints/online 
late in 2013 unclear around:  
 
● Dates especially definitions of pre-prints 
/online early. 
● Checking online early has not been entered by 
another institution 
 
● Help around difficult cases in audit guidance/ FAQs 
● Further clarify the issue of whether early on-line is 
treated the same as a pre-print 
● REF/software could  flag outputs entered previously 
by another institution 
● Clarify the implications of removal of the 
reference to early on-line in 2013, which was in 
the draft Guidance on Submission. If there was an 
early online version out in 2013 but the print 
version came out in 2014, can it be submitted? 
 
Outputs – new research What evidence required for ‘new research’  
 
 
● Guidance around difficult cases in audit guidance/ 
FAQs 
 
Impact What evidence should be held in respect of the 
additional contextual data? 
 
● Recommended template 
 
Audit - format Clarification on what audit will look like, in particular 
for new areas - OA status / staff responsibility for 
research/ independent research/ circumstances. 
 
 
● Base audit around process. 
● Include right of reply and a reasonable time for it. 
● Advise on any trigger points  
● Clarify whether panels will query areas which have 
already been audited by the REF eg OA status / 
processes/ circumstances 
 
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 
 Audit - process If it is a process audit what would this look like? 
What level of detail would be required? 
 
What would be the impact of Audit on administrators, 
in particular in smaller teams. 
● Work with RE in a process, to come up with a 
template that suits institutions, outlining how we 
should evidence our processes 
● Look for evidence in the template of how it is fair 
across various types of institutions. 
Audit process- Open 
Access 
 
What will Audit on Open Access look like? 
 
Several issues around this area including: 
  
● New and complex process. 
● Risk of publications coming up unclassified.  
● Tolerances for OA, evidence for acceptance 
dates 
● Is it a sample of institutions?  
● Can the auditor challenge exceptions? 
● Dealing with human/admin error and 
publisher error  
● Increase to administrative burden 
 
● Evidence of robust process and systematic 
documentation in the institution. 
● Each institution will have a different way but need 
to have documented steps.. 
● Audit on OA process should be investigative not 
punitive. Focus on the process being in place and 
applied  
● Audit should be for learning/ a conversation.  
● Develop process for next REF. 
   
 
Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019 
 


Note of Association of Research Managers and Administrators REF Audit Compliance Meeting

Enlighten

 Note of Association of Research Managers and Administrators  REF Audit Compliance Meeting  11​th​ April 2019, Glasgow.  ARMA LEADS: Anna Grey; Valerie McCutcheon SUPPORT: Michael Eadie; Sue Robson; Shelley Anne Stringer Observer from Research England:  Andy Hepburn 1. Summary The aim of the meeting was to examine audit requirements for Research Excellence Framework (REF). Attendees were involved in managing REF and open access exceptions on a day to day basis.  The meeting discussed the expectations of audit arising from Guidance on Submissions, Panel Criteria and Working Methods and Open Access implementation.  Outcomes of the meeting included key issues and potential solutions identified by the group.  This document summarises the prioritised points rather than every detail of the discussion.  The feedback will be shared with the Research England REF Team.  A further meeting is planned to discuss the new audit guidelines when they are issued.  2. Background/structure The workshop arose from discussions in the ARMA Open Access and REF special interest groups. The agenda was split as follows: ● A short, initial presentation which outlined the 3 stages of audit in 2014 and identified practical aspects of preparations. ● Breakout groups which focused on Generic REF audit (including impact) and Open Access specific issues. These identified key issues for the group and, where possible, identified potential solutions.   3. Overview of audit aspects of REF 3.1. In 2014 there were 3 parts to audit:  ● A sample audit​ which included: proof that the members of staff were employed; sample of outputs and evidence for impact case studies. ● Data comparisons/ reconciliations​ which included: HESA submission versus REF submission in terms of income.  Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   ● Panel instigated audit​ which included:  items raised by the panel for example ‘how much did the author contribute to a paper?’  Last time these were the most stressful due to the 3 day turnaround time.  3.2. Everything in the REF is open to audit and a good audit trail is required. Area of focus include:  ● Staff eligibility and FTE will need to show were they employed on the census date and were they employed when the output came out? ● Staff Circumstances: how will this link with EDAP decisions made prior to the REF 2021 submission?  ● Impact and corroborating evidences ● Publication dates of Outputs:  the change to OA and pre-publication has complicated this.  ● Open Access status ● Co-authorship claims – this can still be queried even if these details were not specifically requested as part of the  the submission ● Staff connection to the unit 3.3. Guidance on Submission and panel criterias are the primary source of information until the REF audit guidelines are issued.  The 2014 ‘grid’ OA guidance provides a starting point for Open access but is difficult to find online.  3.4. Best practice advice​: ● Plan for audit as part of REF preparations and capture evidence as you go along.  ● After the REF there may be staff changes/ team members disband. Build this into your planning, for information sourced from other departments in your University. Make sure that you know who will be answering any queries.  ● Find the people in your institution who do the HESA return. Look at how the UOAs match the cost centres that are used in HESA.  Differences may occur, be aware of them while preparing your submission.  4. Issue gathering Outcomes from the breakout groups are summarised in Table 1. Key areas of discussion included: ● Staff processes outlined in the codes of practice, whether the focus of audit would be process or case driven, and there was a strong preference for a process driven approach. ● Recognition of the difficulties in the embedding of OA processes for changing guidance and a preference for OA audit to be investigative rather than punitive. Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   ● Issues around clarifying and recording OA exceptions – in particular ‘Other Exceptions’. ● Issues arising from the change in the GDPR landscape and from changes to HR systems – in particular, for information on staff who have left the institution.  5. Information links Guidance on submissions (2019/01)  Panel criteria and working methods (2019/02)  Open access in the post-2014 Research Excellence Framework: information and audit requirements    6. For further information contact Valerie McCutcheon ​valerie.mccutcheon@glasgow.ac.uk  Anna Grey ​anna.grey@york.ac.uk     Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   Table 1: Key issues and potential solutions identified by the breakout groups. AREA ISSUES Possible solutions Staff - Evidencing the Eligibility of Former Staff  What is required to evidence the Eligibility of Former Staff?  Guidance around this did not exist at the time of them leaving.  Change of policies around GDPR so this information may no longer exist.  Impact is a much longer time period  Dealing with changes on internal systems (HR) and record keeping ● Verification though previous REF/RAE submission ● Verification through external sources such as JeS, ResearchFish etc. ● Allow for system issues/failure ● Allow if evidence has been destroyed for GDPR reasons  Staff - connection to UOA What is required to evidence Substantial Connection.  ● Research England to give examples of acceptable evidence e.g a screenshot from websites/ publications affiliated with the publication  Staff - employed elsewhere  What evidence do we need to provide of employment outside HEI at the time of acceptance? ● Include detail in FAQs Staff - significant Responsibility for Research   What is required to evidence Significant Responsibility for Research.  Will the process or decisions made and recorded on individuals be audited?  Will a low % staff identified trigger an audit?   Correlation of HESA data. ● HEIs to provide details of the decisions of the deciding body (minutes or recommendations as supplied to the individual).  ● If there is an appeal process HEIs to provide information on the number and outcome of appeals.  Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   ● Research England to advise on what evidence would demonstrate that HEIs have followed the process. ● Would like audit to focus that process in Code of Practice has been followed, rather than queries on individual staff (this was an issue for a number of areas)  Staff- Evidencing Research Independence  What is required to evidence Research Independence? What records do we need to keep? Will they use RA records on HESA? To what extent is ‘academic judgement’ accepted as evidence? What about colleagues from a practice in research background registered for a PhD? ● HEIs to provide details of the decisions of the deciding body (minutes or recommendations as supplied to the individual).  ● If there is an appeal process HEIs to provide information on the number and outcome of appeals.  ● Research England to advise on what evidence would demonstrate that HEIs have followed the process  Staff- Decisions on circumstances  How will decisions on circumstances be audited?  Is there specific guidance for acquiring and retaining staff circumstance information   ● Accessible FAQs ● Research England to provide example cases and advise on acceptable types of evidence  ● The timing of audit should be at the time of EDAP decisions ● Clarify if Panels can raise a query on staff circumstances  Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   Open Access - dates  How will Open Access dates be audited?  What counts as ‘best efforts’ on acceptance dates? What if the author and publisher don’t agree? Variety of practices for difficult circumstance (such as where there is no acceptance date online/in the public domain record). What if there is a delay applying the right embargo? ● Audit to focus on process rather than individual dates/cases. ● HEIs to provide details of the process for when/if proxy dates are applied. ● Research England to provide confirmation around difficult cases in audit guidance/ FAQs ● Community/Research England to lobby for publishers to provide this data  Open Access - Exception Ambiguity  Open Access exceptions are ambiguous (especially ‘Other’ and gold OA).  Some form of template or checklist would be very helpful.  The dates are complex.   This has added to the level of OA burden . ● Community/Research England to establish common understanding and use cases for exceptions, especially for those exceptions such as ‘Other’ and gold OA ● Research England to ensure examples and guidance communicated openly to HEIs ● HEIs to document their process for recording compliance and assigning exceptions and ensure this is applied consistently ● HEIs to ensure control measures are included to protect against risk e.g. to protect against exceptions being applied inconsistently all staff assigning exceptions have regular training etc.  Open Access - Gold Open Access Gold Open Access specific queries  Was an article immediately made gold - not later or retrospective? If an article is published without a clear licence is that acceptable? ● Help and examples for difficult cases in audit guidance/ FAQs ● Audit the process ● If the published version of the article is immediately, permanently,  free to access on the publisher’s website with a licence that permits copying and reuse this is compliant.  See clause 239 in Guidance on Submission.  If the licence is unclear Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   judgement is required.  If uncertain  it is possible to use other options for compliance or exception. Outputs - pre-print/online early Guidance for outputs published as pre-prints/online late in 2013 unclear around:   ● Dates especially definitions of pre-prints /online early. ● Checking online early has not been entered by another institution  ● Help around difficult cases in audit guidance/ FAQs ● Further clarify the issue of whether early on-line is treated the same as a pre-print ● REF/software could  flag outputs entered previously by another institution ● Clarify the implications of removal of the reference to early on-line in 2013, which was in the draft Guidance on Submission. If there was an early online version out in 2013 but the print version came out in 2014, can it be submitted?  Outputs – new research What evidence required for ‘new research’    ● Guidance around difficult cases in audit guidance/ FAQs  Impact What evidence should be held in respect of the additional contextual data?  ● Recommended template  Audit - format Clarification on what audit will look like, in particular for new areas - OA status / staff responsibility for research/ independent research/ circumstances.   ● Base audit around process. ● Include right of reply and a reasonable time for it. ● Advise on any trigger points  ● Clarify whether panels will query areas which have already been audited by the REF eg OA status / processes/ circumstances  Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019   Audit - process If it is a process audit what would this look like? What level of detail would be required?  What would be the impact of Audit on administrators, in particular in smaller teams. ● Work with RE in a process, to come up with a template that suits institutions, outlining how we should evidence our processes ● Look for evidence in the template of how it is fair across various types of institutions. Audit process- Open Access  What will Audit on Open Access look like?  Several issues around this area including:   ● New and complex process. ● Risk of publications coming up unclassified.  ● Tolerances for OA, evidence for acceptance dates ● Is it a sample of institutions?  ● Can the auditor challenge exceptions? ● Dealing with human/admin error and publisher error  ● Increase to administrative burden  ● Evidence of robust process and systematic documentation in the institution. ● Each institution will have a different way but need to have documented steps.. ● Audit on OA process should be investigative not punitive. Focus on the process being in place and applied  ● Audit should be for learning/ a conversation.  ● Develop process for next REF.     Note of ARMA REF Audit Compliance Meeting 11​th​ April 2019  

http://eprints.gla.ac.uk/185233/1/185233.pdf

Note of Association of Research Managers and Administrators REF Audit Compliance Meeting

Abstract

Similar works

Full text

Available Versions

Enlighten: Publications

Enlighten