Carmichael, Patrick

Cook, John

Slater, John

English

Name not available

    1 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/  Research Excellence Framework 9 December 2009 consultation response form the Association for Learning Technology (ALT)  Are you responding On behalf of an organisation  Name of responding organisation/individual Association for Learning Technology (ALT) Type of organisation  Academic association or learned society  Professional body Charity/third sector organisation  Contact name Seb Schmoller Position within organisation  Chief Executive Contact phone number 0114 2586899 Contact e-mail address seb.schmoller@alt.ac.uk  Consultation question 1: Do you agree with the proposed key features of the REF? If not, explain why.  1.1 In general, yes: the key features should be outputs, impact and environment. However we have detailed comments on how these are to be assessed and where they should be applied. We agree that measures should be taken to reduce burden (such as cutting the number of outputs submitted).  We have concerns about the proposed timing, in the light of potential political and financial changes; about the adequacy of arrangements for the recognition and assessment of interdisciplinary work; and about the range of outputs that might be included in submissions.  1.2 Another concern for ALT is that many of its members are located in departments, units and centres that are not themselves responsible for compiling submissions: instead, they might be included in units of assessment (UOAs) such as education and computer science.  While the organisation of submissions clearly lies with institutions, and there is work to be done by ALT in supporting its members so that they are seen as ‘returnable’ by their institutions, it is important that features of the REF do not systematically militate against individuals working on learning technologies being entered into the REF in appropriate UOAs, or that learning technologists are only considered as the providers of ‘impact’ case studies, or excluded from accounts of research community.   ALT response to the REF consultation, 9/12/2009  2 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/   Consultation question 2: What comments do you have on the proposed approach to assessing outputs? If you disagree with any of these proposals please explain why.  Comments are especially welcomed on the following proposals: • that institutions should select research staff and outputs to be assessed • for the categories of staff eligible for selection, and how they are defined  • for encouraging institutions to submit – and for assessing – all types of high-quality research outputs including applied and translational research • for the use of citation information to inform the review of outputs in appropriate UOAs (including the range of appropriate UOAs, the type of citation information that should be provided to panels as outlined in Annex C, and the flexibility panels should have in using the information) and on the following options: • whether there should be a maximum of three or four outputs submitted per researcher • whether certain types of output should be ‘double weighted’ and if so, how these could be defined. 2.1 While it is appropriate that Higher Education Institutions (HEIs) should select staff to be considered by the REF, there needs to be recognition of the following both by HEIs and by the REF Panels. • Many researchers in learning technologies are new: the field itself is new.  It is also a net ‘importer’ of people, concepts and research approaches from other disciplines (which come under the remits of different UOAs and Panels). • Many researchers in learning technologies are employed on part-time and short-term contracts and in posts classified as ‘academic-related’.  The notion of ‘early career researchers’ needs to be broadened to include not only PhD and postdoctoral researchers but also experienced practitioners newly engaged in research and those who have prior research careers in other disciplines. • Learning technology research, and interdisciplinary research in general, are also characterised by team working and this leads to multi-authorship of outputs, often across institutions which itself can lead to problems in situating impact and in describing and attributing the enabling environment. 2.2 The guidelines for both institutions and panels need clearly to be spelled out in relation to all of these issues. Many institutions believe firmly that full-time, established “mainstream” researchers have been and are still preferred in research exercises. Even if it is not the funding bodies or the REF processes that are responsible for resulting discrimination, responsibility lies with funding bodies and the REF to establish and make clear processes that do not permit discrimination.  2.3 It is important that a wide range of outputs should be eligible for submission, including evaluations, reports, reviews, and software and related products.  This is particularly important in learning technology research where significant amounts of high quality research is specifically ALT response to the REF consultation, 9/12/2009  3 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/   funded to produce such outputs in addition to, or instead of, for example, refereed journal articles or books.  2.4 Citation information should be available to all panels, even where it is considered to be insufficiently robust to be used algorithmically in the discipline: the extent to which it is used should be reported formally at main Panel and UOA level. As in the Education UOA as a whole, the wide range of outlets and the existence of many ‘niche’ publications (many of which have high impact as a result) calls the validity of citation metrics into question.  The fact that learning technology research may be published through both specialist and generalist education outlets makes it difficult to gauge how citation metrics would inform the REF process.  2.5 The change from four to three publications per individual returned is welcomed and should encourage individuals to focus on the production of higher quality research outputs.  Double counting has the same effect.   Consultation question 3: What comments do you have on the proposed approach to assessing impact? If you disagree with any of these proposals please explain why.  Comments are especially welcomed on the following: • how we propose to address the key challenges of time lags and attribution • the type of evidence to be submitted, in the form of case studies and an impact statement supported by indicators (including comments on the initial template for case studies and menu of indicators at Annex D) • the criteria for assessing impact and the definition of levels for the impact sub-profile • the role of research users in assessing impact. 3.1 Learning technology is an emerging field and while some impact is almost immediate, other changes are only starting to emerge after longer periods of associated pedagogical development, curriculum reform and institutional response to new learning technologies.  To focus on short term impact seems to suggest that learning technologies are seen as a kind of ‘intervention’ with clear and rapid outcomes, rather than as the catalysts for wider ranging and longer-term changes in teaching and learning.  3.2 The mobility of staff in learning technology highlighted above means that the question of whether it is individuals, the work of individuals, or the progress made by the HEI in implementing the work of the individual (who may now have left and be working elsewhere) needs to be clarified.  Case studies of impact of an HEI that are  based on the work of teams who have subsequently dispersed, or where work has been inspired by a single ‘high impact’ individual but realised by others all seem to be valid accounts.  3.3 The Higher Education Academy and JISC funded research specifically looks for evidence of ‘embedding’ and ‘impact’ within host institutions and encourages sharing of findings within networks of practitioners through case studies, research resources and toolkits and software demonstrators.  While some groups and individuals have been able to achieve this at the same time as also generating more traditional academic outputs such as books and journal articles, ALT response to the REF consultation, 9/12/2009  4 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/   there is a significant amount of work in learning technologies where conditions of funding and the applied nature of the research needs to be recognised in any assessment of impact. Another outcome of this research orientation is that in the short and medium term, the most informative accounts of impact will indeed be partially based on changes to ‘host’ institutions, so we argue that it is valid to include amongst indicators of impact the impact within one’s own HEI. While not as strong an indicator of impact as, for example, wider adoption, this is how impact of learning technologies is evaluated, at least initially.  3.4 Impact is perhaps more useful at a coarser level of granularity to inform subsequent overall funding for disciplines (“Pots of gold”). The funders do not seem to have taken any position on this yet. At the moment it seems that it is possibly just to be used within a UOA to give relative discrimination and so, if it is not used by, say giving everyone pro rata scores from the other elements, then it will be factored out. This seems wrong.   Consultation question 4: Do you have any comments on the proposed approach to assessing research environment?   4. The research environment in learning technology often transcends specific research groups, laboratories and centres.  Concentration of expertise and activity often arises from the formation of cross-disciplinary and cross-institutional groups and through inter-working by individuals. In many cases learning technology is the focus for an extended research environment which links the computing, library, professional development and academic practice activities of one or more usually of several HEIs. Activities that contribute to the development of research culture beyond the formal UoA should also be considered. Thus in assessing environment at an HEI, it is important to consider its ability for interworking effectively with those at other sites. Identifying whether this has  been done well is likely to  require some specialised input from for example JISC.   Consultation question 5: Do you agree with our proposals for combining and weighting the output, impact and environment sub-profiles? If not please propose an alternative and explain why this is preferable.    5. We are happy with the proposed weightings. The concern we have stems (again) from the characteristically interdisciplinary nature of learning technology research and the potential for colleagues working in learning technology to be assessed by different UOAs and Panels with different interpretations of the weightings and of the criteria for impact and environment in particular.   Consultation question 6: What comments do you have on the panel configuration proposed at Annex E? Where suggesting alternative options for specific UOAs, please provide the reasons for this.  6. The decision to reduce the number of UOAs is welcomed and the four main panels seem to be well defined.  Learning technology research is largely within Education, although some colleagues may be entered within Psychology, Computing Sciences and other disciplines.  This ALT response to the REF consultation, 9/12/2009  5 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/   raises the issue addressed in question 1 and further in question 9 – how to make sure that interdisciplinary work such as learning technology is not inadvertently excluded or discriminated against at institutional level.   Consultation question 7: Do you agree with the proposed approach to ensuring consistency between panels?  Yes.   Consultation question 8: Do you have any suggested additions or amendments to the list of nominating bodies? (If suggesting additional bodies, please provide their names and addresses and indicate how they are qualified to make nominations.)   7.1 The Australasian Society for Computers in Learning in Tertiary Education (ascilite). ascilite encourages and supports research into, and exemplary use of technologies for teaching and learning in tertiary education throughout Australasia. Address: ascilite, PO Box 44, Figtree, NSW, Australia. President: Mike Keppell, Ph.D., Professor of Higher Education at Charles Sturt University. Email: mkeppell@csu.edu.au.   7.2 The Sloan Consortium (Sloan-C). The consortium is a well-established US-based consortium of individuals, institutions and organisations committed to quality online education. Address: The Sloan Consortium, Inc., PO Box 1238, Newburyport, MA 01950-8238, USA. Executive Director: John Bourne, Ph.D., Professor of Electrical and Computer Engineering at Olin College, and Professor of Technology Entrepreneurship at Babson College. Email: jbourne@sloanconsortium.org.   Consultation question 9: Do you agree that our proposed approach will ensure that interdisciplinary research is assessed on an equal footing with other types of research? Are there further measures we should consider to ensure that this is the case and that our approach is well understood?      9.1 No. This is a major area of concern; and our concerns are two-fold.   9.2 Our first concern is that, despite declared confidence on the part of Panels that they are competent to assess interdisciplinary research outputs, disciplinary norms will be asserted and interdisciplinary research will be marginalised.  The risk of this would be ameliorated by appropriate nominations to panels and by more transparent processes of assessment, supported by data.  9.3 A second and perhaps more serious concern is that HEIs will discriminate against interdisciplinary research being reported in the REF and against researchers representing themselves as interdisciplinary. This is already having an impact on the work of individual researchers as they bid for project funding and write research outputs.  Some colleagues (such as those working in TLRP-TEL projects, jointly funded by ESRC and EPSRC) can justify ALT response to the REF consultation, 9/12/2009  6 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/   positioning themselves as interdisciplinary researchers, but for others, this may prove to be a high-risk option.  9.4 It is very clear that advice is already being given to our members in their institutions to make sure that they have the right number of “mainstream” outputs rather than those that are interdisciplinary in character, including those that are located in learning technology research.  Thus many HEIs do not seem to currently believe you and that belief is sufficient to cause discrimination.  9.5 A commitment to ensure that major interdisciplinary groups are represented on panels and to a more open process needs to be articulated very early in the cycle so as to be credible.  Ironically, it may be that it is the interdisciplinary activities and associated outputs that have relatively high levels of impact, and contribute to distinctive research environments.   Consultation question 10: Do you agree that our proposals for encouraging and supporting researcher mobility will have a positive effect; and are there other measures that should be taken within the REF to this end?   10. These seem appropriate but actual implementation may well be dependent on specific UOA interpretations of the guidelines.   Consultation question 11: Are there any further ways in which we could improve the measures to promote equalities and diversity?  11.1 The REF, with its emphasis on long-term impact, and with its assumption that researchers have fairly predictable and linear career trajectories may discriminate against part-time and multi-working participants in research.  This in turn may have implications in relation to the gender and other balances of the cohorts entered for the REF.   11.2 More detail and clear guidance that is fair and promotes equality and diversity on how to treat part timers within the exercise, as well as early career researchers and returners, would be of considerable value. Greater diversity would result to the benefit of the system.   Consultation question 12: Do you have any comments about the proposed timetable?  12.1 The timetable as envisaged may be hard to deliver and involve unnecessary burden.   12.2 Activities timetabled for 2010 may need to be significantly reworked, perhaps by new bodies or individuals responding to new sets of governmental priorities and also to new funding strictures.  The field of learning technology research is particularly sensitive to the vagaries of funding regimes: both at source and in how institutions allocate funds to learning technologies development and related pedagogical innovation.  ALT response to the REF consultation, 9/12/2009  7 This document may be accessed from the ALT Open Access repository from http://repository.alt.ac.uk/683/   12.3 To reduce the burden on all concerned, we urge a 1-year delay in the process although the pilots and experiments should continue so that there will be evidence to support future activities.    Consultation question 13: Are there any further areas in which we could reduce burden, without compromising the robustness of the process?  13. See answer to question 12: it is important that HEIs do not “gear up” for things that are then changed. A one year delay makes this extra burden less likely.   Consultation question 14: Do you have any other comments on the proposals?  14.1 The Association for Learning Technology (ALT) is a professional and scholarly association which brings together those with an interest in the use of learning technology. It has over 200 organisational members and over 650 individual members, including academic and other researchers who are creators of educational digital content as well as contributors to research and policy. Nearly all HE establishments in the UK are organisational members as are many FE establishments who deliver HE courses.  14.2 ALT is a nominating body for members of panels. In the last RAE our main nominee was accepted onto the Education panel. In addition, through the good offices of the DfES, we met with key figures involved with the panel to voice our concerns for the learning technology field. In part as a result, we believe that researchers in our field and their endeavours were better understood and treated than in previous exercises. We are thus pleased to see this consultation and the issues involved, many of which, such as those in the areas of interdisciplinary working and impact, have significant bearing on the work of our members. We would hope that, as the process evolves further, it will be possible to have similar dialogue this time.  

ALT Response the Research Excellence Framework Consultation

ALT Open Access Repository

https://repository.alt.ac.uk/683/1/ALT_REF_Consultation_2009_final_for_web_site.pdf

ALT Response the Research Excellence Framework Consultation

Abstract

Similar works

Full text

Available Versions

Name not available

ALT Open Access Repository