Online impact – surveying the websites most commonly cited in impact case studies

Reporting on their recent survey of websites cited in REF 2014 impact case studies, Kayvan Kousha, Mike Thelwall and Mahshid Abdoli, discuss which websites are most commonly used as supporting evidence for impact and how these vary across academic disciplines

Can Microsoft Academic assess the early citation impact of in-press articles? A multi-discipline exploratory analysis

Many journals post accepted articles online before they are formally published in an issue. Early citation impact evidence for these articles could be helpful for timely research evaluation and to identify potentially important articles that quickly attract many citations. This article investigates whether Microsoft Academic can help with this task. For over 65,000 Scopus in-press articles from 2016 and 2017 across 26 fields, Microsoft Academic found 2-5 times as many citations as Scopus, depending on year and field. From manual checks of 1,122 Microsoft Academic citations not found in Scopus, Microsoft Academic���s citation indexing was faster but not much wider than Scopus for journals. It achieved this by associating citations to preprints with their subsequent in-press versions and by extracting citations from in-press articles. In some fields its coverage of scholarly digital libraries, such as arXiv.org, was also an advantage. Thus, Microsoft Academic seems to be a more comprehensive automatic source of citation counts for in-press articles than Scopus

Which types of online evidence show the non-academic benefits of research? Websites cited in UK impact case studies

© 2021 The Authors. Published by MIT Press. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.1162/qss_a_00145Whilst funders increasingly request evidence of the societal benefits of research, all academics in the UK must periodically provide this information to gain part of their block funding within the Research Excellence Framework (REF). The impact case studies produced in the UK are public and can therefore be used to gain insights into the types of sources used to justify societal impact claims. This study focuses on the URLs cited as evidence in the last public REF to help researchers and resource providers to understand what types can be used and the disciplinary differences in their uptake. Based on a new semi-automatic method to classify the URLs cited in impact case studies, the results show that there are a few key online types of source for most broad fields, but these sources differ substantially between subject areas. For example, news websites are more important in some fields than others, and YouTube is sometimes used for multimedia evidence in the arts and humanities. Knowledge of the common sources selected independently by thousands of researchers may help others to identify suitable sources for the complex task of evidencing societal impacts

Gender disparities in UK research publishing: Differences between fields, methods and topics

This is an accepted manuscript of an article published by EPI SCP in El Profesional de la Información on 21/07/2020, available online: https://doi.org/10.3145/epi.2020.jul.15 The accepted version of the publication may differ from the final published version.Gender disparities persist in UK research, with female minorities in most science, technology, engineering and mathematics (STEM) subjects but female majorities in others. The nature of the gender disparity differences between subjects needs to be understood if effective remedial actions are to be targeted at STEM subjects suffering from a lack of women, in contrast to other subjects without shortages. Evidence from the USA suggests that women engage more in people-related subjects, qualitative methods, veterinary science and cell biology and men engage more in thing-related subjects, power/control fields, patient-related research, abstraction and quantitative methods, except surveys. This article investigates gender disparity differences in UK first authorship for journal articles in nearly all of science split into 26 broad and 308 narrow Scopus fields. The results largely replicate the USA but suggest that more life science topics may be female-associated in the UK and patient-related research might not be male-associated. UK STEM gender parity initiatives might therefore emphasise people-oriented, and perhaps socially positive, aspects of currently masculine STEM topics and approaches (e.g., abstraction, mathematical quantitative methods), and promote female-friendly topics, methods and goals within male-dominated fields in addition to tacking implicit and explicit sexism and providing a supportive working environment.Published onlin

Terms in journal articles associating with high quality: Can qualitative research be world-leading?

Purpose: Scholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however, and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence of aspects of research that associate with high or low quality. Design/methodology/approach: We extracted the words and 2-5-word phrases most strongly associating with different quality scores in each of 34 Units of Assessment (UoAs) in the Research Excellence Framework (REF) 2021. We extracted the terms from 122,331 journal articles 2014-2020 with individual REF2021 quality scores. Findings: The terms associating with high- or low-quality scores vary between fields but relate to writing styles, methods, and topics. We show that the first-person writing style strongly associates with higher quality research in many areas because it is the norm for a set of large prestigious journals. We found methods and topics that associate with both high- and low-quality scores. Worryingly, terms associating with educational and qualitative research attract lower quality scores in multiple areas. REF experts may rarely give high scores to qualitative or educational research because the authors tend to be less competent, because it is harder to make world leading research with these themes, or because they do not value them. Originality: This is the first investigation of journal article terms associating with research quality

In which fields are citations indicators of research quality?

Citation counts are widely used as indicators of research quality to support or replace human peer review and for lists of top cited papers, researchers, and institutions. Nevertheless, the extent to which citation counts reflect research quality is not well understood. We report the largest-scale evaluation of the relationship between research quality and citation counts, correlating them for 87,739 journal articles in 34 field-based Units of Assessment (UoAs) from the UK. We show that the two correlate positively in all academic fields examined, from very weak (0.1) to strong (0.5). The highest correlations are in health, life sciences and physical sciences and the lowest are in the arts and humanities. The patterns are similar for the field classification schemes of Scopus and Dimensions.ai. We also show that there is no citation threshold in any field beyond which all articles are excellent quality, so lists of top cited articles are not definitive collections of excellence. Moreover, log transformed citation counts have a close to linear relationship with UK research quality ranked scores that is shallow in some fields but steep in others. In conclusion, whilst appropriately field normalised citations associate positively with research quality in all fields, they never perfectly reflect it, even at very high values