The pros and cons of the use of altmetrics in research assessment

© 2020 The Authors. Published by Levi Library Press. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: http://doi.org/10.29024/sar.10Many indicators derived from the web have been proposed to supplement citation-based indicators in support of research assessments. These indicators, often called altmetrics, are available commercially from Altmetric.com and Elsevier’s Plum Analytics or can be collected directly. These organisations can also deliver altmetrics to support institutional selfevaluations. The potential advantages of altmetrics for research evaluation are that they may reflect important non-academic impacts and may appear before citations when an article is published, thus providing earlier impact evidence. Their disadvantages often include susceptibility to gaming, data sparsity, and difficulties translating the evidence into specific types of impact. Despite these limitations, altmetrics have been widely adopted by publishers, apparently to give authors, editors and readers insights into the level of interest in recently published articles. This article summarises evidence for and against extending the adoption of altmetrics to research evaluations. It argues that whilst systematicallygathered altmetrics are inappropriate for important formal research evaluations, they can play a role in some other contexts. They can be informative when evaluating research units that rarely produce journal articles, when seeking to identify evidence of novel types of impact during institutional or other self-evaluations, and when selected by individuals or groups to support narrative-based non-academic claims. In addition, Mendeley reader counts are uniquely valuable as early (mainly) scholarly impact indicators to replace citations when gaming is not possible and early impact evidence is needed. Organisations using alternative indicators need recruit or develop in-house expertise to ensure that they are not misused, however

Automatically detecting open academic review praise and criticism

This is an accepted manuscript of an article published by Emerald in Online Information Review on 15 June 2020. The accepted version of the publication may differ from the final published version, accessible at https://doi.org/10.1108/OIR-11-2019-0347.Purpose: Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research’s open peer review publishing platform. Design/methodology/approach: PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings. Findings: PeerJudge can predict F1000Research judgements from negative evaluations in reviewers’ comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision. Originality/value: PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments

A Thematic analysis of highly retweeted early COVID-19 tweets: Consensus, information, dissent, and lockdown life

This is an accepted manuscript of an article published by Emerald in Aslib Journal of Information Management on 23/10/2020. The accepted version of the publication may differ from the final published version.Purpose: Public attitudes towards COVID-19 and social distancing are critical in reducing its spread. It is therefore important to understand public reactions and information dissemination in all major forms, including on social media. This article investigates important issues reflected on Twitter in the early stages of the public reaction to COVID-19. Design/methodology/approach: A thematic analysis of the most retweeted English-language tweets mentioning COVID-19 during March 10-29, 2020. Findings: The main themes identified for the 87 qualifying tweets accounting for 14 million retweets were: lockdown life; attitude towards social restrictions; politics; safety messages; people with COVID-19; support for key workers; work; and COVID-19 facts/news. Research limitations/implications: Twitter played many positive roles, mainly through unofficial tweets. Users shared social distancing information, helped build support for social distancing, criticised government responses, expressed support for key workers, and helped each other cope with social isolation. A few popular tweets not supporting social distancing show that government messages sometimes failed. Practical implications: Public health campaigns in future may consider encouraging grass roots social web activity to support campaign goals. At a methodological level, analysing retweet counts emphasised politics and ignored practical implementation issues. Originality/value: This is the first qualitative analysis of general COVID-19-related retweeting

Anthropomorphizing atopy: Tweeting about eczema

This is an accepted manuscript of an article published by Wolters Kluwer in Journal of the Dermatology Nurses' Association on 03/04/2020, available online: https://journals.lww.com/jdnaonline/Abstract/2020/03000/Anthropomorphizing_Atopy__Tweeting_About_Eczema.4.aspx The accepted version of the publication may differ from the final published version.Atopic dermatitis or eczema affects a substantial minority of children and adults. Patients may treat their symptoms through skin care regimes and/or diet restrictions and/or prescribed topical corticosteroids. The patient perspective is important because of the longterm self-administered treatment regime and the potential psychological effects on relationships from a visible disease. This paper assesses the potential of public social media data to give new insights into patient perspectives through a thematic analysis of a random sample of 400 tweets from 2019 matching the query, “my eczema”. Whilst the most common use of Twitter is to announce a flare-up, it is also used to express anger and discuss possible treatments. New themes not previously reported include the use of humor to discuss the condition and giving eczema agency: discussing it as if it had a will of its own. These may be defense strategies against the potential of eczema to strike at any time or the fear of negative reactions or blame from friends. This highlights importance of nurses and others helping patients to deal with the psychological effects of eczema

How has Covid-19 affected published academic research? A content analysis of journal articles mentioning the virus

© 2021 The Authors. Published by De Gruyter. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.2478/jdis-2021-0030Purpose: Methods to tackle Covid-19 have been developed by a wave of biomedical research but the pandemic has also influenced many aspects of society, generating a need for research into its consequences, and potentially changing the way existing topics are investigated. This article investigates the nature of this influence on the wider academic research mission. Design/methodology/approach: This article reports an inductive content analysis of 500 randomly selected journal articles mentioning Covid-19, as recorded by the Dimensions scholarly database on 19 March 2021. Covid-19 mentions were coded for the influence of the disease on the research. Findings: Whilst two thirds of these articles were about biomedicine (e.g., treatments, vaccines, virology), or health services in response to Covid-19, others covered the pandemic economy, society, safety, or education. In addition, some articles were not about the pandemic but stated that Covid-19 had increased or decreased the value of the reported research or changed the context in which it was conducted. Research limitations: The findings relate only to Covid-19 influences declared in published journal articles. Practical implications: Research managers and funders should consider whether their current procedures are effective in supporting researchers to address the evolving demands of pandemic societies, particularly in terms of timeliness. Originality/value: The results show that although health research dominates the academic response to Covid-19, it is more widely disrupting academic research with new demands and challenges

Data in brief: Can a mega-journal for data be useful?

This is an accepted manuscript of an article published by Springer in Scientometrics on 07/04/2020. The accepted version of the publication may differ from the final published version. https://doi.org/10.1007/s11192-020-03437-1As part of the current move towards open science, there is increasing pressure for scientists to share their research data. In support of this, several journals only publish descriptions of data generated from research: data papers. It is not clear whether this service encourages data reuse, however. This article assesses the prevalence and impact of the largest such journal, Data in Brief, comparing it with 24 other general or specialist data journals. The results show that Data in Brief became the largest data journal in 2016 and that its papers attracted over five Mendeley readers each, within a year of publication, as well as a non-trivial amount of citations. Its papers have been cited for relevance or facts contained in them in addition to acknowledging the reuse of associated datasets in about 1% of cases. Some papers describe electronic dataset whereas other papers embedded the tables or images that formed the shared data. Overall, the journal seems to make a positive contribution to science by enabling access to multiple types of data, even though its papers rarely lead to data reuse

Lifestyle information from YouTube influencers: Some consumption patterns

This is an accepted manuscript of an article published by Emerald in Journal of Documentation on 26/05/2021, available online at: https://doi.org/10.1108/JD-02-2021-0033 The accepted version of the publication may differ from the final published version.Purpose: Despite lifestyle information needs being an important part of our daily lives, little is known about the role of common sources. Whilst magazines and television are traditional providers of lifestyle content, including for fashion, makeup, fitness, and cookery, they have been partly replaced by content-creating online influencers. Design/methodology/approach: To investigate this new resource, this article analyses comments on the videos of 223 UK female lifestyle influencers on YouTube for information about possible viewing patterns. Findings: Three quarters of comments are written during the week of the video being published, consistent with videos being consumed with an information browsing function, rather than treated as an information resource to be searched when needed. Commenting on the videos of multiple influencers occurred often, suggesting that many viewers are not loyal to a single influencer. Thus, influencers seem to primarily support active scanning rather than searching for specific information. Typical viewers of UK female lifestyle influencers can therefore expect to accumulate lifestyle ideas and knowledge for potential future use in addition to gaining timely suggestions for near future routine decision making. Practical implications: Public-facing information professionals, health professionals and counsellors may consider recommending selected videos or influencers to help with lifestyle concerns. Originality: This is the first large scale study of content-creating influencers as a lifestyle information resource

Coronavirus research before 2020 is more relevant than ever, especially when interpreted for COVID-19

© 2020 The Authors. Published by MIT Press. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.1162/qss_a_00083The speed with which biomedical specialists were able to identify and characterise COVID-19 was partly due to prior research with other coronaviruses. Early epidemiological comparisons with Severe Acute Respiratory Syndrome (SARS) and Middle East Respiratory Syndrome (MERS), also made it easier to predict COVID-19’s likely spread and lethality. This article assesses whether academic interest in prior coronavirus research has translated into interest in the primary source material, using Mendeley reader counts for early academic impact evidence. The results confirm that SARS and MERS research 2008-2017 experienced anomalously high increases in Mendeley readers in April-May 2020. Nevertheless, studies learning COVID-19 lessons from SARS and MERS or using them as a benchmark for COVID-19 have generated much more academic interest than primary studies of SARS or MERS. Thus, research that interprets prior relevant research for new diseases when they are discovered seems to be particularly important to help researchers to understand its implications in the new context

Author gender differences in psychology citation impact 1996-2018

Academic psychology in the USA is a gender success story in terms of overturning its early male dominance but there are still relatively few senior female psychology researchers. To assess whether there are gender differences in citation impact that might help to explain either of these trends, this study investigates psychology articles since 1996. Seven out of eight Scopus psychology categories had a majority of female first-authored journal articles by 2018. From regression analyses of first and last author gender and team size, female first authors associate with a slightly higher average citation impact, but extra authors have a ten times stronger association with higher average citation impact. Last author gender has little association with citation impact. Female first authors are more likely to be in larger teams and if team size is attributed to the first author’s work, then their apparent influence of female first authors on citation impact doubles. Whilst gender differences in average citation impact are too small to account for gender-related trends in academic psychology, they warn that male dominated citation-based ranking lists of psychologists do not reflect the state of psychology research today

Authorship and citation gender trends in immunology and microbiology

This is an accepted manuscript of an article published by Oxford University Press in FEMS Microbiology Letters on 06/02/2020, available online: https://academic.oup.com/femsle/article/367/2/fnaa021/5728488 The accepted version of the publication may differ from the final published version.Immunology and microbiology research are essential for human and animal health. Unlike many other health fields, they do not usually centre around the curing or helping individual patients but focus on the microscopic scale instead. These fields are interesting from a gender perspective because two theories seeking to explain gender differences in career choices in the USA (people/things and communal/agentic goals) might produce conflicting expectations about their gender balances. This article assesses the gender shares of journal articles and gendered citation rates of five subfields of the Scopus Immunology and Microbiology broad category 1996-2014/18, for research with solely US author affiliations. Only Applied Microbiology and Biotechnology (38% female) had not reached gender parity in publishing by 2018. There was a female first author citation advantage in Parasitology but a disadvantage in Immunology. Immunology, Parasitology and Virology, had female last author citation disadvantages, but all gender effects were much smaller (<5%) than that of an extra author (10%-56%). Citation differences cannot therefore account for the current underrepresentation of women in senior roles