Social media metrics for new research evaluation

This chapter approaches, both from a theoretical and practical perspective, the most important principles and conceptual frameworks that can be considered in the application of social media metrics for scientific evaluation. We propose conceptually valid uses for social media metrics in research evaluation. The chapter discusses frameworks and uses of these metrics as well as principles and recommendations for the consideration and application of current (and potentially new) metrics in research evaluation.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springe

Social media metrics for new research evaluation

Merit, Expertise and Measuremen

“When You Use Social Media You Are Not Working”: Barriers for the Use of Metrics in Social Sciences

The Social Sciences have long been struggling with quantitative forms of research assessment—insufficient coverage in prominent citation indices and overall lower citation counts than in STM subject areas have led to a widespread weariness regarding bibliometric evaluations among social scientists. Fueled by the rise of the social web, new hope is often placed on alternative metrics that measure the attention scholarly publications receive online, in particular on social media. But almost a decade after the coining of the term altmetrics for this new group of indicators, the uptake of the concept in the Social Sciences still seems to be low. Just like with traditional bibliometric indicators, one central problem hindering the applicability of altmetrics for the Social Sciences is the low coverage of social science publications on the respective data sources—which in the case of altmetrics are the various social media platforms on which interactions with scientific outputs can be measured. Another reason is that social scientists have strong opinions about the usefulness of metrics for research evaluation which may hinder broad acceptance of altmetrics too. We conducted qualitative interviews and online surveys with researchers to identify the concerns which inhibit the use of social media and the utilization of metrics for research evaluation in the Social Sciences. By analyzing the response data from the interviews in conjunction with the response data from the surveys, we identify the key concerns that inhibit social scientists from (1) applying social media for professional purposes and (2) making use of the wide array of metrics available. Our findings show that aspects of time consumption, privacy, dealing with information overload, and prevalent styles of communication are predominant concerns inhibiting Social Science researchers from using social media platforms for their work

Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature

Data collected by social media platforms have recently been introduced as a new source for indicators to help measure the impact of scholarly research in ways that are complementary to traditional citation-based indicators. Data generated from social media activities related to scholarly content can be used to reflect broad types of impact. This paper aims to provide systematic evidence regarding how often Twitter is used to diffuse journal articles in the biomedical and life sciences. The analysis is based on a set of 1.4 million documents covered by both PubMed and Web of Science (WoS) and published between 2010 and 2012. The number of tweets containing links to these documents was analyzed to evaluate the degree to which certain journals, disciplines, and specialties were represented on Twitter. It is shown that, with less than 10% of PubMed articles mentioned on Twitter, its uptake is low in general. The relationship between tweets and WoS citations was examined for each document at the level of journals and specialties. The results show that tweeting behavior varies between journals and specialties and correlations between tweets and citations are low, implying that impact metrics based on tweets are different from those based on citations. A framework utilizing the coverage of articles and the correlation between Twitter mentions and citations is proposed to facilitate the evaluation of novel social-media based metrics and to shed light on the question in how far the number of tweets is a valid metric to measure research impact.Comment: 22 pages, 4 figures, 5 table

Social media in scholarly communication : a review of the literature and empirical analysis of Twitter use by SSHRC doctoral award recipients

This report has been commissioned by the Social Sciences and Humanities Research Council (SSHRC) to analyze the role that social media currently plays in scholarly communication as well as to what extent metrics derived from social media activity related to scholarly content can be applied in an evaluation context. Scholarly communication has become more diverse and open with research being discussed, shared and evaluated online. Social media tools are increasingly being used in the research and scholarly communication context, as scholars connect on Facebook, LinkedIn and Twitter or specialized platforms such as ResearchGate, Academia.edu or Mendeley. Research is discussed on blogs or Twitter, while datasets, software code and presentations are shared on Dryad, Github, FigShare and similar websites for reproducibility and reuse. Literature is managed, annotated and shared with online tools such as Mendeley and Zotero, and peer review is starting to be more open and transparent. The changing landscape of scholarly communication has also brought about new possibilities regarding its evaluation. So-called altmetrics are based on scholarly social media activity and have been introduced to reflect scholarly output and impact beyond considering only peer-reviewed journal articles and citations within them to measure scientific success. This includes the measurement of more diverse types of scholarly work and various forms of impact including that on society. This report provides an overview of how various social media tools are used in the research context based on 1) an extensive review of the current literature as well as 2) an empirical analysis of the use of Twitter by the 2010 cohort of SSHRC Doctoral Award recipients was analyzed in depth. Twitter has been chosen as one of the most promising tools regarding interaction with the general public and scholarly communication beyond the scientific community. The report focuses on the opportunities and challenges of social media and derived metrics and attempts to provide SSHRC with information to develop guidelines regarding the use of social media by funded researchers as well support the informed used of social media metrics

“When You Use Social Media You Are Not Working”: Barriers for the Use of Metrics in Social Sciences

The Social Sciences have long been struggling with quantitative forms of research assessment—insufficient coverage in prominent citation indices and overall lower citation counts than in STM subject areas have led to a widespread weariness regarding bibliometric evaluations among social scientists. Fueled by the rise of the social web, new hope is often placed on alternative metrics that measure the attention scholarly publications receive online, in particular on social media. But almost a decade after the coining of the term altmetrics for this new group of indicators, the uptake of the concept in the Social Sciences still seems to be low. Just like with traditional bibliometric indicators, one central problem hindering the applicability of altmetrics for the Social Sciences is the low coverage of social science publications on the respective data sources—which in the case of altmetrics are the various social media platforms on which interactions with scientific outputs can be measured. Another reason is that social scientists have strong opinions about the usefulness of metrics for research evaluation which may hinder broad acceptance of altmetrics too. We conducted qualitative interviews and online surveys with researchers to identify the concerns which inhibit the use of social media and the utilization of metrics for research evaluation in the Social Sciences. By analyzing the response data from the interviews in conjunction with the response data from the surveys, we identify the key concerns that inhibit social scientists from (1) applying social media for professional purposes and (2) making use of the wide array of metrics available. Our findings show that aspects of time consumption, privacy, dealing with information overload, and prevalent styles of communication are predominant concerns inhibiting Social Science researchers from using social media platforms for their work. Regarding indicators for research impact we identify a widespread lack of knowledge about existing metrics, their methodologies and meanings as a major hindrance for their uptake through social scientists. The results have implications for future developments of scholarly online tools and show that researchers could benefit considerably from additional formal training regarding the correct application and interpretation of metrics

TweetBubble: A Twitter Extension Stimulates Exploratory Browsing

We extend the Twitter interface to stimulate exploratory browsing of social media and develop a method to establish its efficacy. In exploratory browsing, users seek and traverse diverse and novel information as they investigate a conceptual space. Social media has become a predominant source of information. Social media is characterized by rapidly evolving content and relationships. We need interface mechanisms to effectively and iteratively browse, search, and filter, i.e., explore vast social media networks. We develop the TweetBubble browser extension, extending Twitter to enable expansion of social media associations @usernames and #hashtags in-context, with-out overwriting initial content. We build on a prior metadata type system, developing new presentation semantics, which enable a look and feel consistent with Twitter. We identify exploratory browsing as a creative ideation process. We use prior ideation metrics as a basis for new ideation metrics of exploratory browsing. We conducted a study, with data from 54 participants, amidst the 2014 Academy Awards. Quantitative and qualitative findings validate the technique of in-context exploratory browsing interfaces for social media. Their consistency supports the validity of ideation metrics of exploratory browsing as an evaluation methodology for interactive systems designed to promote creative engagement. This research impacts the design and evaluation of interfaces that stimulate intrapersonal creativity, and thereby mutual understanding, by supporting exploratory browsing of connected perspectives in a shared, structured, conceptual space

Evaluation of Media-Based Social Interactions in Virtual Environments

The evaluation of users’ experiences in virtual environments is an important task for researchers in the fields of human-computer interaction and extended reality. It can be used to understand and enhance the quality of users’ mediated interactions and communications. In a constantly evolving world, where people are growing with technology, it is important to understand, evaluate and enhance the use of immersive media. In the research agenda of this Ph.D. thesis, the challenge of developing multi-user experiences in virtual environments and setting evaluation metrics for researchers are considered. This Ph.D. thesis showcases an interest in how to enhance trust formation in media-based social environments. The findings of this Ph.D. are expected to help create new open-source tools to facilitate the understanding of individuals and groups in extended reality applications