Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective

An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some altmetric activity and concentrating in the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities and the medical and life sciences show the highest presence of altmetrics, indicating their potential value and interest for these fields. The analysis of the relationships between altmetrics and citations confirms previous claims of positive correlations but relatively weak, thus supporting the idea that altmetrics do not reflect the same concept of impact as citations. Also, altmetric counts do not always present a better filtering of highly cited publications than journal citation scores. Altmetrics scores (particularly mentions in blogs) are able to identify highly cited publications with higher levels of precision than journal citation scores (JCS), but they have a lower level of recall. The value of altmetrics as a complementary tool of citation analysis is highlighted, although more research is suggested to disentangle the potential meaning and value of altmetric indicators for research evaluation

Scholarly use of social media and altmetrics : a review of the literature

Social media has become integrated into the fabric of the scholarly communication system in fundamental ways: principally through scholarly use of social media platforms and the promotion of new indicators on the basis of interactions with these platforms. Research and scholarship in this area has accelerated since the coining and subsequent advocacy for altmetrics—that is, research indicators based on social media activity. This review provides an extensive account of the state-of-the art in both scholarly use of social media and altmetrics. The review consists of two main parts: the first examines the use of social media in academia, examining the various functions these platforms have in the scholarly communication process and the factors that affect this use. The second part reviews empirical studies of altmetrics, discussing the various interpretations of altmetrics, data collection and methodological limitations, and differences according to platform. The review ends with a critical discussion of the implications of this transformation in the scholarly communication system

Social media in scholarly communication : a review of the literature and empirical analysis of Twitter use by SSHRC doctoral award recipients

This report has been commissioned by the Social Sciences and Humanities Research Council (SSHRC) to analyze the role that social media currently plays in scholarly communication as well as to what extent metrics derived from social media activity related to scholarly content can be applied in an evaluation context. Scholarly communication has become more diverse and open with research being discussed, shared and evaluated online. Social media tools are increasingly being used in the research and scholarly communication context, as scholars connect on Facebook, LinkedIn and Twitter or specialized platforms such as ResearchGate, Academia.edu or Mendeley. Research is discussed on blogs or Twitter, while datasets, software code and presentations are shared on Dryad, Github, FigShare and similar websites for reproducibility and reuse. Literature is managed, annotated and shared with online tools such as Mendeley and Zotero, and peer review is starting to be more open and transparent. The changing landscape of scholarly communication has also brought about new possibilities regarding its evaluation. So-called altmetrics are based on scholarly social media activity and have been introduced to reflect scholarly output and impact beyond considering only peer-reviewed journal articles and citations within them to measure scientific success. This includes the measurement of more diverse types of scholarly work and various forms of impact including that on society. This report provides an overview of how various social media tools are used in the research context based on 1) an extensive review of the current literature as well as 2) an empirical analysis of the use of Twitter by the 2010 cohort of SSHRC Doctoral Award recipients was analyzed in depth. Twitter has been chosen as one of the most promising tools regarding interaction with the general public and scholarly communication beyond the scientific community. The report focuses on the opportunities and challenges of social media and derived metrics and attempts to provide SSHRC with information to develop guidelines regarding the use of social media by funded researchers as well support the informed used of social media metrics

TWEETS OF AN ARTICLE AND ITS CITATION: AN ALTMETRIC STUDY OF MOST PROLIFIC AUTHORS

The present study was carried to find out the association between twitter and citation pattern for scholarly articles. This study was carried out with the most prolific authors of 2014 from the four subject domain “Clinical medicine, Microbiology, Molecular Biology, and Neuroscience” and 4886 papers were identified to studied their tweets and citation counts. From the study, it was found that the articles of the most prolific authors have a strong correlation with the citation and its value ρ =.518**. The linear relationship for individual subjects was between .386** to .559**, significant at .01 level

Interpreting “altmetrics": Viewing acts on social media through the lens of citation and social theories"

Merit, Expertise and Measuremen

Posted, Visited, Exported: Altmetrics in the Social Tagging System BibSonomy

In social tagging systems, like Mendeley, CiteULike, and BibSonomy, users can post, tag, visit, or export scholarly publications. In this paper, we compare citations with metrics derived from users’ activities (altmetrics) in the popular social bookmarking system BibSonomy. Our analysis, using a corpus of more than 250,000 publications published before 2010, reveals that overall, citations and altmetrics in BibSonomy are mildly correlated. Furthermore, grouping publications by user-generated tags results in topic-homogeneous subsets that exhibit higher correlations with citations than the full corpus. We find that posts, exports, and visits of publications are correlated with citations and even bear predictive power over future impact. Machine learning classifiers predict whether the number of citations that a publication receives in a year exceeds the median number of citations in that year, based on the usage counts of the preceding year. In that setup, a Random Forest predictor outperforms the baseline on average by seven percentage points

Mapping Scholarly Communication Infrastructure: A Bibliographic Scan of Digital Scholarly Communication Infrastructure

This bibliography scan covers a lot of ground. In it, I have attempted to capture relevant recent literature across the whole of the digital scholarly communications infrastructure. I have used that literature to identify significant projects and then document them with descriptions and basic information. Structurally, this review has three parts. In the first, I begin with a diagram showing the way the projects reviewed fit into the research workflow; then I cover a number of topics and functional areas related to digital scholarly communication. I make no attempt to be comprehensive, especially regarding the technical literature; rather, I have tried to identify major articles and reports, particularly those addressing the library community. The second part of this review is a list of projects or programs arranged by broad functional categories. The third part lists individual projects and the organizations—both commercial and nonprofit—that support them. I have identified 206 projects. Of these, 139 are nonprofit and 67 are commercial. There are 17 organizations that support multiple projects, and six of these—Artefactual Systems, Atypon/Wiley, Clarivate Analytics, Digital Science, Elsevier, and MDPI—are commercial. The remaining 11—Center for Open Science, Collaborative Knowledge Foundation (Coko), LYRASIS/DuraSpace, Educopia Institute, Internet Archive, JISC, OCLC, OpenAIRE, Open Access Button, Our Research (formerly Impactstory), and the Public Knowledge Project—are nonprofit.Andrew W. Mellon Foundatio

Mining, Modeling, and Leveraging Multidimensional Web Metrics to Support Scholarly Communities

The significant proliferation of scholarly output and the emergence of multidisciplinary research areas are rendering the research environment increasingly complex. In addition, an increasing number of researchers are using academic social networks to discover and store scholarly content. The spread of scientific discourse and research activities across the web, especially on social media platforms, suggests that far-reaching changes are taking place in scholarly communication and the geography of science. This dissertation provides integrated techniques and methods designed to address the information overload problem facing scholarly environments and to enhance the research process. There are four main contributions in this dissertation. First, this study identifies, quantifies, and analyzes international researchers’ dynamic scholarly information behaviors, activities, and needs, especially after the emergence of social media platforms. The findings based on qualitative and quantitative analysis report new scholarly patterns and reveals differences between researchers according to academic status and discipline. Second, this study mines massive scholarly datasets, models diverse multidimensional non-traditional web-based indicators (altmetrics), and evaluates and predicts scholarly and societal impact at various levels. The results address some of the limitations of traditional citation-based metrics and broaden the understanding and utilization of altmetrics. Third, this study recommends scholarly venues semantically related to researchers’ current interests. The results provide important up-to-the-minute signals that represent a closer reflection of research interests than post-publication usage-based metrics. Finally, this study develops a new scholarly framework by supporting the construction of online scholarly communities and bibliographies through reputation-based social collaboration, through the introduction of a collaborative, self-promoting system for users to advance their participation through analysis of the quality, timeliness and quantity of contributions. The framework improves the precision and quality of social reference management systems. By analyzing and modeling digital footprints, this dissertation provides a basis for tracking and documenting the impact of scholarship using new models that are more akin to reading breaking news than to watching a historical documentary made several years after the events it describes

Finding high-quality grey literature for use as evidence in software engineering research.

Background: Software engineering research often uses practitioners as a source of evidence in their studies. This evidence is usually gathered through empirical methods such as surveys, interviews and ethnographic research. The web has brought with it the emergence of the social programmer. Software practitioners are publishing their opinions online through blog articles, discussion boards and Q&A sites. Mining these online sources of information could provide a new source of evidence which complements traditional evidence sources. There are benefits to the adoption of grey literature in software engineering research (such as bridging the gap between the state–of–art where research typically operates and the state–of–practice), but also significant challenges. The main challenge is finding grey literature which is of high– quality to the researcher given the vast volume of grey literature available on the web. The thesis defines the quality of grey literature in terms of its relevance to the research being undertaken and its credibility. The thesis also focuses on a particular type of grey literature that has been written by soft- ware practitioners. A typical example of such grey literature is blog articles, which are specifically used as examples throughout the thesis. Objectives: There are two main objectives to the thesis; to investigate the problems of finding high–quality grey literature, and to make progress in addressing those problems. In working towards these objectives, we investigate our main research question, how can researchers more effectively and efficiently search for and then select the higher–quality blog–like content relevant to their research? We divide this question into twelve sub–questions, and more formally define what we mean by ‘blog–like content.’ Method: To achieve the objectives, we first investigate how software engineering researchers define and assess quality when working with grey literature; and then work towards a methodology and also a tool–suite which can semi–automate the identification and the quality assessment of relevant grey literature for use as evidence in the researchers study. To investigate how software engineering researchers define and assess quality, we first conduct a literature review of credibility assessment to gather a set of credibility criteria. We then validate those criteria through a survey of software engineering researchers. This gives us an overall model of credibility assessment within software engineering research. We next investigate the empirical challenges of measuring quality and develop a methodology which has been adapted from the case survey methodology and aims to address the problems and challenges identified. Along with the methodology is a suggested tool–suite which is intended to help researchers in automating the application of a subset of the credibility model. The tool–suite developed supports the methodology by, for example, automating tasks in order to scale the analysis. The use of the methodology and tool–suite is then demonstrated through three examples. These examples include a partial evaluation of the methodology and tool–suite. Results: Our literature review of credibility assessment identified a set of criteria that have been used in previous research. However, we also found a lack of definitions for both the criteria and, more generally, the term credibility. Credibility assessment is a difficult and subjective task that is particular to each individual. Research has addressed this subjectivity by conducting studies that look at how particular user groups assess credibility e.g. pensioners, university students, the visually impaired, however none of the studies reviewed software engineering researchers. Informed by the literature review, we conducted a survey which we believe is the first study on the credibility assessment of software engineering researchers. The results of the survey are a more refined set of criteria, but also a set that many (approximately 60%) of the survey participants believed generalise to other types of media (both practitioner–generated and researcher–generated). We found that there are significant challenges in using blog–like content as evidence in research. For example, there are the challenges of identifying the high–quality content from the vast quantity available on the web, and then creating methods of analysis which are scalable to handle that vast quantity. In addressing these challenges, we produce: a set of heuristics which can help in finding higher–quality results when searching using traditional search engines, a validated list of reasoning markers that can aid in assessing the amount of reasoning within a document, a review of the current state of the experience mining domain, and a modifiable classification schema for classifying the source of URLs. With credibility assessment being such a subjective task, there can be no one–size–fits–all method to automating quality assessment. Instead, our methodology is intended to be used as a framework in which the researcher using it can swap out and adapt the criteria that we assess for their own criteria based on the context of the study being undertaken and the personal preference of the researcher. We find from the survey that there are a variety of attitude’s towards using grey literature in software engineering research and not all respondents view the use of grey literature as evidence in the way that we do (i.e. as having the same benefits and threats as other traditional methods of evidence gathering). Conclusion: The work presented in this thesis makes significant progress towards answering our research question and the thesis provides a foundation for future research on automated quality assessment and credibility. Adoption of the tools and methodology presented in this thesis can help more effectively and efficiently search for and select higher–quality blog–like content, but there is a need for more substantial research on the credibility assessment of software engineering researchers, and a more extensive credibility model to be produced. This can be achieved through replicating the literature review systematically, accepting more studies for analysis, and by conducting a more extensive survey with a greater number, and more representative selection, of survey respondents. With a more robust credibility model, we can have more confidence in the criteria that we choose to include within the methodology and tools, as well as automating the assessment of more criteria. Throughout the re- search, there has been a challenge in aggregating the results after assessing each criterion. Future research should look towards the adoption of machine learning methods to aid with this aggregation. We believe that the criteria and measures used by our tools can serve as features to machine learning classifiers which will be able to more accurately assess quality. However, be- fore such work is to take place, there is a need for annotated data–sets to be developed