Currently, there is little exchange between the different communities interested in the domain of bibliometrics. A recent conference aimed to bridge this gap. Peter Kraker, Katrin Weller, Isabella Peters and Elisabeth Lex report on the multitude of topics and viewpoints covered on the quantitative analysis of scientific research. A key theme was the strong need for more openness and transparency: transparency in research evaluation processes to avoid biases, transparency of algorithms that compute new scores and openness of useful technology

Kraker, Peter

Lex, Elisabeth

Peters, Isabella

Weller, Katrin

LSE Research Online

12/4/2015Bringing together bibliometrics research from differentdisciplines – what can we learn from each other?blogs.lse.ac.uk/impactofsocialsciences/2015/12/04/bringing-together-bibliometrics-research-from-different-disciplines/Currently, there is little exchange between the different communities interested in thedomain of bibliometrics. A recent conference aimed to bridge this gap. PeterKraker, Katrin Weller, Isabella Peters and Elisabeth Lex report on the multitude oftopics and viewpoints covered on the quantitative analysis of scientific research. A keytheme was the strong need for more openness and transparency: transparency inresearch evaluation processes to avoid biases, transparency of algorithms that computenew scores and openness of useful technology.We are currently witnessing a transition in scholarly communication from an offline,paper-based mode to a digital and online mode. New and complementary means ofcommunication have arisen in the form of social networks and other online collaborative environments. New formsof communicating also pose new challenges to the field of bibliometrics, the science of studying and measuringscholarly communication and scholarly reputation. Key concepts of bibliometrics (such as citation indexes, co-citation analysis, maps of science etc.) are being transferred to the new Web environments. This development alsoled to the formation of the term altmetrics to describe the study of alternative indicators for scholarly communication,such as social media mentions (e.g. tweets) referring to scholarly articles or bookmarks in academic socialbookmarking systems.Given these developments, it is hardly surprising that we see an increased interest in bibliometrics from computerscientists and related disciplines. In addition, there has been a longstanding tradition of bibliometric research inseveral fields, with the medical and the physics community being probably the most active. Yet, basic theories andmodels relevant for bibliometrics are developed in science and technology studies (STS).At the same time, metrics are becoming ubiquitous in research assessment, as evidenced in the recent HEFCEreport “The Metric Tide”. Even though researchers have called for sensible use of metrics in research evaluation,e.g. in the San Francisco Declaration of Research Evaluation DORA, and endeavours promoting researcherportfolios, in which the researchers themselves choose which of their research output is important and for whatreason (e.g. ACUMEN), metrics are more widespread than ever, even in public discourse. Drivers of thisdevelopment are, amongst others, policy makers that increasingly ask for evidence on the impact of research.The workshopIn the wake of these developments, the bibliometrics community has become more committed to informingstakeholders on practical implications of its research, exemplified by the recently published Leiden Manifesto forresearch metrics. With bibliometrics research being more open and outward and increasingly the center of attentionof many disciplines, it seemed more than apt to bring together different communities that do bibliometrics researchto find out what they can learn from each other.1/5Image credit: Jopa Elleul CC BY-NCThis was the goal of the workshop “Quantifying and Analysing Scholarly Communication on the Web (ASCW’15)” ,which was made clear by its tagline: “What can bibliometrics do for you? What can you do for bibliometrics?”. Whenstudying Web-based scholarly communication, bibliometricians increasingly need skills from computer science, suchas collecting and processing large amounts of heterogeneous data. On the other hand, computer scientists may notbe aware of the issues of quantitative analyses of scholarly communication and would benefit from the knowledge ofinformation scientists. Currently, there is little exchange between the different communities interested in the domainof bibliometrics, which is exemplified by countless parallel research efforts when it comes to supporting andunderstanding scholarly communication on the Web. This effect is reinforced by the diverse publication outlets thedifferent communities serve.The workshop aimed at bridging the gap and was therefore held at this year’s Web Science Conference in Oxford.The conference has a distinct interdisciplinary approach to studying phenomena on the Web, and has also servedas an incubator for the altmetrics community in the past. We organized the workshop with two principles in mind:open science and a focus on discourse. We first called for the submission of position papers, which were publishedon our website after an initial editorial check. Then, we invited our program committee and the whole researchcommunity to an open peer review. In a parallel effort, we invited response papers to all position papers. All papersand responses are available under a Creative Commons license from our website here.Which issues were discussed?We received a number of highly relevant  contributions from a variety of disciplines (psychology, computer science,political science, educational technology, bibliometrics) that lead up to an equally diverse workshop.Under the theme “What can bibliometrics do for you?”, Alan Dix (Talis & University of Birmingham) kicked off with acitation analysis of the results of the 2014 edition of UK’s Research Excellence Framework (REF2014), which issolely based on expert assessment. In his analysis of the discipline computer science, Alan found that there was alatent bias against applied research fields such as web science. In the expert assessment, these fields had receivedfar fewer 4* ratings (indicating world-class research) than what citation analysis would have predicted. Vice versa,more theoretical areas such as logic fared much better in the expert assessment than what the citation analysiswould have predicted. In his response, Robert Jäschke (L3S), discussed some of the possible explanations for this2/5latent bias, such as a halo effect when assessing papers from institutions that are perceived to be excellent. Hesuggested to extend this research to other disciplines in the REF, and to involve altmetrics in the evaluation.Next up, Peter Kraker (Know-Center) presented a critical assessment of the ResearchGate Score , which was co-authored by Elisabeth Lex (Graz University of Technology). In the evaluation, they found that the ResearchGateScore has serious shortcomings, including that the score is intransparent and that the algorithm is changing overtime with no indication as to what had changed. Furthermore, it includes the much-criticized journal impact factor toevaluate individual researchers. They conclude that the ResearchGate Score should not be considered in theevaluation of academics in its current form. In her response, Katy Jordan showed that even though theResearchGate Score is intransparent, it is possible to reverse-engineer the score to a large extent using multipleregression. She presented a model that explains over 95% of the score.After these talks, we switched to the theme “What can you do for bibliometrics?” to see how current approaches fromcomputer science are finding their way into bibliometric research. Ryan Whalen (Northwestern University) presenteda paper which was co-written with his colleagues from Northwestern University Yun Huang, Anup Sawant, BrianUzzi, and Noshir Contractor. They advocate the use of textual data and natural language processing methods todevelop new bibliometric methods. To illustrate their point, they show how accounting for topical diversity of citationscan improve impact predictions: citations from both topically distant and proximate papers provide more insight intoan article’s impact potential than those from papers with middling similarity. In his response, Brett Buttliere(Knowledge Media Research Center) provided a number of ways how this method may be improved and expandedon. For example, he suggested using social media data and the keywords therein following Whalen’s method. Healso discussed the potential benefits of using more theory from fields such as psychology in bibliometric researchand how to nuance our idea of impact to more than one dimension to create better metrics. He expanded on thesethoughts in a recent LSE Impact Blog post.Here is the overview of all workshop papers with their responses:Dix, A: Citations and Sub-Area Bias in the UK Research Assessment ProcessResponse: Jäschke, R: Response to the Submission “Citations and Sub-Area Bias in the UK ResearchAssessment Process” by Alan DixKraker, P. & Lex, E.: A Critical Look at the ResearchGate Score as a Measure of Scientific ReputationJordan, K: Exploring the ResearchGate score as an academic metric: Reflections and implications forpracticeWhalen, R., Huang, Y., Sawant, A., Uzzi, B. & Contractor, N.: Natural Language Processing, Article Content &Bibliometrics: Predicting High Impact ScienceButtliere, B: Identifying high impact scientific work using Natural Language Processing and PsychologyGeneral themes emerging from the discussion and conclusionsThe three presentations helped to reflect upon three core players in the current discussion of bibliometrics andaltmetrics: policy makers, platform providers, and researchers – both as the subjects of bibliometric analyses and asthe conductors of bibliometric research. ASCW’15 mainly assembled the third group, whereas extending thedialogue to include policy makers and platform providers has been approached in other venues. But this eventshowed that once you put metrics researchers from different disciplines in the same room, good things happen. Despite – or more likely because of – the multitude of topics and different viewpoints, the exchange on how to furtherthe field of quantitative analysis of science proved to be very fruitful.Bibliometric indicators and computational approaches were critically discussed at a fine-grained level, including3/5topics such as hidden biases, intentional gaming of indicators, and lack of reflection on different motivations forcitation behavior. In the end, this line of discussion more than once concluded with the demand for more opennessand transparency: transparency in research evaluation processes to avoid biases, transparency of algorithms thatcompute new scores behind the closed doors of commercial platform providers, openness of useful technology (e.g.from Semantic Web research, including named entity recognition or sentiment analyses) to be reused in newcontexts.Another recurring theme was the idea to involve broader (user) communities to ensure quality standards, e.g. tohave a community that alerts platform providers such as ResearchGate or Mendeley of algorithmic mismatches andwrongly extracted metadata. Another idea was to enrich research assessment with expertise of the crowd.One of the main conclusions was that metrics will gain in importance and that it is our responsibility as metricsresearchers to take part in the ongoing discussion, and to remind other stakeholders of the lack of transparency andits potential consequences. Knowledge on parallel research efforts and approaches stemming from other disciplinesis also critical since orchestrated endeavours would be more efficient and effective – given the diversity of fields andstakeholders that metrics are applied to. Another main conclusion was that more disciplinary and interdisciplinaryresearch is needed to provide decision makers, funding agencies and politicians with the necessary insights tomake informed decisions. Therefore, we want to continue the interdisciplinary exchange in further events andpublication outlets. Also, watch out for further posts from the workshop – we hope to talk to you soon!Note: This article gives the views of the author, and not the position of the LSE Impact blog, nor of the LondonSchool of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.About the AuthorsPeter Kraker is a postdoctoral researcher at Know-Center of Graz University of Technology and a 2013/14 PantonFellow. His main research interests are visualizations based on scholarly communication on the web, open science,and altmetrics. Peter is an open science advocate collaborating with the Open Knowledge Foundation and the OpenAccess Network Austria.Katrin Weller is a postdoctoral researcher at GESIS Leibniz Institute for the Social Sciences, department ofComputational Social Science. She studies social media users and uses in different contexts, including the use ofsocial media for scholarly communication. Katrin was one of the inaugural Digital Studies Fellows at the Library ofCongress’ John W. Kluge Center.Isabella Peters is Professor of Web Science at ZBW Leibniz Information Centre for Economics and Kiel University.In her research she focuses on scholarly communication in social media environments as well as the evaluation ofresearch outputs by incorporating user-generated content.Elisabeth Lex is assistant professor at Graz University of Technology and she heads the Social Computing researcharea at Know-Center GmbH. In her research, she explores how digital traces humans leave behind on the Web canbe exploited to model and shape the way people work, learn and interact. At Graz University of Technology,Elisabeth teaches Web Science as well as Science 2.0.This is part of a series of pieces from the Quantifying and Analysing Scholarly Communication on the Webworkshop. More from this series:4/5The ResearchGate Score: a good example of a bad metricAccording to ResearchGate, the academic social networking site, their RG Score is “a new way tomeasure your scientific reputation”. With such high aims, Peter Kraker, Katy Jordanand Elisabeth Lex take a closer look at the opaque metric. By reverse engineering the score, theyfind that a significant weight is linked to ‘impact points’ – a similar metric to the widely discreditedjournal impact factor.Transparency in metrics is the only way scholarly measures can be put intocontext and the only way biases – which are inherent in all socially created metrics – can beuncovered.We need informative metrics that will help, not hurt, the scientific endeavor –let’s work to make metrics better.Rather than expecting people to stop utilizing metrics altogether, we would be better off focusing onmaking sure the metrics are effective and accurate, argues Brett Buttliere. By looking across avariety of indicators, supporting a centralised, interoperable metrics hub, and utilizing more theoryin building metrics, scientists can better understand the diverse facets of research impact andresearch quality.Context is everything: Making the case for more nuanced citation impactmeasures.Access to more and more publication and citation data offers the potential for more powerful impactmeasures than traditional bibliometrics. Accounting for more of the context in the relationshipbetween the citing and cited publications could provide more subtle and nuanced impactmeasurement. Ryan Whalen looks at the different ways that scientific content are related, and howthese relationships could be explored further to improve measures of scientific impact.Copyright © The Author (or The Authors) - Unless otherwise stated, this work is licensed under a CreativeCommons Attribution Unported 3.0 License.5/5

Bringing together bibliometrics research from different disciplines – what can we learn from each other?

Abstract

Similar works

Full text

Available Versions

LSE Research Online