The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

How journal rankings can suppress interdisciplinary research. A comparison between Innovation Studies and Business & Management

This study provides quantitative evidence on how the use of journal rankings can disadvantage interdisciplinary research in research evaluations. Using publication and citation data, it compares the degree of interdisciplinarity and the research performance of a number of Innovation Studies units with that of leading Business & Management schools in the UK. On the basis of various mappings and metrics, this study shows that: (i) Innovation Studies units are consistently more interdisciplinary in their research than Business & Management schools; (ii) the top journals in the Association of Business Schools' rankings span a less diverse set of disciplines than lower-ranked journals; (iii) this results in a more favourable assessment of the performance of Business & Management schools, which are more disciplinary-focused. This citation-based analysis challenges the journal ranking-based assessment. In short, the investigation illustrates how ostensibly 'excellence-based' journal rankings exhibit a systematic bias in favour of mono-disciplinary research. The paper concludes with a discussion of implications of these phenomena, in particular how the bias is likely to affect negatively the evaluation and associated financial resourcing of interdisciplinary research organisations, and may result in researchers becoming more compliant with disciplinary authority over time.Comment: 41 pages, 10 figure

Building a Disciplinary, World-Wide Data Infrastructure

Sharing scientific data, with the objective of making it fully discoverable, accessible, assessable, intelligible, usable, and interoperable, requires work at the disciplinary level to define in particular how the data should be formatted and described. Each discipline has its own organization and history as a starting point, and this paper explores the way a range of disciplines, namely materials science, crystallography, astronomy, earth sciences, humanities and linguistics get organized at the international level to tackle this question. In each case, the disciplinary culture with respect to data sharing, science drivers, organization and lessons learnt are briefly described, as well as the elements of the specific data infrastructure which are or could be shared with others. Commonalities and differences are assessed. Common key elements for success are identified: data sharing should be science driven; defining the disciplinary part of the interdisciplinary standards is mandatory but challenging; sharing of applications should accompany data sharing. Incentives such as journal and funding agency requirements are also similar. For all, it also appears that social aspects are more challenging than technological ones. Governance is more diverse, and linked to the discipline organization. CODATA, the RDA and the WDS can facilitate the establishment of disciplinary interoperability frameworks. Being problem-driven is also a key factor of success for building bridges to enable interdisciplinary research.Comment: Proceedings of the session "Building a disciplinary, world-wide data infrastructure" of SciDataCon 2016, held in Denver, CO, USA, 12-14 September 2016, to be published in ICSU CODATA Data Science Journal in 201

Building a Model of Collaboration Between Historically Black and Historically White Universities

Despite increases over the last two decades in the number of degrees awarded to students from underrepresented groups in science, technology, engineering, and mathematics (STEM) disciplines, enhancing diversity in these disciplines remains a challenge. This article describes a strategic approach to this challenge—the development of a collaborative partnership between two universities: the historically Black Elizabeth City State University and the historically White University of New Hampshire. The partnership, a type of learning organization built on three mutually agreed upon principles, strives to enhance opportunities for underrepresented students to pursue careers in the STEM disciplines. This article further describes six promising practices that framed the partnership, which resulted in the submission of nine proposals to federal agencies and the funding of four grants that led to the implementation, research, learning, and evaluation that followed

The selection, appraisal and retention of digital scientific data: dighlights of an ERPANET/CODATA workshop

CODATA and ERPANET collaborated to convene an international archiving workshop on the selection, appraisal, and retention of digital scientific data, which was held on 15-17 December 2003 at the Biblioteca Nacional in Lisbon, Portugal. The workshop brought together more than 65 researchers, data and information managers, archivists, and librarians from 13 countries to discuss the issues involved in making critical decisions regarding the long-term preservation of the scientific record. One of the major aims for this workshop was to provide an international forum to exchange information about data archiving policies and practices across different scientific, institutional, and national contexts. Highlights from the workshop discussions are presented

Mapping Diversity of Publication Patterns in the Social Sciences and Humanities: An Approach Making Use of Fuzzy Cluster Analysis

&lt;b&gt;Purpose:&lt;/b&gt; To present a method for systematically mapping diversity of publication patterns at the author level in the social sciences and humanities in terms of publication type, publication language and co-authorship.&lt;br&gt;&lt;b&gt;Design/methodology/approach:&lt;/b&gt; In a follow-up to the hard partitioning clustering by Verleysen and Weeren in 2016, we now propose the complementary use of fuzzy cluster analysis, making use of a membership coefficient to study gradual differences between publication styles among authors within a scholarly discipline. The analysis of the probability density function of the membership coefficient allows to assess the distribution of publication styles within and between disciplines.&lt;br&gt;&lt;b&gt;Findings:&lt;/b&gt; As an illustration we analyze 1,828 productive authors affiliated in Flanders, Belgium. Whereas a hard partitioning previously identified two broad publication styles, an international one vs. a domestic one, fuzzy analysis now shows gradual differences among authors. Internal diversity also varies across disciplines and can be explained by researchers&#39; specialization and dissemination strategies.&lt;br&gt;&lt;b&gt;Research limitations:&lt;/b&gt; The dataset used is limited to one country for the years 2000-2011; a cognitive classification of authors may yield a different result from the affiliation-based classification used here.&lt;br&gt;&lt;b&gt;Practical implications:&lt;/b&gt; Our method is applicable to other bibliometric and research evaluation contexts, especially for the social sciences and humanities in non-Anglophone countries.&lt;br&gt;&lt;b&gt;Originality/value:&lt;/b&gt; The method proposed is a novel application of cluster analysis to the field of bibliometrics. Applied to publication patterns at the author level in the social sciences and humanities, for the first time it systematically documents intra-disciplinary diversity.&lt;b&gt;Purpose:&lt;/b&gt; To present a method for systematically mapping diversity of publication patterns at the author level in the social sciences and humanities in terms of publication type, publication language and co-authorship.&lt;br&gt;&lt;b&gt;Design/methodology/approach:&lt;/b&gt; In a follow-up to the hard partitioning clustering by Verleysen and Weeren in 2016, we now propose the complementary use of fuzzy cluster analysis, making use of a membership coefficient to study gradual differences between publication styles among authors within a scholarly discipline. The analysis of the probability density function of the membership coefficient allows to assess the distribution of publication styles within and between disciplines.&lt;br&gt;&lt;b&gt;Findings:&lt;/b&gt; As an illustration we analyze 1,828 productive authors affiliated in Flanders, Belgium. Whereas a hard partitioning previously identified two broad publication styles, an international one vs. a domestic one, fuzzy analysis now shows gradual differences among authors. Internal diversity also varies across disciplines and can be explained by researchers&#39; specialization and dissemination strategies.&lt;br&gt;&lt;b&gt;Research limitations:&lt;/b&gt; The dataset used is limited to one country for the years 2000-2011; a cognitive classification of authors may yield a different result from the affiliation-based classification used here.&lt;br&gt;&lt;b&gt;Practical implications:&lt;/b&gt; Our method is applicable to other bibliometric and research evaluation contexts, especially for the social sciences and humanities in non-Anglophone countries.&lt;br&gt;&lt;b&gt;Originality/value:&lt;/b&gt; The method proposed is a novel application of cluster analysis to the field of bibliometrics. Applied to publication patterns at the author level in the social sciences and humanities, for the first time it systematically documents intra-disciplinary diversity.</span

The effect of multidisciplinary collaborations on research diversification

This work verifies whether research diversification by a scientist is in some measure related to their collaboration with multidisciplinary teams. The analysis considers the publications achieved by 5300 Italian academics in the sciences over the period 2004-2008. The findings show that a scientist's outputs resulting from research diversification are more often than not the result of collaborations with multidisciplinary teams. The effect becomes more pronounced with larger and particularly with more diversified teams. This phenomenon is observed both at the overall level and for the disciplinary macro-areas