The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

Building a Disciplinary, World-Wide Data Infrastructure

Sharing scientific data, with the objective of making it fully discoverable, accessible, assessable, intelligible, usable, and interoperable, requires work at the disciplinary level to define in particular how the data should be formatted and described. Each discipline has its own organization and history as a starting point, and this paper explores the way a range of disciplines, namely materials science, crystallography, astronomy, earth sciences, humanities and linguistics get organized at the international level to tackle this question. In each case, the disciplinary culture with respect to data sharing, science drivers, organization and lessons learnt are briefly described, as well as the elements of the specific data infrastructure which are or could be shared with others. Commonalities and differences are assessed. Common key elements for success are identified: data sharing should be science driven; defining the disciplinary part of the interdisciplinary standards is mandatory but challenging; sharing of applications should accompany data sharing. Incentives such as journal and funding agency requirements are also similar. For all, it also appears that social aspects are more challenging than technological ones. Governance is more diverse, and linked to the discipline organization. CODATA, the RDA and the WDS can facilitate the establishment of disciplinary interoperability frameworks. Being problem-driven is also a key factor of success for building bridges to enable interdisciplinary research.Comment: Proceedings of the session "Building a disciplinary, world-wide data infrastructure" of SciDataCon 2016, held in Denver, CO, USA, 12-14 September 2016, to be published in ICSU CODATA Data Science Journal in 201

Do Open Access Articles Have a Greater Research Impact?

While many authors believe that their work has a greater research impact if it is freely available, studies to demonstrate that impact are few. This study looks at articles in four disciplines at varying stages of adoption of open access—philosophy, political science, electrical and electronic engineering and mathematics—to see if they have a greater impact, as measured by citations in the ISI Web of Science database, if their authors make them freely available on the Internet. The finding is that, across all four disciplines, freely available articles do have a greater research impact. Shedding light on this category of open access reveals that scholars in diverse disciplines are both adopting open access practices and being rewarded for it

Towards Convergence: How to Do Transdisciplinary Environmental Health Disparities Research.

Increasingly, funders (i.e., national, public funders, such as the National Institutes of Health and National Science Foundation in the U.S.) and scholars agree that single disciplines are ill equipped to study the pressing social, health, and environmental problems we face alone, particularly environmental exposures, increasing health disparities, and climate change. To better understand these pressing social problems, funders and scholars have advocated for transdisciplinary approaches in order to harness the analytical power of diverse and multiple disciplines to tackle these problems and improve our understanding. However, few studies look into how to conduct such research. To this end, this article provides a review of transdisciplinary science, particularly as it relates to environmental research and public health. To further the field, this article provides in-depth information on how to conduct transdisciplinary research. Using the case of a transdisciplinary, community-based, participatory action, environmental health disparities study in California's Central Valley provides an in-depth look at how to do transdisciplinary research. Working with researchers from the fields of social sciences, public health, biological engineering, and land, air, and water resources, this study aims to answer community residents' questions related to the health disparities they face due to environmental exposure. Through this case study, I articulate not only the logistics of how to conduct transdisciplinary research but also the logics. The implications for transdisciplinary methodologies in health disparity research are further discussed, particularly in the context of team science and convergence science

Hack Weeks as a model for Data Science Education and Collaboration

Across almost all scientific disciplines, the instruments that record our experimental data and the methods required for storage and data analysis are rapidly increasing in complexity. This gives rise to the need for scientific communities to adapt on shorter time scales than traditional university curricula allow for, and therefore requires new modes of knowledge transfer. The universal applicability of data science tools to a broad range of problems has generated new opportunities to foster exchange of ideas and computational workflows across disciplines. In recent years, hack weeks have emerged as an effective tool for fostering these exchanges by providing training in modern data analysis workflows. While there are variations in hack week implementation, all events consist of a common core of three components: tutorials in state-of-the-art methodology, peer-learning and project work in a collaborative environment. In this paper, we present the concept of a hack week in the larger context of scientific meetings and point out similarities and differences to traditional conferences. We motivate the need for such an event and present in detail its strengths and challenges. We find that hack weeks are successful at cultivating collaboration and the exchange of knowledge. Participants self-report that these events help them both in their day-to-day research as well as their careers. Based on our results, we conclude that hack weeks present an effective, easy-to-implement, fairly low-cost tool to positively impact data analysis literacy in academic disciplines, foster collaboration and cultivate best practices.Comment: 15 pages, 2 figures, submitted to PNAS, all relevant code available at https://github.com/uwescience/HackWeek-Writeu

Composing Possibilities: Open Educational Resources and K-12 Music Education

Music open educational resources (OER) have the potential to fill gaps in access to instructional materials for K-12 music teachers and learners, and to support teachers and learners as content creators and collaborators in meeting educational goals. This study explores the current state of music OER, the audiences that these resources serve to benefit, and the opportunities and challenges involved in adopting an open approach to the development of music education resources

Electronic doctoral theses in the UK: a sector-wide survey into policies, practice and barriers to Open Access

Sharing knowledge and research outputs is critical to the progress of science and human development, and a central tenet of academia. The Internet itself is a product of the academic community, and opening access to that community’s most important body of research, doctoral theses, is both a logical and an inevitable development. Progress toward open access to electronic theses has been slow in the UK. Much has been written on the perceived barriers and practical/infrastructural considerations that might explain this, but a comprehensive picture of that progress, and obstacles to it, was lacking. In 2010, a survey of policy and practice in UK HEIs was conducted by UCL (University College London) Library Services (commissioned by the Joint Information Systems Committee, JISC) to address this very issue. Incorporating inputs from 144 institutions currently awarding doctoral degrees, the work provides the first clear and detailed picture of the status of open access to doctoral research in the UK. The mission of the UK Council for Graduate Education (UKCGE) is to promote and support the interests of graduate education, and this it does through dissemination of best practice and intelligence on emergent trends; helping to shape policy and practice for the benefit of the UK HEI sector. This report contributes to that mission by bringing to the membership’s attention the results of this important work by UCL Library Services; a collaboration between UKCGE and the authors of the original work, it sets out the policies and practices that emerged from the survey and also considers what has been learned about the perceived barriers to the implementation of open access to electronic theses. The 2010 survey has enabled, for the first time, a differentiation to be made between barriers that are “real” and those which are unfounded and/or yet to be properly validated. At the same time, the work highlights the progress made in certain critical areas, as well as those that require our greater attention. A positive picture emerges for the UK on the adoption of the electronic thesis, with the majority of HEIs surveyed expected to be providing open access to their theses in five years’ time. A more detailed picture also emerges regarding the primary reasons for requests to restrict access to theses, some of which, notably, apply only to electronic (not print) theses. This has necessarily given rise to new policy developments. There is positive evidence also of collaboration among HEIs to provide an efficient and robust service for accessing electronic theses; pooling their resources and expertise either in the development of their institutional repositories or in operating a joint service. The key driver of open access to electronic theses is the opportunity for UK HEIs to “showcase” their research outputs to the widest possible audience and enhance their impact. There are no reliable means as yet to measure this impact, but there are encouraging early indications that electronic doctoral theses attract significant attention when made openly accessible. Open access to electronic theses may therefore indeed accelerate the sharing of knowledge and the progress of scientific discovery and human development