Introduction: Future pathways for science policy and research assessment: metrics vs peer review, quality vs impact

Copyright @ 2007 Beech Tree PublishingThe idea for this special issue arose from observing contrary developments in the design of national research assessment schemes in the UK and Australia during 2006 and 2007. Alternative pathways were being forged, determined, on the one hand, by the perceived relative merits of 'metrics' (quantitative measures of research performance) and peer judgement and, on the other hand, by the value attached to scientific excellence ('quality') versus usefulness ('impact'). This special issue presents a broad range of provocative academic opinion on preferred future pathways for science policy and research assessment. It unpacks the apparent dichotomies of metrics vs peer review and quality vs impact, and considers the hazards of adopting research evaluation policies in isolation from wider developments in scientometrics (the science of research evaluation) and divorced from the practical experience of other nations (policy learning)

Collaboration networks from a large CV database: dynamics, topology and bonus impact

Understanding the dynamics of research production and collaboration may reveal better strategies for scientific careers, academic institutions and funding agencies. Here we propose the use of a large and multidisciplinar database of scientific curricula in Brazil, namely, the Lattes Platform, to study patterns of scientific production and collaboration. In this database, detailed information about publications and researchers are made available by themselves so that coauthorship is unambiguous and individuals can be evaluated by scientific productivity, geographical location and field of expertise. Our results show that the collaboration network is growing exponentially for the last three decades, with a distribution of number of collaborators per researcher that approaches a power-law as the network gets older. Moreover, both the distributions of number of collaborators and production per researcher obey power-law behaviors, regardless of the geographical location or field, suggesting that the same universal mechanism might be responsible for network growth and productivity.We also show that the collaboration network under investigation displays a typical assortative mixing behavior, where teeming researchers (i.e., with high degree) tend to collaborate with others alike. Finally, our analysis reveals that the distinctive collaboration profile of researchers awarded with governmental scholarships suggests a strong bonus impact on their productivity.Comment: 8 pages, 8 figure

Throwing Out the Baby with the Bathwater: The Undesirable Effects of National Research Assessment Exercises on Research

The evaluation of the quality of research at a national level has become increasingly common. The UK has been at the forefront of this trend having undertaken many assessments since 1986, the latest being the “Research Excellence Framework” in 2014. The argument of this paper is that, whatever the intended results in terms of evaluating and improving research, there have been many, presumably unintended, results that are highly undesirable for research and the university community more generally. We situate our analysis using Bourdieu’s theory of cultural reproduction and then focus on the peculiarities of the 2008 RAE and the 2014 REF the rules of which allowed for, and indeed encouraged, significant game-playing on the part of striving universities. We conclude with practical recommendations to maintain the general intention of research assessment without the undesirable side-effects

"Open Innovation" and "Triple Helix" Models of Innovation: Can Synergy in Innovation Systems Be Measured?

The model of "Open Innovations" (OI) can be compared with the "Triple Helix of University-Industry-Government Relations" (TH) as attempts to find surplus value in bringing industrial innovation closer to public R&D. Whereas the firm is central in the model of OI, the TH adds multi-centeredness: in addition to firms, universities and (e.g., regional) governments can take leading roles in innovation eco-systems. In addition to the (transversal) technology transfer at each moment of time, one can focus on the dynamics in the feedback loops. Under specifiable conditions, feedback loops can be turned into feedforward ones that drive innovation eco-systems towards self-organization and the auto-catalytic generation of new options. The generation of options can be more important than historical realizations ("best practices") for the longer-term viability of knowledge-based innovation systems. A system without sufficient options, for example, is locked-in. The generation of redundancy -- the Triple Helix indicator -- can be used as a measure of unrealized but technologically feasible options given a historical configuration. Different coordination mechanisms (markets, policies, knowledge) provide different perspectives on the same information and thus generate redundancy. Increased redundancy not only stimulates innovation in an eco-system by reducing the prevailing uncertainty; it also enhances the synergy in and innovativeness of an innovation system.Comment: Journal of Open Innovations: Technology, Market and Complexity, 2(1) (2016) 1-12; doi:10.1186/s40852-016-0039-

A Systematic Identification and Analysis of Scientists on Twitter

Metrics derived from Twitter and other social media---often referred to as altmetrics---are increasingly used to estimate the broader social impacts of scholarship. Such efforts, however, may produce highly misleading results, as the entities that participate in conversations about science on these platforms are largely unknown. For instance, if altmetric activities are generated mainly by scientists, does it really capture broader social impacts of science? Here we present a systematic approach to identifying and analyzing scientists on Twitter. Our method can identify scientists across many disciplines, without relying on external bibliographic data, and be easily adapted to identify other stakeholder groups in science. We investigate the demographics, sharing behaviors, and interconnectivity of the identified scientists. We find that Twitter has been employed by scholars across the disciplinary spectrum, with an over-representation of social and computer and information scientists; under-representation of mathematical, physical, and life scientists; and a better representation of women compared to scholarly publishing. Analysis of the sharing of URLs reveals a distinct imprint of scholarly sites, yet only a small fraction of shared URLs are science-related. We find an assortative mixing with respect to disciplines in the networks between scientists, suggesting the maintenance of disciplinary walls in social media. Our work contributes to the literature both methodologically and conceptually---we provide new methods for disambiguating and identifying particular actors on social media and describing the behaviors of scientists, thus providing foundational information for the construction and use of indicators on the basis of social media metrics