The rise of responsible metrics as a professional reform movement: a collective action frames account

Merit, Expertise and Measuremen

Quantified academic selves: The gamification of science through social networking services

Introduction. Our study critically engages with techniques of self-quantification in contemporary academia, by demonstrating how social networking services enact research and scholarly communication as a 'game'. Method. The empirical part of the study involves an analysis of two leading platforms: Impactstory and ResearchGate. Observed qualities of these platforms will be analyzed in detail with concrete examples of gaming features in focus. Subsequently, we relate the development of these digital platforms to a broader 'quantified self movement'. Special attention will also be paid to how these platforms contribute to a general quantification of the academic (authorial) self.  Theory. Theoretically we relate the 'gamification' of research to neoliberal ideas about markets and competition. Our analysis then extends to long-standing and fundamental ideas about self-betterment expressed in the philosophy of Peter Sloterdijk.  Findings. Our study shows how social networking services, such as ResearchGate and Impactstory, enact researchers as 'entrepreneurs of themselves' in a marketplace of ideas, and the quantification of scholarly reputation to a single number plays an important role in this process. Moreover, the technologies that afford these types of quantifiable interactions affect the 'unfolding ontology' of algorithmic academic identities.  Conclusions. The gamification of quantified academic selves intensifies the competitive nature of scholarship, it commodifies academic outputs and it might lead to goal displacement and cheating. However, self-quantification might also serve as a liberating and empowering activity for the individual researcher as alternative measures of impact and productivity are provided by these platforms.Merit, Expertise and Measuremen

New indicators and indexes for benchmarking university–industry–government innovation in medical and life science clusters: results from the European FP7 Regions of Knowledge HealthTIES project

Background: While the European Union is striving to become the ‘Innovation Union’, there remains a lack of quantifiable indicators to compare and benchmark regional innovation clusters. To address this issue, a HealthTIES (Healthcare, Technology and Innovation for Economic Success) consortium was funded by the European Union’s Regions of Knowledge initiative, research and innovation funding programme FP7. HealthTIES examined whether the health technology innovation cycle was functioning differently in five European regional innovation clusters and proposed regional and joint actions to improve their performance. The clusters included BioCat (Barcelona, Catalonia, Spain), Medical Delta (Leiden, Rotterdam and Delft, South Holland, Netherlands), Oxford and Thames Valley (United Kingdom), Life Science Zürich (Switzerland), and Innova Észak-Alföld (Debrecen, Hungary). Methods: Appreciation of the ‘triple helix’ of university–industry–government innovation provided the impetus for the development of two quantifiable innovation indexes and related indicators. The HealthTIES H-index is calculated for disease and technology platforms based on the h-index proposed by Hirsch. The HealthTIES Innovation Index is calculated for regions based on 32 relevant quantitative and discriminative indicators grouped into 12 categories and 3 innovation phases, namely ‘Input’ (n = 12), ‘Innovation System’ (n = 9) and ‘Output’ (n = 11). Results: The HealthTIES regions had developed relatively similar disease and technology platform profiles, yet with distinctive strengths and weaknesses. The regional profiles of the innovation cycle in each of the three phases were surprisingly divergent. Comparative assessments based on the indicators and indexes helped identify and share best practice and inform regional and joint action plans to strengthen the competitiveness of the HealthTIES regions. Conclusion: The HealthTIES indicators and indexes provide useful practical tools for the measurement and benchmarking of university–industry–government innovation in European medical and life science clusters. They are validated internally within the HealthTIES consortium and appear to have a degree of external prima facie validity. Potentially, the tools and accompanying analyses can be used beyond the HealthTIES consortium to inform other regional governments, researchers and, possibly, large companies searching for their next location, analyse and benchmark ‘triple helix’ dynamics within their own networks over time, and to develop integrated public–private and cross-regional research and innovation strategies in Europe and beyond

What's in a slogan? The meanings of translational science among academic cancer researchers in an English medical school

Merit, Expertise and Measuremen

The rise of ‘responsible metrics’ as a professional reform movement: a collective action frames perspective

Recent years have seen a rise in awareness around “responsible metrics” and calls for research assessment reforms internationally. Yet within the field of quantitative science studies and in research policy contexts, concerns about the limitations of evaluative bibliometrics are almost as old as the tools themselves. Given that many of the concerns articulated in recent reform movements go back decades, why has momentum for change grown only in the past ten years? In this paper, we draw on analytical insights from the sociology of social movements on collective action frames to chart the emergence, development, and expansion of “responsible metrics” as a professional reform movement. Through reviewing important texts that have shaped reform efforts, we argue that hitherto, three framings have underpinned the responsible metrics reform agenda: the metrics scepticism framing, the professional-expert framing, and the reflexivity framing. We suggest that while these three framings have co-existed within the responsible metrics movement to date, the “truce” negotiated between these framings may not last indefinitely, especially as the responsible metrics movement extends into wider research assessment reform movements.</p

All or nothing? Debating the role of evaluative bibliometrics in the research system

Merit, Expertise and Measuremen

Practicing responsible research assessment: qualitative study of faculty hiring, promotion, and tenure assessments in the United States

Merit, Expertise and Measuremen

To intervene, or not to intervene, is that the question? On the role of scientometrics in research evaluation

Recent high-profile statements, criticisms, and boycotts organized against certain quantitative indicators (e.g. the DORA declaration) have brought misuses of performance metrics to the center of attention. A key concern captured in these movements is that the metrics appear to carry authority even where established agents of quality control have explicitly outlined limits to their validity and reliability as measurement tools. This raises a number of challenging questions for those readers of this journal that are implicated in questions of indicator ‘production’ and, by extension, ‘effects’. In this opinion piece we wish to critically engage the question of how producers of indicators can come to terms with their role as (partly-) responsible parties in the current age of ‘evaluative bibliometrics’. We do so through the illuminating case of the professional scientometrics community. The article first describes how the landscape in which scientometrics operates has rapidly changed, becoming much more polyvocal. Inevitably this means more open competition for the attention space of users, and as a worst-case scenario the expertise of scientometrics risks being relegated to but one voice in a crowded marketplace. We then problematize how the community has often managed this relationship. We argue that the recourse taken thus far is towards an upstream solution, framed in terms that reinforce one’s own epistemic capabilities and professional position. This kind of boundary drawing between professional scientometrics and that of its intended audiences has contradictory effects, because it reproduces the gap between scientometric expertise and the practice level that the community is trying to bridge.Improving validity and reliability and standards development are useful endeavors, but if scientometricians draw the line of intervention here they risk - at best - maintaining the status quo in terms of how bibliometric products get (mis-) used in practice.The community can potentially draw on a large technical and social-scientific knowledge base, making them well placed to help analyze and change the ranges of conceivable types of actions and norms in current practices of research evaluation. The role of ‘objective outsiders’ who produce measures and standards but take no part in their intervention is no longer credible as a normative stance. In that particular arrangement the field situates itself almost entirely outside the practices of which scientometricians are also inherently part.Merit, Expertise and Measuremen

Plant-herbivore interactions in streams near Mt. St. Helens

1. In four separate field experiments near Mount St Helens (Washington, U.S.A.) during 1986, the grazing effects of two large benthic herbivores, tadpoles of the tailed frog Ascaphus truei and larvae of the caddisfly Dicosmoecus gilvipes, were investigated using streamside channels and in-stream manipulations. In the experimental channels, abundances of periphyton and small benthic invertebrates declined significantly with increasing density of these larger herbivores. 2. In eleven small, high-gradient streams affected to varying degrees by the May 1980 eruption, in-stream platforms were used to reduce grazing by A, truei tadpoles on tile substrates. Single platforms erected in each tributary and compared to grazed controls revealed only minor grazing effects, and no significant differences among streams varying in disturbance intensity (and, consequently, tadpole density). However, results probably were confounded by high variability among streams in factors other than tadpole abundance. 3. Grazing effects were further examined in two unshaded streams with different tadpole densities, using five platforms per stream. In the stream with five tadpoles m−2, grazing reduced periphyton biomass by 98% and chlorophyll a by 82%. In the stream lacking tadpoles, no significant grazing effects were revealed. Low algal abundance on both platforms and controls, and high invertebrate density in that stream (c. 30000m−2) suggests that grazing by small, vagile invertebrates was approximately equivalent to that of tadpoles. 4. The influence of large benthic herbivores on algal and invertebrate communities in streams of Mount St Helens can be important, but reponses vary spatially in relation to stream disturbance history, local environmental factors, and herbivore distributional patterns and abundanc