Blind luck - could lotteries be a more efficient mechanism for allocating research funds than peer review?

Peer review is integral to the award of funds for academic research. However, as an increasingly large number of researchers attempt to secure limited funding, it is clear that much funding is awarded based on marginal assessments of the quality of different proposals. In this post, Lambros Roumbanis argues that randomly awarding research funding via lotteries presents a more rational, efficient and most importantly unbiased means of distributing research funding

The oracles of science : On grant peer review and competitive funding

From a purely epistemological point of view, evaluating and predicting the future success of new research projects is often considered very difficult. Is it possible to forecast important findings and breakthrough in science, and if not, then what is the point trying to do it anyway? Still, that is what funding agencies all over the world expect their reviewers to do, but a number of previous studies has shown that this form of evaluation of innovation, promise and future impact are a fundamentally uncertain and arbitrary practice. This is the context that I will discuss in the present essay, and I will claim that there is a deeply irrational element embedded in today's heavy reliance on experts to screen, rank and select among the increasing numbers of good research projects, because they can, in principal, never discern the true potential behind the written proposals. Hence, I think it is motivated to see grant peer review as an 'oracle of science'. My overall focus will be on the limits of competitive funding and also that the writing and reviewing of proposals is a waste of researchers' precious time. And I will propose that we really need to develop new ways of thinking about how we organize research and distribute opportunities within academia

New Arguments for a pure lottery in Research Funding : A Sketch for a Future Science Policy Without Time-Consuming Grant Competitions

A critical debate has blossomed within the field of research policy, science and technology studies, and philosophy of science regarding the possible benefits and limitations of allocating extramural grants using a lottery system. The most common view among those supporting the lottery idea is that some form of modified lottery is acceptable, if properly combined with peer review. This means that partial randomization can be applied only after experts have screened the pursuit-worthiness of all submitted proposals and sorted out those of lowest quality. In the present paper, I will argue against the use of partial lotteries or partial randomization and instead promote use of a pure lottery in combination with a radical increase in block funding. The main reason for holding this position is that a partial lottery cannot solve the problems inherent in the current funding system, which is based on grant competitions and peer review. A partial lottery cannot decrease the enormous time-waste, reduce the uneven distribution of time between researchers, neutralize expert biases or mitigate academic power asymmetries. Instead, we need a stronger focus on improving general time management in academia by implementing a more holistic model for organizing research opportunities in the future

[Reviewer Report] Comments by Roumbanis to Conix, De Block and Vaesen (2021) “Grant writing and grant peer review as questionable research practices”

Peer review report for Grant writing and grant peer review as questionable research practices [version 1; peer review: 1 approved].F1000Research 2021, 10:1126 (https://doi.org/10.5256/f1000research.77582.r99418)12 Nov 2021 for Version 1.</p

Symbolic Violence in Academic Life : A Study on How Junior Scholars are Educated in the Art of Getting Funded

It is widely recognized that universities all around Europe have taken on a more market-oriented approach that has changed the core of academic work life. This has led to a precarious situation for many junior scholars, who have to seek research funding to cover their own wages in an increasingly fierce competition over scarce resources. Thus, at present, research funding is a Gordian knot that must be cut by each individual researcher. As a response to this situation, some Swedish universities provide guidance to junior scholars on how to navigate in an increasingly entrepreneurial academia through open lectures by senior and successful professors. In this paper, I study these lectures as socialization processes and the role of symbolic violence in the justification of a competitive academic work ethos as well as a pragmatic acceptance of the prevailing funding conditions. The aim is to explore the role of a subtle form of power wielding that is not immediately understood or recognized as power, but that nonetheless reproduces a market-like behavior and legitimizes a career system marked by uncertainty, shortcomings and contradictions

Disagreement and Agonistic Chance in Peer Review

The purpose of grant peer review is to identify the most excellent and pro- mising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to the- orize the notions of academic judgment and agonistic chance and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong dis- agreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer-review process and becomes part of the consensus that is created

Tvetydigheter, oenighet och slumpens diskreta inflytande i peer review processen

Den moderna vetenskapen genomsyras av kollegial bedömning – peer review – som på olika sätt fungerar som en intern granskningsprocedur. När anslag ska fördelas till nya forskningsprojekt spelar denna sakkunnighetsbaserade instans en central roll i att generera kollektiviserade beslut. Bedömningen av forskningsansökningars kvalitet har dock visat sig innehålla ett ganska stort mått av tvetydigheter och slump som på ett ofrånkomligt sätt inverkar på de beslutsunderlag som skapas. Trots att peer review kretsar kring att de sakkunniga tillsammans eftersträvar konsensus, präglas denna konsensus likväl av stora variationer och oenighet om hur vissa ansökningar ska värderas. Genom att närmare studera hur oenighet kommer till uttryck och hanteras inom olika beredningsgrupper är det emellertid möjligt att få en fördjupad förståelse för den subtila relationen mellan slump och konsensus i peer review processen. Följande studie baseras på observationer gjorda på tio av Vetenskapsrådets beredningsgrupper inom natur- och teknikvetenskaper. Den metodologiska utgångspunkten som valts består i att illustrera ett antal fall av oenighet, för att därigenom belysa skillnaderna i hur man kom fram till en praktisk lösning. En problematik som särskilt kommer beröras har att göra med det faktum att oenighet påfallande ofta leder till att bedömarna slår ut varandras favoritansökningar. Detta tar i sin tur på den mer övergripande frågan om hur den enskildes expertkänsla i vissa situationer ställs i direkt motsättning till gruppens samlade expertis, något som kanske inte alltid medför de mest tillfredställande besluten i termer av innovation och risk

[Reviewer Report] Comments by Roumbanis to Conix, De Block and Vaesen (2021) “Grant writing and grant peer review as questionable research practices”

Peer review report for Grant writing and grant peer review as questionable research practices [version 1; peer review: 1 approved].F1000Research 2021, 10:1126 (https://doi.org/10.5256/f1000research.77582.r99418)12 Nov 2021 for Version 1.</p

Krönika : Den akademiska etiken och entreprenörhögskolans anda

Sociologisk Forsknings digitala arkiv</p

Akademiska omdömen under osäkerhet : en studie av kollektiva ankringseffekter i Vetenskapsrådets beredningsgrupper

The main focus of this study concerns the existence of anchoring effects in the peer review process of research proposals. According to the psychological theory of anchoring effects, once an anchor (represented by a number) is set, other judgments are made by adjusting away from the anchor, and there is a common human tendency toward interpreting other information around the anchor. This effect is commonly seen as a general flaw in human judgment, a cognitive bias that stems from a specific heuristic that guides people when they involve their intuition in order to solve a problem. In the following study, this cognitive bias will be analysed from a sociological point of view, that is, as a collective phenomenon rather than an individual one. It will also be applied to a new empirical case, the peer review process of research proposals. The anchoring effects have here been investigated using a meeting ethnographic approach. The evidence comes from direct observations of ten panel groups at the Swedish Research Council (Vetenskapsrådet). The analysis suggest that collective anchoring effects emerge as a result of the combination of the evaluation techniques that are being used (grading scales and middle ranking) and the efforts of the evaluators to reach consensus. The author suggests that what many commentators and evaluators have earlier interpreted as a certain proportion of chance in the peer review process, might instead be better understood as anchoring effects