When competition is pushed too hard. An agent-based model of strategic behaviour of referees in peer review

This paper examines the impact of strategic behaviour of referees on the quality and efficiency of peer review. We modelled peer review as a process based on knowledge asymmetry and subject to evaluation bias. We built two simulation scenarios to investigate largescale implications of referee behaviour and judgment bias. The first one was inspired by "the luck of the reviewer draw" idea. In this case, we assumed that referees randomly fell into Type I and Type II errors, i.e., recommending submissions of low quality to be published or recommending against the publishing of submissions which should have been published. In the second scenario, we assumed that certain referees tried intentionally to outperform potential competitors by underrating the value of their submissions. We found that when publication selection increased, the presence of a minority of cheaters may dramatically undermine the quality and efficiency of peer review even compared with a scenario purely dominated by "the luck of the reviewer draw". We also found that peer review outcomes are significantly influenced by differences in the way scientists identify potential competitors in the system

Do editors have a silver bullet? an agent-based model of peer review

This paper presents an agent-based model of peer review that looks at the effect of different editorial policies of referee selection. We tested four author/referee matching scenarios as follows: random selection of referees, selection of referees with a similar status to submission authors, selection of higher-skilled and lower skilled referees. We tested these scenarios against three types of referee behaviour, ie., fair, unreliable and strategic and measured their implications for the quality and efficiency of the process. Results show that in case of randomness of referee judgment, any editorial policy is detrimental for peer review. If referees behave strategically, certain matching policies, such as selecting referees of good quality, might counteract possible bias

The transfer of western management to China: Context, content and constraints

This paper examines the issue of transferring Western management knowhow to China, focusing on the context, content and constraints of the transfer. A model of the transfer of management knowhow is developed which contains five closely-related stages: selection, adaptation, application, evaluation and integration. Factors which influence management transfer are identified as environmental, organisational, and cultural and behavioural; while cultural constraint are discussed in detail. It concludes that China cannot depend on the West for the solutions to its problems but develop its own indigenous knowhow by blending the best of both East and West

Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and execute an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We then use the collected data to conduct a series of real and semi-synthetic evaluations that demonstrate a strong detection power of our test

Against the Tide. A Critical Review by Scientists of How Physics and Astronomy Get Done

Nobody should have a monopoly of the truth in this universe. The censorship and suppression of challenging ideas against the tide of mainstream research, the blacklisting of scientists, for instance, is neither the best way to do and filter science, nor to promote progress in the human knowledge. The removal of good and novel ideas from the scientific stage is very detrimental to the pursuit of the truth. There are instances in which a mere unqualified belief can occasionally be converted into a generally accepted scientific theory through the screening action of refereed literature and meetings planned by the scientific organizing committees and through the distribution of funds controlled by "club opinions". It leads to unitary paradigms and unitary thinking not necessarily associated to the unique truth. This is the topic of this book: to critically analyze the problems of the official (and sometimes illicit) mechanisms under which current science (physics and astronomy in particular) is being administered and filtered today, along with the onerous consequences these mechanisms have on all of us.\ud \ud The authors, all of them professional researchers, reveal a pessimistic view of the miseries of the actual system, while a glimmer of hope remains in the "leitmotiv" claim towards the freedom in doing research and attaining an acceptable level of ethics in science

The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures

This paper looks at peer review as a cooperation dilemma through a game-theory framework. We built an agent-based model to estimate how much the quality of peer review is influenced by different resource allocation strategies followed by scientists dealing with multiple tasks, i.e., publishing and reviewing. We assumed that scientists were sensitive to acceptance or rejection of their manuscripts and the fairness of peer review to which they were exposed before reviewing. We also assumed that they could be realistic or excessively over-confident about the quality of their manuscripts when reviewing. Furthermore,\ua0we assumed they could be sensitive to competitive pressures provided by the institutional context in which they were embedded. Results showed that the bias and quality of publications greatly depend on reviewer motivations but also that context pressures can have a negative effect. However, while excessive competition can be detrimental to minimising publication bias, a certain level of competition is instrumental to ensure the high quality of publication especially when scientists accept reviewing for reciprocity motives

Government R&D funding: new approaches in the allocation policies for public and private beneficiaries

The objective of this paper is to perform a first experiment of quantitative assessment on changes in allocation mechanisms and in their underlying delegation models, using the quantitative information and the descriptions of national funding systems produced in the PRIME project funding activity. Delegation has been explored through changes in instrument portfolios and in evaluation modes, as proofs of an evolution in research governance. Some common trends can be identified: the reinforcing of both priority setting and peer review processes. The general result of our analysis is that some change in delegation modes took place, but there is not a simple transition from one delegation regime to another, while a "contract" delegation model (the NPM reform) is not detectable through project funding analysis.R/D funding, allocation policy, project funding, research governance, evaluation modes, delegation models

Analysis of Emerging Reputation and Funding Mechanisms in the Context of Open Science 2.0

This report covers the outcomes of two studies funded by JRC IPTS to explore emerging drivers for Open Science 2.0. In general, Open Science 2.0 is associated with themes such as open access to scientific outputs, open data, citizen science and open peer evaluation systems. This study, however, focused on less explored themes, namely on alternative funding mechanisms for scientific research and on emerging reputation mechanisms for scholars resulting from Web 2.0 platforms and applications. It has been demonstrated that both are providing significant new opportunities for researchers to disseminate, share, explore and collaborate with other researchers, but it remains to be seen whether they will be able to bring about more disruptive change in how science and research systems function in the future. They could well do so, especially if related changes being considered by the European Commission on ‘Science 2.0: Science in Transition’ are taken into account.JRC.J.3-Information Societ

Exploring Referee Abuse Through the Lens of the Collegiate Rugby Coach

Referees are a valuable contributor to the legitimacy of a sporting contest. Despite this, abuse in sport has become a growing concern and is regularly noted as an obstacle with which referees must contend. Continued abuses have been associated with referee discontinuation and have been noted as a key factor influencing the recruitment and retention of referees. Sporting organizations, coaches, and players all feel the impact when there is an inadequate supply of referees. Though the continuance of the referee is related to the continuance of the sport itself, little research has focused on the referee.Further, the studies that have considered referee abuse have focused only on the perspective of the referee seemingly ignoring the perceptions of other sporting stakeholders. Thus, the purpose of this study was to explore perceptions of referee abuse through the lens of the collegiate rugby coach. Through a phenomenological research design utilizing semi-structured telephone interviews, the topic of referee abuse in rugby was explored. Purposive sampling allowed for a selection of participants that had adequate knowledge of the phenomenon under investigation. Participants were solicited via USA Rugby utilizing a pre-scripted email provided by the researcher. During the data collection phase, saturation was reached as 15 participants completed two phone interviews. The findings, derived from participant data, identified 12 themes that addressed coach perceptions of referee abuse, the multitude of factors that impact perceptions of abuse, and numerous coach-generated solutions to help curtail the abuse of referees. From these findings, an Ecological Transactional Model of Factors Influencing Referee Abuse is proposed. This model categorizes the factors influencing referee abuse into various ecological levels, beginning with an individual’s personal characteristics (the ontological level) and continuing through their cultural existence (the macrosystem). The model suggests that these factors influence both perceptions of referee abuse and the solutions generated regarding referee abuse. The transactional portion of this model also highlights the interaction between various levels of one’s environment, proposing that the influences of numerous factors may exist at one time and that each of these factors may also influence one another. Future research is required to explore referee abuse in other contexts and to illuminate the perceptions of referee abuse through the lenses of additional sporting stakeholders. Recommendations for such research include the application of this proposed model to other sporting contexts, the consideration of generational differences on abuse perceptions, and a further exploration of cultural differences as related to perceptions of abuse. Violence prevention is currently a primary focus in many realms, including the entertainment industry, higher education, and sport. Studies of this nature help to ensure that referee abuse is included in the ongoing dialogue regarding violence prevention in sport