Mapping the Bid Behavior of Conference Referees

The peer-review process, in its present form, has been repeatedly criticized. Of the many critiques ranging from publication delays to referee bias, this paper will focus specifically on the issue of how submitted manuscripts are distributed to qualified referees. Unqualified referees, without the proper knowledge of a manuscript's domain, may reject a perfectly valid study or potentially more damaging, unknowingly accept a faulty or fraudulent result. In this paper, referee competence is analyzed with respect to referee bid data collected from the 2005 Joint Conference on Digital Libraries (JCDL). The analysis of the referee bid behavior provides a validation of the intuition that referees are bidding on conference submissions with regards to the subject domain of the submission. Unfortunately, this relationship is not strong and therefore suggests that there exists other factors beyond subject domain that may be influencing referees to bid for particular submissions

Favouritism and financial incentives: A natural experiment

Principals who exercise favouritism towards certain agents may harm those who are not so favoured. Other papers have produced evidence consistent with the presence of such favouritism but have been unable to consider methods for controlling it. We address this issue in the context of a natural experiment from English soccer, where one particular league introduced professional referees in 2001-02, thereby changing the financial incentives and monitoring regime faced by these referees. Because the change was not effected in all leagues, the ‘experiment’ has both cross-sectional and intertemporal dimensions. We study the effects of professional referees on an established measure of referee bias: length of injury time in close matches. We find that referees exercised favouritism prior to professionalism but not afterwards, having controlled for selection and soccer-wide effects. The results are consistent with a financial incentive effect as a result of professional referees and indicate that subtle aspects of principal-agent relationships (such as favouritism) are amenable to contractual influence.Favouritism, financial incentives, soccer, referee

Peer-review in a world with rational scientists: Toward selection of the average

One of the virtues of peer review is that it provides a self-regulating selection mechanism for scientific work, papers and projects. Peer review as a selection mechanism is hard to evaluate in terms of its efficiency. Serious efforts to understand its strengths and weaknesses have not yet lead to clear answers. In theory peer review works if the involved parties (editors and referees) conform to a set of requirements, such as love for high quality science, objectiveness, and absence of biases, nepotism, friend and clique networks, selfishness, etc. If these requirements are violated, what is the effect on the selection of high quality work? We study this question with a simple agent based model. In particular we are interested in the effects of rational referees, who might not have any incentive to see high quality work other than their own published or promoted. We find that a small fraction of incorrect (selfish or rational) referees can drastically reduce the quality of the published (accepted) scientific standard. We quantify the fraction for which peer review will no longer select better than pure chance. Decline of quality of accepted scientific work is shown as a function of the fraction of rational and unqualified referees. We show how a simple quality-increasing policy of e.g. a journal can lead to a loss in overall scientific quality, and how mutual support-networks of authors and referees deteriorate the system.Comment: 5 pages 4 figure

Asymptotics of the two-stage spatial sign correlation

Acknowledgments This research was supported in part by the Collaborative Research Grant 823 of the German Research Foundation. The authors wish to thank the editors and referees for their careful handling of the manuscript. They further acknowledge the anonymous referees of the article Spatial sign correlation (J. Multivariate Anal. 135, pages 89–105, 2015), who independently of each other suggested to further explore the properties of two-stage spatial sign correlation.Non peer reviewedPreprin

An Auction Market for Journal Articles

Economic articles are published very slowly. We believe this results mainly from the poor incentives referees face. We recommend that an auction market replace the current system for submitting papers and demonstrate a strict Pareto-improvement of equilibrium. Besides the benefits of speed, this mechanism increases the average quality of articles and journals and rewards editors and referees for their effort. In addition, the "academic dollars" for papers sold at auction go to the authors, editors and referees of cited articles. This income indicates academic productivity (facilitating decisions on tenure and promotion); its recirculation to journals further stimulates quality competition.Academic Journals;Academic Productivity;Market Design. JEL codes