Anomalies in the peer-review system: A case study of the journal of High Energy Physics

Peer-review system has long been relied upon for bringing quality research to the notice of the scientific community and also preventing flawed research from entering into the literature. The need for the peer-review system has often been debated as in numerous cases it has failed in its task and in most of these cases editors and the reviewers were thought to be responsible for not being able to correctly judge the quality of the work. This raises a question "Can the peer-review system be improved?" Since editors and reviewers are the most important pillars of a reviewing system, we in this work, attempt to address a related question - given the editing/reviewing history of the editors or re- viewers "can we identify the under-performing ones?", with citations received by the edited/reviewed papers being used as proxy for quantifying performance. We term such review- ers and editors as anomalous and we believe identifying and removing them shall improve the performance of the peer- review system. Using a massive dataset of Journal of High Energy Physics (JHEP) consisting of 29k papers submitted between 1997 and 2015 with 95 editors and 4035 reviewers and their review history, we identify several factors which point to anomalous behavior of referees and editors. In fact the anomalous editors and reviewers account for 26.8% and 14.5% of the total editors and reviewers respectively and for most of these anomalous reviewers the performance degrades alarmingly over time.Comment: 25th ACM International Conference on Information and Knowledge Management (CIKM 2016

Referee assignment in the Chilean football league using integer programming and patterns

This article uses integer linear programming to address the referee assignment problem in the First Division of the Chilean professional football league. The proposed approach considers balance in the number of matches each referee must officiate, the frequency of each referee being assigned to a given team, the distance each referee must travel over the course of a season, and the appropriate pairings of referee experience or skill category with the importance of the matches. Two methodologies are studied, one traditional and the other a pattern-based formulation inspired by the home-away patterns for scheduling season match calendars. Both methodologies are tested in real-world and experimental instances, reporting results that improve significantly on the manual assignments. The pattern-based formulation attains major reductions in execution times, solving real instances to optimality in just a few seconds, while the traditional one takes anywhere from several minutes to more than an hour.Fil: Alarcón, Fernando. Universidad de Chile; ChileFil: Duran, Guillermo Alfredo. Universidad de Buenos Aires. Facultad de Ciencias Exactas y Naturales. Instituto de Cálculo; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas; ArgentinaFil: Guajardo, Mario. Norwegian School of Economics; Norueg

How Do Editors Select Papers, and How Good are They at Doing It?

Using data on the B.E. Journals that rank articles into four quality tiers, this paper examines the accuracy of the research evaluation process in economics. We find that submissions by authors with strong publication records and authors affiliated with highly-ranked institutions are significantly more likely to be published in higher tiers. Citation success as measured by RePEc statistics also depends heavily on the overall research records of the authors. Finally and most importantly, we measure how successful the B.E. Journals’ editors and their reviewers have been at assigning articles to quality tiers. While, on average, they are able to distinguish more influential from less influential manuscripts, we also observe many assignments that are not compatible with the belief that research quality is reflected by the number of citations.Peer Review, Research Evaluation, Citations, Journal Quality

Guest editorial: Special issue on matching under preferences

No abstract available

An Auction Market for Journal Articles

Economic articles are published very slowly. We believe this results from the poor incentives referees face. We recommend that an auction market replace the current, push system for submitting papers and demonstrate that our proposed market has a stable, Pareto-improving equilibrium. Besides the benefits of speed, this pull mechanism increases the quality of articles and journals and rewards referees for their effort. Although the auction price gives a prior on a paper's future value, its actual value|as a published article|depends on later citations. Since the auction price of later papers goes to the editors, authors and referees of earlier, cited articles, "auction earnings" give a direct measure of the value of articles, journals (the sum of articles) and academics - as authors, editors and reviewers - rewarding good writing, decisions and effort, respectively.Academic Journals;Academic Productivity;Market Design.

Mapping the Bid Behavior of Conference Referees

The peer-review process, in its present form, has been repeatedly criticized. Of the many critiques ranging from publication delays to referee bias, this paper will focus specifically on the issue of how submitted manuscripts are distributed to qualified referees. Unqualified referees, without the proper knowledge of a manuscript's domain, may reject a perfectly valid study or potentially more damaging, unknowingly accept a faulty or fraudulent result. In this paper, referee competence is analyzed with respect to referee bid data collected from the 2005 Joint Conference on Digital Libraries (JCDL). The analysis of the referee bid behavior provides a validation of the intuition that referees are bidding on conference submissions with regards to the subject domain of the submission. Unfortunately, this relationship is not strong and therefore suggests that there exists other factors beyond subject domain that may be influencing referees to bid for particular submissions