Peer-review system has long been relied upon for bringing quality research to
the notice of the scientific community and also preventing flawed research from
entering into the literature. The need for the peer-review system has often
been debated as in numerous cases it has failed in its task and in most of
these cases editors and the reviewers were thought to be responsible for not
being able to correctly judge the quality of the work. This raises a question
"Can the peer-review system be improved?" Since editors and reviewers are the
most important pillars of a reviewing system, we in this work, attempt to
address a related question - given the editing/reviewing history of the editors
or re- viewers "can we identify the under-performing ones?", with citations
received by the edited/reviewed papers being used as proxy for quantifying
performance. We term such review- ers and editors as anomalous and we believe
identifying and removing them shall improve the performance of the peer- review
system. Using a massive dataset of Journal of High Energy Physics (JHEP)
consisting of 29k papers submitted between 1997 and 2015 with 95 editors and
4035 reviewers and their review history, we identify several factors which
point to anomalous behavior of referees and editors. In fact the anomalous
editors and reviewers account for 26.8% and 14.5% of the total editors and
reviewers respectively and for most of these anomalous reviewers the
performance degrades alarmingly over time.Comment: 25th ACM International Conference on Information and Knowledge
Management (CIKM 2016
