Reviewing, Reviewers and the Scientific Enterprise

Despite their critical importance to the scientific enterprise, reviewers receive no formal training and reviewing has become a skill that they pick up through trial and error. Additionally, because most reviewers do not receive any feedback on their performance, any bad reviewing habits become entrenched over time. This has contributed to significant and unnecessary anxiety about reviewing and to antagonistic encounters between reviewers and authors. This paper seeks to correct this situation by defining reviewers as co-creators of scholarship and the reviewing as a quality control process in the production of scientific scholarship. The paper provides three groups of activities aimed at creating the right mindset among reviewers to facilitate this co-creation and quality control perspective: relationships, commitment and honest decisions and recommendations.reviewers, reviewing, scientific enterprise, scholarship, co-creations, Teaching/Communication/Extension/Profession,

Anomalies in the peer-review system: A case study of the journal of High Energy Physics

Peer-review system has long been relied upon for bringing quality research to the notice of the scientific community and also preventing flawed research from entering into the literature. The need for the peer-review system has often been debated as in numerous cases it has failed in its task and in most of these cases editors and the reviewers were thought to be responsible for not being able to correctly judge the quality of the work. This raises a question "Can the peer-review system be improved?" Since editors and reviewers are the most important pillars of a reviewing system, we in this work, attempt to address a related question - given the editing/reviewing history of the editors or re- viewers "can we identify the under-performing ones?", with citations received by the edited/reviewed papers being used as proxy for quantifying performance. We term such review- ers and editors as anomalous and we believe identifying and removing them shall improve the performance of the peer- review system. Using a massive dataset of Journal of High Energy Physics (JHEP) consisting of 29k papers submitted between 1997 and 2015 with 95 editors and 4035 reviewers and their review history, we identify several factors which point to anomalous behavior of referees and editors. In fact the anomalous editors and reviewers account for 26.8% and 14.5% of the total editors and reviewers respectively and for most of these anomalous reviewers the performance degrades alarmingly over time.Comment: 25th ACM International Conference on Information and Knowledge Management (CIKM 2016

Peer Review system: A Golden standard for publications process

Peer review process helps in evaluating and validating of research that is published in the journals. U.S. Office of Research Integrity reported that data fraudulence was found to be involved in 94% cases of misconduct from 228 identified articles between 1994–2012. If fraud in published article are significantly as high as reported, the question arise in mind, were these articles peer reviewed? Another report said that the reviewers failed to detect 16 cases of fabricated article of Jan Hendrick Schon. Superficial peer reviewing process does not reveals suspicion of misconduct. Lack of knowledge of systemic review process not only demolish the academic integrity in publication but also loss the trust of the people of the institution, the nation, and the world. The aim of this review article is to aware stakeholders specially novice reviewers about the peer review system. Beginners will understand how to review an article and they can justify better action choices in dealing with reviewing an article

Overburdening of peer review: A multi-stakeholder perspective on causes and effects

Peer review of manuscripts is labour-intensive and time-consuming. Individual reviewers might feel themselves overburdened with the amount of reviewing they are requested to do. Aiming to explore how stakeholder groups perceive reviewing burden and what they believe to be the causes of a potential overburdening of reviewers, we conducted focus groups with early-, mid-, and senior career scholars, editors, and publishers. By means of a thematic analysis, we aimed to identify the causes of overburdening of reviewers. First, we show that, across disciplines and roles, stakeholders believed that the reviewing burden is distributed unequally across members of the academic community, resulting in the overburdening of small groups of reviewers. Second, stakeholders believed this to be caused by (i) an increase in manuscript submissions; (ii) inefficient manuscript handling; (iii) lack of institutionalization of peer review; (iv) lack of reviewing instructions and (v) inadequate reviewer recruiting strategies. These themes were assumed to relate to an inadequate incentive structure in academia that favours publications over peer review. In order to alleviate reviewing burden, a holistic approach is required that addresses both the increased demand for and the insufficient supply of reviewing resources

How to Improve the Quality of Peer Reviews? Three Suggestions for System-level Changes

Peer reviewing is critical in the process of legitimizing new scientific knowledge. Yet, concerns about its quality exist, especially if one considers developmental reviewing as an ideal. In this essay, I suggest three ways to improve review quality: provide reviewers with systematic feedback about their performance, reward active and good reviewers, and make reviewers more accountable by revealing their identities to the authors in certain conditions

Reviewing the reviewers

In this article, the author reflects on his 12 years' experience of the academic refereeing process. In doing so, he aims to provide some encouragement to more junior academics faced with harsh and disheartening reviews, and perhaps to ask some reviewers to consider their approach to the review process. At its best, and done in a constructive manner, the review process can be a valuable learning experience for the author, with the finished product much improved as a result of constructive, informed criticism. However, on occasions reviews are marred by uninformed and/or unfair criticism, sometimes reflecting the reviewer's narrowness, or mode of thinking. Faced with such risks, authors must remain resilient and believe in the value of their contribution

Recognition and reward system for peer-reviewers

Peer reviewing plays an important role in the academic publishing process that scrutinizes and provides feedback on the scientific work prior to publication. Peer-reviewers put their efforts in reviewing others research work voluntarily, without any expectations of incentives or rewards. The peer-review process has been criticized for its defects like slowness, bias and abuse of the process. In this paper, we present a model to address these issues by using the approach of recording peer-review data on the blockchain. By using the semantic web and linked data technologies, this system would be able to expose its data and interact with other systems. This system will be used to quantify, recognize and incentivize the peer-reviewing efforts by researchers

Growth in the Amount of Literature Reviewed in a Meta-Analysis and Reviewer Resources

Accurate and consistent reviews of documents describing research studies are essential to valid and generalizable inferences in a meta-analysis. Traditionally a small number of reviewers screen studies using the title, abstract, and possibly the full text to determine a study\u27s eligibility for a meta-analysis. This study explores whether reviewing loads and resources to support accurate and consistent reviewing have increased over time. A survey of N = 193 meta-analyses published between 1980 – 2019 showed that the average number of documents reviewed has increased, especially since 2010, but the typical number of reviewers has not changed over the past 40 years. The importance of meta-analysts providing information about reviewers and the review process to help readers evaluate the validity and generalizability of inferences is emphasized. This information would typically include the number of reviewers and their qualifications, number of titles, abstracts, and full-text documents reviewed, time spent reviewing documents, evidence of the accuracy and consistency of reviews, and the role of software in facilitating reviewing