Doing Peer Review: Reflections From an International Group of Postdoctoral Fellows

There is very little written regarding developing the skills of doing peer reviews. In this piece we use our own experience as postdoctoral fellows to offer our reflections on how to get the most out of doing peer reviews as a trainee researcher. We touch upon the variety and complexity of peer reviews, the debates concerning the nature and validity of peer reviews, the issue of conflict of interest, the menace of predatory journals, but also the potential gain from doing peer reviews. In sharing our reflections, we hope that future graduate students and postdoctoral fellows may be better prepared to do peer reviews and benefit from the experience

A Critical Examination of the Ethics of AI-Mediated Peer Review

Recent advancements in artificial intelligence (AI) systems, including large language models like ChatGPT, offer promise and peril for scholarly peer review. On the one hand, AI can enhance efficiency by addressing issues like long publication delays. On the other hand, it brings ethical and social concerns that could compromise the integrity of the peer review process and outcomes. However, human peer review systems are also fraught with related problems, such as biases, abuses, and a lack of transparency, which already diminish credibility. While there is increasing attention to the use of AI in peer review, discussions revolve mainly around plagiarism and authorship in academic journal publishing, ignoring the broader epistemic, social, cultural, and societal epistemic in which peer review is positioned. The legitimacy of AI-driven peer review hinges on the alignment with the scientific ethos, encompassing moral and epistemic norms that define appropriate conduct in the scholarly community. In this regard, there is a "norm-counternorm continuum," where the acceptability of AI in peer review is shaped by institutional logics, ethical practices, and internal regulatory mechanisms. The discussion here emphasizes the need to critically assess the legitimacy of AI-driven peer review, addressing the benefits and downsides relative to the broader epistemic, social, ethical, and regulatory factors that sculpt its implementation and impact.Comment: 21 pages, 1 figur

Practical Suggestions for Improving Scholarly Peer Review Quality and Reducing Cycle Times

Scholarly peer review is both central to scientific progress and deeply flawed. Peer review is prejudiced, capricious, inefficient, ineffective, and generally unscientific. Management journals have longer review cycles than journals in other fields. Long cycle times demonstrably harm early-career researchers. Meanwhile, a lack of transparency conceals and facilitates editorial misconduct, and some dismiss legitimate criticism of peer review as unfounded resentment. We can address these problems by eliminating unnecessary reviewing, simplifying the peer review process, introducing author rebuttals, creating an AIS ombudsman, and enforcing the relationship between submitting and reviewing. These problems are, however, entangled with fundamental problems with journals. Ultimately, therefore, we can only fix peer review in conjunction with replacing journals with repositories

Empirical Standards for Software Engineering Research

Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer Review Quality Initiative generated empirical standards for research methods commonly used in software engineering. These living documents, which should be continuously revised to reflect evolving consensus around research best practices, will improve research quality and make peer review more effective, reliable, transparent and fair.Comment: For the complete standards, supplements and other resources, see https://github.com/acmsigsoft/EmpiricalStandard