Assessing Peer Review by Gauging the Fate of Rejected Manuscripts: the case of the Journal of Artificial Societies and Social Simulation

This paper investigates the fate of manuscripts that were rejected from JASSS-The Journal of Artificial Societies and Social Simulation, the flagship journal of social simulation. We tracked 456 manuscripts that were rejected from 1997 to 2011 and traced their subsequent publication as journal articles, conference papers or working papers. We compared the impact factor of the publishing journal and the citations of those manuscripts that were eventually published against the yearly impact factor of JASSS and the number of citations achieved by the JASSS mean and top cited articles. Only 10% of the rejected manuscripts were eventually published in a journal that was indexed in the Web of Science, although most of the rejected manuscripts were published elsewhere. Being exposed to more than one round of reviews before rejection, having received a more detailed reviewer report and being subjected to higher inter-reviewer disagreement were all associated with the number of citations received when the manuscript was eventually published. This indicates that peer review could contribute to increasing the quality even of rejected manuscripts

Assessing peer review by gauging the fate of rejected manuscripts: the case of the Journal of Artificial Societies and Social Simulation

This paper investigates the fate of manuscripts that were rejected from JASSS-The Journal of Artificial Societies and Social Simulation, the flagship journal of social simulation. We tracked 456 manuscripts that were rejected from 1997 to 2011 and traced their subsequent publication as journal articles, conference papers or working papers. We compared the impact factor of the publishing journal and the citations of those manuscripts that were eventually published against the yearly impact factor of JASSS and the number of citations achieved by the JASSS mean and top cited articles. Only 10% of the rejected manuscripts were eventually published in a journal that was indexed in the Web of Science, although most of the rejected manuscripts were published elsewhere. Being exposed to more than one round of reviews before rejection, having received a more detailed reviewer report and being subjected to higher inter-reviewer disagreement were all associated with the number of citations received when the manuscript was eventually published. This indicates that peer review could contribute to increasing the quality even of rejected manuscript

Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers

Originality is an essential element of academic research and the peer review system plays a key gatekeeping role in its acceptance. However, there is no consensus as to the precise definition of the concept, its measurement nor the importance attached to it. Primary data from 26 interviews with editors or editorial board members of top ranking tourism journals inform a discussion of the nuanced understanding of the concept and of how different levels of originality (radical vs. incremental), among other peer review assessment criteria, influence tourism publication. Finally, the main challenges relating to recognising originality in the peer review process are identified leading to recommendations for improvements to how originality is assessed

Fontes dos problemas na revisão por pares que levam à retratação de artigos divulgados no Retraction Watch

Peer review is plagued by several problems that might affect its integrity, eventually leading to retractions. Understanding those problems can help to improve it; therefore, our research question is: Who is the originator of those problems leading to retractions, and what other liable parties are there? The objective of this research was to survey and categorize information on those problem sources. We analyzed a sample of 24 blog posts mentioning "peer review" among 440 published by Retraction Watch between January 1st and June 30th, 2016. From each post, we extracted the originator and other liable agents, referring to agent categories from a peer review system model. The author was the originator in 16 cases and the referee, in 3. The primary cause of peer review problems is mostly the author, who is usually the only one punished for misconduct according to the literature. However, it is naïve to think that we can solve peer review's misfortunes without taking into account the context of academic productivism and without advancing legislation on editor's and referee's responsibilities, including punishing measures

Fontes dos problemas na revisão por pares que levam à retratação de artigos divulgados no Retraction Watch

Peer review is plagued by several problems that might affect its integrity, eventually leading to retractions. Understanding those problems can help to improve it; therefore, our research question is: Who is the originator of those problems leading to retractions, and what other liable parties are there? The objective of this research was to survey and categorize information on those problem sources. We analyzed a sample of 24 blog posts mentioning "peer review" among 440 published by Retraction Watch between January 1st and June 30th, 2016. From each post, we extracted the originator and other liable agents, referring to agent categories from a peer review system model. The author was the originator in 16 cases and the referee, in 3. The primary cause of peer review problems is mostly the author, who is usually the only one punished for misconduct according to the literature. However, it is naïve to think that we can solve peer review's misfortunes without taking into account the context of academic productivism and without advancing legislation on editor's and referee's responsibilities, including punishing measures

Mining Scholarly Publications for Research Evaluation

Scientific research can lead to breakthroughs that revolutionise society by solving long-standing problems. However, investment of public funds into research requires the ability to clearly demonstrate beneficial returns, accountability, and good management. At the same time, with the amount of scholarly literature rapidly expanding, recognising key research that presents the most important contributions to science is becoming increasingly difficult and time-consuming. This creates a need for effective and appropriate research evaluation methods. However, the question of how to evaluate the quality of research outcomes is very difficult to answer and despite decades of research, there is still no standard solution to this problem. Given this growing need for research evaluation, it is increasingly important to understand how research should be evaluated, and whether the existing methods meet this need. However, the current solutions, which are predominantly based on counting the number of interactions in the scholarly communication network, are insufficient for a number of reasons. In particular, they struggle in capturing many aspects of the academic culture and often significantly lag behind current developments. This work focuses on the evaluation of research publications and aims at creating new methods which utilise publication content. It studies the concept of research publication quality, methods assessing the performance of new research publication evaluation methods, analyses and extends the existing methods, and, most importantly, presents a new class of metrics which are based on publication manuscripts. By bridging the fields of research evaluation and text- and data-mining, this work provides tools for analysing the outcomes of research, and for relieving information overload in scholarly publishing

Priority criteria in peer review of scientific articles

Number of researchers, journals and articles has significantly increased in the last few years and peer review is still the most reliable instrument to sort out innovative, valuable, scientifically sound information from the pool of submitted results. Editors and publishers join their efforts to improve peer review process and to be able to do so properly, they need "field information" from contributors. Editorial board of the Journal of the Serbian Chemical Society performed its own survey to find out what reviewers recognize as priority criteria in manuscript evaluation and whether the professional background (title, number of years in research or experience in reviewing) influences these criteria. Most reviewers declared that they consider peer review as an essential component of the scientific professionalism. Scientific contribution and originality were the most important criteria in the evaluation of papers. Most reviewers preferred to see conclusions completely supported by experimental data, without additional speculations. Although there were no large differences between early stage and experienced researchers, early stage researchers and less experienced reviewers used grade 5 (indicating the highest priority) much more often in their evaluation of priority criteria than experienced researchers and/or reviewers, suggesting possible evolution of tolerance with experience

Priority criteria in peer review of scientific articles

Number of researchers, journals and articles has significantly increased in the last few years and peer review is still the most reliable instrument to sort out innovative, valuable, scientifically sound information from the pool of submitted results. Editors and publishers join their efforts to improve peer review process and to be able to do so properly, they need "field information" from contributors. Editorial board of the Journal of the Serbian Chemical Society performed its own survey to find out what reviewers recognize as priority criteria in manuscript evaluation and whether the professional background (title, number of years in research or experience in reviewing) influences these criteria. Most reviewers declared that they consider peer review as an essential component of the scientific professionalism. Scientific contribution and originality were the most important criteria in the evaluation of papers. Most reviewers preferred to see conclusions completely supported by experimental data, without additional speculations. Although there were no large differences between early stage and experienced researchers, early stage researchers and less experienced reviewers used grade 5 (indicating the highest priority) much more often in their evaluation of priority criteria than experienced researchers and/or reviewers, suggesting possible evolution of tolerance with experience