Peer reviewing plays an important role in the academic publishing process that scrutinizes and provides feedback on the scientific work prior to publication. Peer-reviewers put their efforts in reviewing others research work voluntarily, without any expectations of incentives or rewards. The peer-review process has been criticized for its defects like slowness, bias and abuse of the process. In this paper, we present a model to address these issues by using the approach of recording peer-review data on the blockchain. By using the semantic web and linked data technologies, this system would be able to expose its data and interact with other systems. This system will be used to quantify, recognize and incentivize the peer-reviewing efforts by researchers

Jan, Zeeshan

English

Open Research Online

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsRecognition and reward system for peer-reviewersConference or Workshop ItemHow to cite:Jan, Zeeshan (2018). Recognition and reward system for peer-reviewers. In: CEUR Workshop Proceedings(Kirrane, Sabrina and Kagal, Lalana eds.), CEUR-WS.org, 2181 pp. 46–54.For guidance on citations see FAQs.c© 2018 The AuthorVersion: Version of RecordLink(s) to article on publisher’s website:http://ceur-ws.org/Vol-2181/paper-06.pdfCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukRecognition and reward system forpeer-reviewersZeeshan JanKMI - Knowledge Media Institute,The Open University, UK.zeeshan.jan@open.ac.ukAbstract. Peer reviewing plays an important role in academic publish-ing process that scrutinizes and provides feedback of the scientific workprior to publication. Peer-reviewers are responsible for filtering out in-correct and fraudulent work and suggest improvements with clarity inthe presented work. Peer-reviewers put their efforts in reviewing oth-ers research work voluntarily, without any expectations of incentives orrewards. The peer-review process is considered as a make or break mech-anism for researchers careers, scientific discoveries and funding grants.The peer-review process has been criticized for its defects like slowness,bias and abuse of the process. However, these defects of the peer-reviewprocess need to be addressed and fixed, along with peer-reviewing effortsput by researchers need to be recognized, quantified and incentivized bythe academia. In this paper, we present a model that introduces peer-review data-recording on the blockchain, as well as recognizes, quantifiesand incentivizes peer-reviewers.Keywords: Peer-review · Incentive · Linked-data · Blockchain.1 Problem StatementPeer-reviewing is the activity when researchers assess the scientific work of otherresearchers before publishing. The process of peer-reviewing is questioned andcriticized for the identified defects in it [12]. These defects include slowness inthe process, abuse, bias and inconsistency raise questions about validity and’working’ of the peer-review process.Researchers spend their valuable time in peer-reviewing that somehow, is lessrecognized and quantified by the community. Such efforts of researchers shouldbe recognized, quantified and incentivized at some public platform.There is a need of defining the actual and potential requirements of the peer-review process, along with a model to incentivize peer-reviewers.2 RelevancyThe first description of the peer-reviewing process is likely to be found in abook Ethics of the Physician by Ishap bin Ali Al Rahwi (CE 854-931) of AlRaha, Syria [13]. This history indicates that duplicate notes of condition of thepatient were taken by the visiting physician on their each visit and then sentto the local council after the patient had been cured or died, to decide if thephysician followed the standard operating procedures or not. The modern formof peer-review process used by journals, conferences and publishers is influencedby the review process adopted by the Royal Society of Edinburgh in 1731, whenpublications were inspected by a selected group of knowledgeable members [13].Peer-review is at the heart of the processes of science [12]. A process thatdecides if an article is relevant to the journal and should be published or not.A peer-review can be defined informally as a fellow researcher expert in thesame area of research assesses the scientific work presented by an individual ora group of researchers. The process of peer-reviewing is an important part ofscholarly communication process and it has been discussed and criticized for itsmethods, approaches and defects but yet stands as the backbone of the scientificpublication process. The mechanism of assessing the scientific work done by otherresearchers needs dedicated time and efforts by peers. This is how scientific workbecomes contribution to the science when peers asses the quality, significance,originality and validity prior to publication. Published research work is piled upwhen new researchers develop their understanding by reading previous studies,identifying research questions and gaps, discovering new research areas of studyand contributing their research work [14].By bringing improvements in the mechanism of peer-reviewing and making itattractive for researchers will have a constructive impact on research and science.3 Related WorkIn this section peer-review patterns, their problems and incentive models forpeer-reviewers are discussed.3.1 Peer Review PatternsPeer reviewing practice follows a number of patterns, for instance, in [8], casestudies of six peer-review patterns were presented. However, we focus on OpenPeer review process for the pilot study.Closed Review: This process is categorized in two ways, single blind and doubleblind. Single blind process ensures that identities of reviewers remain unknownto the authors, while in Double blind process, in addition to the withholding ofpeer-reviewers names from authors, the names of the authors are not revealedto the reviewers. However, peer-reviewers can identify the articles after they arepublished.Open Review: Open peer-review process consists of publishing the manuscriptsalong with the reviews they get, to ensure the transparency in peer-reviewing.In open peer-review, names of the authors and reviewers are mutually revealed,to avoid the potential bias.23.2 Defects of Peer-reviewSmith [12] mentions about a systematic review of all the available evidence onpeer-review, that concluded ’the practice of peer review is based on faith in itseffects, rather than on facts’ [6]. Stephen Lock [7] as an editor of the BMJ, con-ducted a study in which only he decided about the publication of a consecutiveseries of papers submitted to the journal, he would publish. Lock, then sent thepapers for usual process of peer-review and there was a little difference betweenthe papers he selected for publication and those selected after peer-review pro-cess [7]. Smith [12] enlists a number of defects in peer-review process includingbeing poor at detecting gross defects, almost useful for detecting fraud, slow, ex-pensive, profligate of academic time, highly subjective, a kind of lottery, proneto bias and easily abused [12].Slowness in the process: There are many journals take even more than a yearto review, accept and then publish a paper [12]. That makes peer-review processas the frustrating phase of publishing scientific work.Prone to bias: Traditionally, peer-review scores are not made public [15] there-fore, there are high chances of occurrence of bias. Wenneras finds strong evidenceof gender bias against women in the process of peer-review while awarding fi-nancial grants [15]. According to the study, peer-reviewers deemed women to bedeficient in scientific competence [15].Abuse of the system: Peer-review process can be abused at many differentlevels. From stealing research ideas to producing harsh review to block the re-search progress of competitor, many researchers have been victim of the abuse[12]. Another possible (potential) abuse of the system can be the misuse of theaccess over peer-review database, as such systems are centralized. Someone caneasily alter the data and change the decision if they want to abuse the accessover system.Fake peer-reviews: Haug, explains about the 64 retracted articles from 10 dif-ferent journals by the publisher Springer. Haug further mentions about a SouthKorean researcher Hyung-in Moon, who admitted to have email addresses sothat he could provide peer-reviews of his own articles using those email and fakeidentities. Through such editorial checks, more than 250 articles were retractedbecause of the fake-reviews, about 15% of the total number of retractions [3].3.3 IncentiveThere have been several approaches to incentivising peer-reviewing, some ofwhich are listed in table 1 below. To improve the peer-review process, punishingthe peer-reviewers is also brought under discussion.Gropp [2] raised a question that if peer-review is properly incentivized, andif peer-reviewers are being asked to evaluate the right things? Gropp proposedan incentive model to filter out the good peer-reviewers who produce thoroughreviews, submit them timely and are responsive to any query sent about thereview. Such peer-reviewers’ name are put into a research funds lottery, to givethem incentive for peer-reviewing.3TheWinnower is a platform that publishes post-publication peer-reviews andis exploring incentivizing peer-reviewers to highlight their work by elevating thereview report to the level of an original research publication [10].ReviewerCredits1 came up with an idea to give credit to peer-reviewers. Peer-reviewers would have to contact ReviewerCredits which would contact the journalconcerned for verification. After receiving verification, a peer-reviewer’s profileis credited with ReviewerCredits.Punishing the peer-reviewer submitting after deadline, is suggested by Hauser[4]. Hauser opines, if habitual late-reviewers stick to their habit of being latein submitting the peer-review, another week delay (for their own publication)should be added as a top-up.Table 1. Comparison of incentive modelsApproach Punishment Model Incentive ModelHauser [4] Reviewer’s article in edito-rial limbo for a certain pe-riod. (even if peer-reviewersrefuse to review)Reviewers’ articles in priority queue for publication, if accepted.Gropp [2] No Filter good peer-reviewers and put them in to a lottery fundReviewerCredit2 No Credit awards for peer-reviewers’ profiles as reputation indicators.ScienceMiles No Digital Currency, to be spent on other platforms, as well as measuredas reputation.4 Research QuestionsGiven the problems discussed in section 1, we intend to answer these researchquestions:Q1 : What are the actual and potential requirements of peer-review pro-cesses? (e.g., open, closed, blind, double-blind, conferences, journals etc.)Q2 : How are currently implemented processes addressing and failing to ad-dress these requirements?Q3 : How does blockchain-based (recording data and incentivizing) processaddress and fail to address these requirements?Q4 : How can currently implemented processes be compared with the blockchain-based processes in terms of addressing the requirements answered by Q1?Q5 : By knowing that the peer-review data would be on the blockchain, doesit affect the researchers’ behavior in terms of:i Papers they acceptii Papers they rejectiii Papers they rate5 HypothesesBy having above research questions, we derive hypotheses as:H1 : Blockchain-based data-recording model for peer-reviewing process keepsthe data tamper-free, traceable and immutable.1 https://www.reviewercredits.com/4H2 : Blockchain-based incentivizing model for peer-reviewing process ensuresfair and transparent distribution of rewards.H3 : Blockchain offers the features of immutability and transparency to en-sure tamper-free and traceable data. Therefore, researchers would be more care-ful in terms of accepting, rejecting and rating papers.6 Preliminary resultsWe are in a process to annotate the open-review data from ESWC. We have alsodeployed Ethereum3 based blockchain in our private network of the institute.Once we have annotated data, we would proceed with the implementation of theSmartContracts for this data.7 ApproachWe plan to implement Ethereum [1] based blockchain that would record peer-review linked-data.Annotation (Linked-data)The approach would be to annotate the peer-review data by reusing Confer-ence [9] and ScholOnto [11] ontologies. Annotating the data using linked-datatechnologies would ensure the integration of third-party applications with thesystem.BlockchainSmartContract is a piece of code embedded and stored in blockchain network,that is responsible for managing the transactions and is protected from anyamendments, deletion and tampering [5]. We plan to implement SmartCon-tract(s) enabling storage of linked-data on the blockchain.3 https://www.ethereum.org/5Fig. 1. ScienceMiles Model8 Evaluation planBased on the research questions posed in section 4, the evaluation plan for eachquestion is:E1 : Are actual and potential requirements of peer-review process addressedby currently implemented peer-reviewing processes?E2 : Does blockchain-based data-recording model fulfill these requirementsand add the promising features (i.e., transparency, immutability, decentraliza-tion) to the peer-review process?E3 : Does blockchain-based incentive model for peer-review process ensurethe transparent and fair distribution of awards amongst researchers (peer-reviewers)?E4 : Comparison between currently implemented peer-review processes andblockchain-based peer-review process in terms of the number of:i Papers accepted by a journal/conference.ii Papers rejected by a journal/conference.iii Papers rated by peer-reviewers.9 ReflectionsExisting approaches to incentivize peer-reviewers are based on centralized andmanual systems requiring trust in third parties. Apart from the idea of a lotteryfund for researchers, these models offer no real-world value incentives. While6we mentioned about the defects in peer-review process and identified that wecan address these issues by developing a system based on the linked-data andblockchains technologies that guarantee data integration, re-usability of data,security and transparency.Benefits of peer-review process on Blockchain– Transparency: Blockchain offers transparency in the transactions beingrecorded in the network to make them accountable and audit-able. In peer-review system, the impact of transparency would strengthen the importance,reliability and trust in peer-review process. With distributed database tech-nology, each member in the network would be able to lookup, trace andvalidate any record of peer-review submissions and incentives given to peer-reviewers. The novel concept of immutability of data in blockchain wouldensure the sanctity of records and prevent any fabrication of data.– Decentralized data: Blockchain follows peer-to-peer network infrastruc-ture where the network peers(nodes) validate, store and reconcile data. Con-sensus is a concept of reviewing such transactions and taking decisions abouttheir validity. By implementing peer-review system on blockchain, each nodeof the network would have the same copy of data which makes it persistent,incorruptible, secure and immutable, in the presence of Consensus. Thatsafeguards the data from hacking attacks and attempts to manipulate it.– Trust in peer-review process: Features of blockchain guarantee immutabil-ity, persistence, security and publicly availability of data, that would increasethe trust in peer-review process.– Recognition: With the implementation of peer-review process on the blockchain,the contributions by peer-reviewers can be quantified as the recognition.Peer-reviewers would get recognized based on their reviews and rewardsavailable publicly. Journals and conferences can also announce the best re-viewers of the year title awards to the researchers.– Preventing possible abuse: Currently, peer-review systems are imple-mented by some centralized third-party entities that share the access to datawith conference/journal program committee. The centralized-stored data canbe abused potentially. Implementing peer-review system on blockchain canhelp in preventing any possible abuse of authority over a reviews database.– Less bias: Reviews and comments about them are publicly available as wellas immutable, so reviewers would want to come up with a valid and clearstance about the article, hence it would reduce the bias in the reviewingprocess if any.– Improved quality of the scientific output: Encouraging timely and well-thought unbiased reviews will improve the quality of the scientific output.– Rewards: By participating in the public platform, authors, reviewers andcommentators would be entitled to be awarded by ScienceMiles. These Sci-enceMiles in the form of a digital currency, are the acknowledgement by theresearch community, for the research community.7– Large pool of peer-reviewers: This system would gather a large pool ofwell-reputed (verified) peer-reviewers, that would be helpful for journals tofind the suitable peer-reviewers for their publications.AcknowledgementsI would like to thank Prof. John Domingue and Dr. Allan Third for supervisingand guiding me through my PhD research.References1. V. Buterin et al. Ethereum white paper. GitHub repository, 2013.2. R. E. Gropp, S. Glisson, S. Gallo, and L. Thompson. Peer review: A system understress. BioScience, 67(5):407–410, 2017.3. C. J. Haug. Peer-review fraudhacking the scientific publication process. NewEngland Journal of Medicine, 373(25):2393–2395, 2015.4. M. Hauser and E. Fehr. An incentive solution to the peer review problem. PLoSBiology, 5(4):e107, 2007.5. M. Iansiti and K. R. Lakhani. The truth about blockchain. Harvard BusinessReview, 95(1):118–127, 2017.6. T. Jefferson, P. Alderson, E. Wager, and F. Davidoff. Effects of editorial peerreview: a systematic review. Jama, 287(21):2784–2786, 2002.7. S. Lock et al. A difficult balance: editorial peer review in medicinecontinued.Nuffield Provincial Hospitals Trust, 1985.8. D. Millard, P. Sinclair, and D. Newman. Peerpigeon: A web application to supportgeneralised peer review. 2008.9. A. G. Nuzzolese, A. L. Gentile, V. Presutti, and A. Gangemi. Semantic webconference ontology - a refactoring solution. In The Semantic Web: ESWC 2016Satellite Events, volume 9989 of Lecture Notes in Computer Science, pages 84–87.Springer, 2016.10. B. Schmidt, A. Deppe, J. Bordier, and T. Ross-Hellauer. Peer review on the movefrom closed to open. In ELPUB, pages 91–98, 2016.11. S. B. Shum, E. Motta, and J. Domingue. Scholonto: an ontology-based digitallibrary server for research documents and discourse. International Journal onDigital Libraries, 3(3):237–248, 2000.12. R. Smith. Peer review: a flawed process at the heart of science and journals.Journal of the royal society of medicine, 99(4):178–182, 2006.13. R. Spier. The history of the peer-review process. TRENDS in Biotechnology,20(8):357–358, 2002.14. M. Ware. Peer review: An introduction and guide. Bristol: Publishing ResearchConsortium, 2013.15. C. Wenneras and A. Wold. Nepotism and sexism in peer-review. Women, sienceand technology: A reader in feminist science studies, pages 46–52, 2001.8

Recognition and reward system for peer-reviewers

http://oro.open.ac.uk/57436/1/DoctoralConsortiumPaper_v2.1_20180413.pdf

Recognition and reward system for peer-reviewers

Abstract

Similar works

Full text

Available Versions

Open Research Online