Following previous agent-based research on peer review, this paper presents a game theory-inspired model that looks at peer review as a cooperation dilemma. We tested different scientist behaviours and network topologies in order to understand their implications on the quality, efficiency and type of resource distribution in the science system. We tested random, scale-free and small world networks connecting scientists and three types of referee behaviour: self-interested (providing unreliable opinion), normative referees (providing reliable opinion) and conformist reviewers (conforming to other referees' behaviour). Preliminary results indicate that differences in the combination of referee behaviour have significant impact on the quality of the process and that the percentage of conformists is one of the most crucial model parameters

Bravo, Giangiacomo

Cabotà, Juan

Cadavid, Lorena

Grimaldo, Francisco

Squazzoni, Flaminio

Diposit Digital de Documents de la UAB

Abstract—Following previous agent-based research on peerreview, this paper presents a game theory-inspired model thatlooks  at  peer  review  as  a  cooperation  dilemma.  We  testeddifferent scientist behaviours and network topologies in orderto understand their implications on the quality, efficiency andtype of resource distribution in the science system. We testedrandom,  scale-free  and  small  world  networks  connectingscientists and three types of referee behaviour: self-interested(providing  unreliable  opinion),  normative  referees  (providingreliable opinion) and conformist reviewers (conforming to otherreferees’  behaviour).  Preliminary  results  indicate  thatdifferences  in  the  combination  of  referee  behaviour  havesignificant impact on the quality of  the process  and that  thepercentage  of  conformists  is  one  of  the  most  crucial  modelparameters.I. INTRODUCTIONEER  review is  recently under  the  spotlight.  Cases  ofmisconduct [1], [2], proofs of biased referee behaviour[3] and studies about the process quality and scientists’ satis-faction [4] called for reconsideration of the quality and sus-tainability of this important institution, especially in periodsof explosion of online journals and publications.PPrevious examples of applications of agent-based modelshave testified  to  the advantages  of  looking at  the internalmechanisms of peer review, e.g.,  scientist behaviour at themicro level [5], [6].This paper proposes a game theory-inspired approach thatlooks at peer review as a cooperation problem with networkexternalities, with room for strategic and normative behav-iour and reciprocal influence by all scientists involved.This approach brings a theoretical basis to previous workand a more realistic modelling, in which the efforts and theThis  work  was  supported  by  the  European  Network  for  SocialIntelligence  (SINTELNET,  FP7-ICT-2009-C  Project  No.  286370,www.sintelnet.eu).resources are limited, and so the strategic behaviours makemore sense.II. THE PEER REVIEW GAMEWe assumed that peer review was a cooperation dilemmawhere players can take on two different roles, authors andreferees.  Following [5], each player  i   ∈ {1,…,n} had a cer-tain amount of resources Ri that could be seen as productiv-ity, available time, funds, human resources, etc. according toacademic status, position, experience, and scientific achieve-ment.They could choose how much to invest in writing articlesand reviewing. We assumed that this investment had a costc>0 independently of the player’s role, given that both activi-ties (writing and reviewing) have a cost on resources.In  case  of  publication,  authors  received  a  benefit  b>c,since publishing usually leads to achieve resources in a sci-entist’s career (such as getting grants, funding for attendingconferences, developing projects, hiring researchers, etc.).On the other hand, reviewers do not receive any benefit,since we understand that the reviewing activity does not pro-duce any profit in terms of consumable resources.The game assumed continuous investment choices, mim-icking successive rounds of the publication process.We defined esi, eri  [0, 1] as∈  the efforts made for submit-ting  and reviewing respectively. Then, we assumed that thecost of both submitting and reviewing articles was propor-tional to these efforts. In the simplest form, we had  csi=esicand crj=erjc.The author’s submission quality was given by Qsi=esiRi (1)while the quality of the review performed by a reviewer   j∈ {1,…,n} | j ≠ i was:Qrj=erjRj (2)A Few Bad Apples Are Enough. An Agent-Based Peer Review GameJuan Cabotà, Francisco GrimaldoDepartament d'Informàtica, Universitat de València,Avinguda de la Universitat, s/n, 46100 Burjassot -València, SpainEmail: {Juan.Cabota, Francisco.Grimaldo}@uv.esLorena CadavidDepartamento de Ciencias de la Computación y laDecisión, Universidad Nacional de Colombia – sedeMedellín, Carrera 80 No 65-223, ColombiaEmail: dlcadavi@unal.edu.coGiangiacomo BravoDepartment of Social Studies, Linnaeus University,Universitetsplatsen 1, 35252, Växjö, SwedenEmail: giangiacomo.bravo@lnu.seFlaminio SquazzoniGECS-Department of Economics and Management,University of Brescia, Via San Faustino 74/B, 25122Brescia, ItalyEmail: flaminio.squazzoni@unibs.itMiguel, Amblard, Barceló & Madella (eds.) Advances in Computational Social Science and Social SimulationBarcelona: Autònoma University of Barcelona, 2014, DDD repository <http://ddd.uab.cat/record/125597>We  assumed  the  peer  review  game  had  a  certainpublication threshold (T).A  review  was  considered  fair,  i.e.,  reflecting  the  truevalue of the paper, if Qrj ≥ T.If the review was fair, a given submission was accepted ifQsi ≥  T and the published  author received  the publicationbenefit. Otherwise, if the submission was rejected, the authorgot  nothing.  Note  that  the  condition  Qsi ≥  T meant  thatauthors  with  higher  resources  could  meet  the  publicationthreshold more easily (i.e., with lower effort).If  the  review was  unfair,  the  paper  was  accepted  withprobability 0.5, which meant that the expected utility of thesubmission was ½b.In  each period, costs were  subtracted and benefits wereadded to the authors’ and referees’ own resources, which leadto the period payoff table shown in Table 1.The equilibrium of the game was as follows: referees hada dominant strategy of putting a zero effort (erj = 0) in theirreview. Knowing this, the best response for authors was toset  esi = 0 in turn,  leading to a unique equilibrium of thegame where only low quality were produced and publicationwas at random.III. THE AGENT-BASED MODELWe built an agent-based model that implemented the peerreview game by adding behavioural  heterogeneity.  We as-sumed one type of agent, namely scientists.In each time step, agents played twice, once as author andonce  as  referee.  The  role  order  was  random and  coupleschanged with roles, meaning that the same two agents wereunlikely to play together twice in the same time step.A fixed global resource  R was divided among scientistsfollowing an individual resource share that varied at the be-ginning of each time step. Thus, the resource available foreach scientist i was Ri.The benefit received in case of publication was given byb, where b was a parameter of the model.The basic cost  of  both producing a  submission and re-viewing was the parameter c, which was multiplied by the ef-fort.The effort for reviewing articles depended on three typesof  referee  behaviour,  i.e.,  self-interest,  normativism  andconformism:• Self-interested  referees  did  not  contribute  to  thequality of peer review by putting little effort in re-viewing (erj = 0.5), trying to save resources for pub-lishing.• Normative referees were intrinsically motivated byMertonian norms of scientist conduct [7], e.g., theyalways put a great  effort in reviewing in order toprovide  pertinent  judgment  so  intentionally  con-tributing to the quality of peer review (erj = 0.75).• Conformists  were  referees  whose effort  dependedon the behaviour of other scientists which they wereconnected with.  While self-interested and altruistswere not influenced by others’ behaviour, conform-ists were sensitive to social  influence.  They com-miserated their effort by looking at the average ef-fort by their connected scientists.We manipulated the initial combination of self-interested,normative and conformists in order to understand interactioneffects among these types of behaviour.  As it  is known inbehavioural game theory, the combination of heterogeneousbehaviour over time can have dramatic implications for theaggregate level of cooperation [8]. Furthermore, we assumed that scientists were connectedin networks, which defined the neighbourhoods affecting thebehaviour  of  conformists.  We  tested  different  networktopologies,  by initializing the  system with  random, small-world  and  scale-free  networks  [9].  These  topologies  havebeen tested by previous studies to reproduce co-authorshipnetworks of scientists.IV. RESULTSPreliminary results by implementing the proposed modelin NetLogo indicated that different behavioural combinationscould dramatically affect the quality of peer review.While  the  percentage  of  self-interested  referees  in  thepopulation could condition cooperative equilibria, we foundthat also the presence of a significant number of conformistscould  have  negative  effects.  This  is  especially  evident  inconjunction with the presence of a minimal number of self-interested and small world network topologies.V. FUTURE WORKThis model represents a first step in an ongoing researchon peer review processes under the scope of the New Fron-tiers of Peer Review (PEERE) COST action.It is known that reputation is crucial in science dynamics,although it is not implemented in the presented model. Fur-ther steps are the inclusion of reputation mechanisms that re-flect better the real scientific world in the long term, in orderTABLE I.PEER REVIEW GAME FOR AUTHOR I SUBMITTING A PAPER HAVING J AS REFEREERefereeQrj ≥ T Qrj < TAuthorQsi ≥ T b – csi, – crj ½b – csi, – crjQsi < T – csi, – crj ½b – csi, – crjMiguel, Amblard, Barceló & Madella (eds.) Advances in Computational Social Science and Social SimulationBarcelona: Autònoma University of Barcelona, 2014, DDD repository <http://ddd.uab.cat/record/125597>to study the effects on the system, and the validation of themodel against real-world data and other game-theory models[10].REFERENCES[1] Bornmann, L.: Scientific Peer Review. Annual Review of InformationScience and Technology, 45, 199-245 (2011).[2] Crocker, J., Cooper, M. L.: Addressing Scientific Fraud. Science , 334(6060), 1182 (2011).[3] Bailey, C. D., Hermanson, D. R., Louwers, T. J.: An Examination ofthe  Peer  Review  Process  in  Accounting  Journals.  Journal  ofAccounting Education, 26, 55-72 (2008).[4] Mulligan,  A.,  Hall,  L.  and  Raphael,  E.  (2013),  Peer  review  in  achanging  world:  An international  study  measuring  the  attitudes  ofresearchers. J. Am. Soc. Inf. Sci., 64: 132–161 (2013).[5] Squazzoni,  F.  and  Gandelli,  C.:  Opening  the  Black  Box  of  PeerReview. An Agent-Based Model of Scientist Behaviour. JASSS- TheJournal  of  Artificial  Societies  and  Social  Simulation,  16(2)  3:http://jasss.soc.surrey.ac.uk/16/2/3.html (2013).[6] Paolucci,  M.,  Grimaldo,  F.:  Mechanism change  in  a  simulation  ofpeer  review:  from  junk  support  to  elitism.  Scientometrics.  [DOI:10.1007/s11192-014-1239-1] (2014).[7] Merton R. K.: The Sociology of Science. Theoretical and EmpiricalInvestigations. Chicago: University of Chicago Press (1973).[8] Bowles  S.  and  Gintis  H.:  A  Cooperative  Species.  Princeton,  NJ:Princeton University Press (2011).[9] Barabási, A.L., Jeong, H., Néda, Z., Ravasz, E., Schubert, A., Vicsek,T.:  Evolution  of  the  social  network  of  scientific  collaborations.Physica  A: Statistical  Mechanics  and  its  Applications,  311  (3–4),590-614 (2002)[10] Squazzoni,  F.,  Bravo,  G.  and  Takács,  K.  (2013).  Does  incentiveprovision increase the quality of peer review? An experimental study.Research Policy, 42(1):287-294.Miguel, Amblard, Barceló & Madella (eds.) Advances in Computational Social Science and Social SimulationBarcelona: Autònoma University of Barcelona, 2014, DDD repository <http://ddd.uab.cat/record/125597>

A few bad apples are enough: an agent-based peer review game

http://ddd.uab.cat/pub/poncom/2014/128567/ssc14_a2014a97iENG.pdf

A few bad apples are enough: an agent-based peer review game

Abstract

Similar works

Full text

Available Versions

Diposit Digital de Documents de la UAB