Today marks the beginning of Peer Review Week 2017. Here on the Impact Blog, we’ll be featuring posts covering a variety of perspectives on and issues relating to peer review, and which also consider this year’s theme of “Transparency”. To kick things off, Jon Tennant, Daniel Graziotin and Sarah Kearns consider what can be done to address the various shortcomings and problems of the peer review process. While there is obviously substantial scope for improvement, none of the ideas proposed here are beyond our current technical and social means. The key challenge may lie in galvanising our scholarly communities

Graziotin, Daniel

Kearns, Sarah

Tennant, Jon

LSE Research Online

We	have	the	technology	to	save	peer	review	–	now	itis	up	to	our	communities	to	implement	itToday	marks	the	beginning	of	Peer	Review	Week	2017.	Here	on	the	Impact	Blog,	we’ll	befeaturing	posts	covering	a	variety	of	perspectives	on	and	issues	relating	to	peer	review,	andwhich	also	consider	this	year’s	theme	of	“Transparency”.	To	kick	things	off,	Jon	Tennant,	DanielGraziotin	and	Sarah	Kearns	consider	what	can	be	done	to	address	the	various	shortcomingsand	problems	of	the	peer	review	process.	While	there	is	obviously	substantial	scope	forimprovement,	none	of	the	ideas	proposed	here	are	beyond	our	current	technical	and	socialmeans.	The	key	challenge	may	lie	in	galvanising	our	scholarly	communities.Peer	review	of	scientific	research	papers	forms	one	of	the	cornerstones	of	our	knowledge	generation	process.Since	its	origins	in	the	19th	century,	it	has	become	a	diverse	and	complex	process.	The	purpose	of	peer	review	isto	evaluate	and	improve	the	quality,	novelty,	validity,	and	impact	of	submitted	papers	and	the	underlyingresearch.	Despite	being	viewed	by	many	as	the	gold	standard	of	certification	for	research,	there	is	now	increasingevidence	that	the	ideal	of	peer	review	is	not	equally	matched	by	its	process	or	practice.Research	has	shown	that	peer	review	is	prone	to	bias	in	numerous	dimensions,	frequently	unreliable,	and	caneven	fail	to	detect	fraudulent	research.	This	is	a	critical	issue	at	a	time	when	public	education	and	engagementwith	science,	and	trust	in	research,	are	needed	due	to	the	proliferation	of	“alternative	facts”,	where	expertise	isoften	casually	dismissed	in	important	socio-political	domains.	While	we	believe	that	the	ideal	of	peer	review	is	stillneeded,	it	is	its	implementation,	and	the	present	lack	of	any	viable	alternative,	that	must	be	looked	at	forimprovement.In	our	latest	research	paper,	published	at	F1000	Research,	we	brought	together	an	international,	cross-disciplinary	team	of	33	authors	to	look	at	the	history,	present	status,	and	potential	future	of	peer	review.	As	ateam,	we	felt	there	were	some	important	questions	about	peer	review	that	needed	to	be	examined	in	greaterdetail.	For	example,	what	role	does	it	play	in	our	modern	digital	research	and	communications	infrastructure?Does	it	perform	to	the	high	standards	with	which	it	is	generally	regarded?	How	can	the	power	and	practices	of	theweb,	particularly	the	social	aspects	of	Web	2.0,	be	leveraged	to	think	about	innovative	models	for	peer	review?We	showed	that	there	has	been	an	explosion	in	innovation	and	experimentation	in	peer	review	in	the	last	fiveyears.	This	has	been	fuelled	by	the	advent	of	web	technologies,	and	an	increasing	realisation	that	there	issubstantial	scope	to	improve	the	process	of	peer	review.	By	combining	the	knowledge	and	experiences	fromacross	a	diverse	range	of	disciplines,	we	took	an	introspective	look	at	peer	review,	one	we	hope	will	be	useful	forfuture	discussions	on	the	topic.We	believe	that	there	are	three	core	traits	that	underpin	any	viable	peer-review	system:	quality	control	andmoderation,	performance	and	engagement	incentives,	and	certification	and	reputation.	We	also	strongly	believethat	any	new	system	of	peer	review	must	be	able	to	demonstrate	that	it	not	only	outperforms	the	current	models,but	also	that	it	avoids	or	eliminates	as	many	of	the	biases	in	existing	systems	as	possible.Quality	control	and	moderationQuality	control	is	the	core	function	of	peer	review,	and	is	what	distinguishes	the	scholarly	literature	from	almostany	other	type.	Typically	this	has	been	administered	in	a	closed,	venue-coupled	system	with	few	actors;	namelyauthors,	reviewers,	and	editors,	with	the	latter	managing	the	process.	A	strong	coupling	of	peer	review	to	journalsplays	an	important	part	in	this	due	to	the	common,	albeit	deeply	flawed,	association	of	researcher	prestige	withjournal	brand.	The	issue	here	is	that	“quality”	of	peer	review	remains	based	on	trust,	rather	than	anythingsubstantive.	While,	intuitively,	the	quality	of	peer	review	at	more	prestigious	journals	might	be	considered	higherthan	at	smaller	journals,	we	cannot	objectively	state	it	is	as	there	is	simply	not	enough	evidence	due	to	theopacity	of	the	process.Impact of Social Sciences Blog: We have the technology to save peer review – now it is up to our communities to implement it Page 1 of 4		Date originally posted: 2017-09-11Permalink: http://blogs.lse.ac.uk/impactofsocialsciences/2017/09/11/we-have-the-technology-to-save-peer-review-now-it-is-up-to-our-communities-to-implement-it/Blog homepage: http://blogs.lse.ac.uk/impactofsocialsciences/Other	social	knowledge-exchange	platforms	such	as	Wikipedia,	Stack	Exchange,	and	Reddit	have	self-organisedcommunities	and	governance	structures	(e.g.	moderators)	that	represent	possible	alternative	models.	Here,moderators	have	the	same	operational	functionality	as	journal	editors	in	terms	of	gate-keeping	and	facilitating	theprocess	of	engagement.	Individual	research	communities	could	transparently	elect	groups	of	moderators	basedon	expertise	(e.g.	automated	through	ORCID,	using	the	number	of	previous	publications	as	a	threshold),	priorengagement	with	peer	review,	and	assessment	of	their	reputation.	Different	communities	could	use	specific	socialnorms	and	procedures	to	govern	content	and	engagement,	and	to	self-organise	into	individual	but	connectedplatforms.In	such	a	system,	published	objects	could	be	preprints,	data,	software,	or	any	other	digital	research	output.Quality	control	would	be	provided	by	having	a	system	of	semi-automated	but	managed	and	open	peer	review,with	public	interaction,	collaboration,	and	transparent	refinement	through	version	control.	Community	moderationand	crowdsourcing	would	play	an	important	role,	preventing	underdeveloped	feedback	that	is	not	constructiveand	could	delay	efficient	research	progress.When	authors	and	moderators	collectively	deem	the	peer-review	process	to	have	been	sufficient	for	an	object	tohave	reached	a	community-decided	level	of	quality	or	acceptance,	the	review	is	complete.	Some	journals,	suchas	the	Journal	of	Open	Source	Software,	already	implement	this	process	successfully.	While	traditional	editorialroles	are	not	foreseen	in	our	vision,	we	recognise	there	are	still	potential	extreme	cases	where	consensus	is	notachieved	and	third-party	involvement	is	required.	This	can	be	achieved,	for	example,	through	impromptu	electionof	a	highly	ranked,	super-moderator	or	arbiter,	or	an	F1000-like	system	of	discontinued	peer	review.	Followingthis	process,	the	objects	can	be	indexed,	and	the	updated	version	can	be	assigned	a	persistent	identifier	such	asa	DOI,	as	well	as	an	appropriate	license	allowing	for	maximum	reuse	(including	then	sending	to	a	traditionaljournal)	and	process	sustainability.The	important	distinction	here	from	the	traditional	model	is	the	active	promotion	of	inclusive	participation	andcommunity	interaction,	with	quality	defined	and	controlled	by	a	process	of	engagement	and	digestion	of	content.	Ifdesired,	these	objects	could	then	form	the	basis	for	manuscript	submissions	to	journals,	perhaps	even	fast-tracking	them	as	the	quality	assessment	would	already	have	been	completed.	The	role	of	peer	review	would	thenbe	coupled	with	the	concept	of	a	“living	published	unit”,	and	with	dissemination	and	validation	of	researchoccurring	independent	of	journals.Image	credit:	Gold	stars	by	Tina	Saey.	This	work	is	licensed	under	a	CC	BY	2.0	license.Performance	and	engagement	incentivesImpact of Social Sciences Blog: We have the technology to save peer review – now it is up to our communities to implement it Page 2 of 4		Date originally posted: 2017-09-11Permalink: http://blogs.lse.ac.uk/impactofsocialsciences/2017/09/11/we-have-the-technology-to-save-peer-review-now-it-is-up-to-our-communities-to-implement-it/Blog homepage: http://blogs.lse.ac.uk/impactofsocialsciences/To	motivate	and	encourage	participation	with	peer	review,	incentives	will	increase	wider	engagement.	Loweringthe	threshold	of	entry	for	different	research	communities	starts	to	make	open	peer	review	more	accessible	andless	burdensome.	One	of	the	most	widely-held	reasons	for	performing	peer	review	is	a	quid	pro	quo	sense	ofacademic	altruism	or	duty	to	the	research	community.	However,	at	present	this	is	imbalanced	and	researchersstill	receive	far	too	little	credit	or	recognition	for	their	efforts.	Directly	tying	to	certification	and	reputation	is	theultimate	goal	of	most	academic	incentive	systems.New	ways	of	encouraging	peer	review	can	be	developed	by	quantifying	engagement	with	the	process	and	linkingthis	to	academic	profiles.	To	some	extent,	this	is	already	performed	at	platforms	like	Publons	or	ScienceOpenwhere	the	records	of	individuals	reviewing	for	a	particular	journal	can	be	integrated	into	ORCID.	Platforms	suchas	Reddit,	Amazon,	and	Stack	Exchange	use	gamification	and	represent	a	model	in	which	participants	receivevirtual	rewards	for	engaging	with	review.	Those	activities	are	further	evaluated	and	ranked	by	the	community	viaupvotes	and	usefulness	scores.	A	hierarchical	reward	system	based	on	badges	could	be	integrated	into	this,including	features	for	content	or	individuals	such	as	“Top	5%	reviewer”,	“Successfully	replicated”,	“500	upvotes”,or	whatever	individual	communities	decide	is	best.The	distinction	from	the	traditional	process	is	that	highly	rated	reviews	gain	more	exposure,	more	scrutiny	andrecognition,	and	ultimately	more	credit.	This	creates	the	incentive	to	engage	with	the	process	in	a	way	that	ismost	beneficial	to	the	community,	which	can	then	be	used	as	a	way	of	establishing	prestige	for	individuals	and	forcontent.Certification	and	reputationThe	current	peer-review	process	is	generally	poorly	rewarded	as	a	scholarly	activity.	Performance	metrics	provideone	way	of	certifying	peer	review,	and	provide	the	basis	for	incentivising	participation.	As	outlined	above,	a	fullytransparent	and	interactive	process,	combined	with	reviewer	identification,	makes	clear	the	level	of	engagementand	added	value	from	each	participant.Certification	can	be	provided	to	contributors	based	on	their	engagement	with	the	process:	community	evaluationof	their	contributions	(e.g.	as	implemented	at	Amazon,	Reddit,	or	Stack	Exchange),	combined	with	theirreputation	as	authors.	Rather	than	having	anonymous	or	pseudonymous	participants,	for	peer	review	to	work	wellin	this	open	system	requires	full	identification,	to	connect	on-platform	reputation	with	authorship	history.	Ratherthan	a	journal-based	form,	certification	is	granted	based	on	continuing	engagement	with	the	research	processand	is	revealed	at	the	object	and	individual	level.The	distinction	from	the	traditional	model	here	is	that	achievement	of	certification	takes	place	via	an	evolving	andcontinuous	process	of	community	engagement	and	can	be	quantified.	Models	like	Stack	Exchange	are	idealcandidates	for	such	a	system,	and	operate	through	a	simple	and	transparent	up-voting	and	down-voting	scheme,combined	with	achievement	badges.	Such	quantifiable	performance	metrics	could	easily	be	tied	to	the	academicreputation	of	individuals.	As	this	is	decoupled	from	journals,	it	alleviates	all	of	the	well-known	issues	with	journal-based	ranking	systems	and	is	fully	transparent.	This	should	be	highly	appealing	not	just	to	researchers,	but	alsoto	those	in	charge	of	hiring,	tenure,	promotion,	grant	funding,	and	research	assessment,	and	therefore	couldbecome	an	important	factor	in	future	policy	development.Challenges	and	future	considerationsNone	of	the	ideas	proposed	above	is	radical,	unachievable,	or	beyond	current	technical	and	social	means.	Thereare	working	models	that	demonstrate	the	potential	feasibility	of	such	a	system,	as	exemplified	by	a	huge	range	ofweb-native	platforms,	many	of	which	scholars	already	engage	with.	Instead,	what	we	suggest	here	is	the	simplerecombination	of	existing	traits	from	successful	social	platforms	into	a	single,	hypothetical	hybrid	platform.Impact of Social Sciences Blog: We have the technology to save peer review – now it is up to our communities to implement it Page 3 of 4		Date originally posted: 2017-09-11Permalink: http://blogs.lse.ac.uk/impactofsocialsciences/2017/09/11/we-have-the-technology-to-save-peer-review-now-it-is-up-to-our-communities-to-implement-it/Blog homepage: http://blogs.lse.ac.uk/impactofsocialsciences/A	key	challenge	our	proposed	hybrid	system	will	have	to	overcome	is	simultaneous	uptake	across	the	wholescholarly	ecosystem.	In	particular,	this	proposal	involves	a	requirement	for	standardised	communication	betweena	range	of	key	participants.	Real	shifts	will	occur	where	elements	of	this	system	can	be	taken	up	by	specificcommunities,	but	remain	interoperable	between	them.	Identifying	sites	where	stepwise	changes	in	practice	aredesirable	to	a	community	is	an	important	next	step.	Increasing	the	–	currently	almost	non-existent	–	role	andrecognition	of	peer	review	in	promotion,	hiring	and	tenure	processes	could	be	a	critical	step	forward	forincentivising	the	changes	we	have	discussed.	As	such,	we	expect	that	research	funders	at	a	range	of	levelsshould	be	interested	in	pooling	knowledge	and	resources	to	build	such	a	platform	as	a	consortium.By	looking	at	the	increasing	adoption	of	social	technologies	by	digital	communities,	we	can	see	that	there	isconsiderable	scope	and	appetite	for	the	significant	development	and	adoption	of	new	peer-review	initiativesproposed	herein.	Such	an	initiative	has	the	potential	to	resolve	many	of	the	technical	and	social	issues	associatedwith	peer	review,	and	also	disrupt	our	entire	system	of	scholarly	communication.	High-quality	implementations	ofthese	ideas	in	systems	that	communities	can	choose	to	adopt	may	act	as	de	facto	standards	that	help	buildtowards	consistent	practice	and	adoption.	We	look	forward	to	seeing	progressive	developments	in	this	domain	inthe	near	future.Note:	This	article	gives	the	views	of	the	authors,	and	not	the	position	of	the	LSE	Impact	Blog,	nor	of	the	LondonSchool	of	Economics.	Please	review	our	comments	policy	if	you	have	any	concerns	on	posting	a	comment	below.About	the	authorsJon	Tennant	completed	his	PhD	at	Imperial	College	London	and	his	research	looks	at	deep	timeevolutionary	patterns	in	groups	like	dinosaurs	and	crocodiles.	Alongside	this,	he	currently	works	asa	PLOS	Paleo	Community	Editor,	is	Communications	Director	for	ScienceOpen,	a	freelance	sciencewriter,	founder	of	paleorXiv	and	the	Open	Science	MOOC,	and	author	of	the	kids’	dinosaur	bookExcavate	Dinosaurs!	He	can	be	found	on	Twitter	at	@Protohedgehog,	talking	about	open	access.Daniel	Graziotin	received	his	PhD	at	the	Free	University	of	Bozen-Bolzano,	Italy.	Heis	a	postdoctoral	researcher	at	the	Institute	of	Software	Technology,	University	ofStuttgart,	Germany.	His	research	interests	include	behavioral	software	engineering,empirical	software	engineering,	open	science,	and	studies	of	science	in	general.	He	isassociate	editor	at	the	Journal	of	Open	Research	Software	and	academic	editor	at	theResearch	Ideas	and	Outcomes	(RIO)	journal.	Daniel	is	recipient	of	the	Alexander	vonHumboldt	Fellowship	(2017),	the	European	Design	Award	(bronze;	2016),	and	the	Data	Journalism	Award	(2015).He	is	an	open	science	advocate;	lately,	he	is	attempting	to	open	up	software	engineering	academic	conferences.He	can	be	found	on	Twitter	at	@dgraziotin	and	on	his	website	ineed.coffee.Sarah	Kearns	is	currently	in	the	University	of	Michigan’s	Chemical	Biology	PhD	program	where	shestudies	non-conventional	chemical	interactions	to	better	drug	development.	Outside	of	lab,	shewrites	for	Michigan	Science	Writers,	where	she	serves	as	the	communications	director,	and	her	ownblog	Annotated	Science.	She	is	interested	in	improving	scientific	literacy	and	increasingtransdisciplinary	sciences	along	with	promoting	open	access/science	and	advancing	the	currentscholarly	infrastructure.	Sarah	can	be	found	on	Twitter	@annotated_sci	talking	about	science	andopen	access.Impact of Social Sciences Blog: We have the technology to save peer review – now it is up to our communities to implement it Page 4 of 4		Date originally posted: 2017-09-11Permalink: http://blogs.lse.ac.uk/impactofsocialsciences/2017/09/11/we-have-the-technology-to-save-peer-review-now-it-is-up-to-our-communities-to-implement-it/Blog homepage: http://blogs.lse.ac.uk/impactofsocialsciences/

We have the technology to save peer review - now it is up to our communities to implement it

Abstract

Similar works

Full text

Available Versions

LSE Research Online