At a loss for how to demonstrate impact? Laura Meagher and David Edwards outline a dynamic understanding of impact evaluation comprised of ‘building blocks’. These building blocks are five types of impacts; five broad categories of stakeholders; and eight causal factors, along with a set of over-arching reflective questions

Edwards, David

Meagher, Laura

English

LSE Research Online

How	to	tell	an	impact	story?	The	building	blocks	youneedAt	a	loss	for	how	to	demonstrate	impact?	Laura	Meagher	and	David	Edwards	outline	a	dynamic	understanding	ofimpact	evaluation	comprised	of	‘building	blocks’.	These	building	blocks	are	five	types	of	impacts;	five	broadcategories	of	stakeholders;	and	eight	causal	factors,	along	with	a	set	of	over-arching	reflective	questions.	The	increasing	pressure	placed	upon	research	organisations	of	all	kinds	(not	just	universities)	to	demonstrateimpact	is	often	passed	along	to	researchers	and	research	managers	without	the	support	of	an	accompanyingcultural	change	within	the	institution.With	this	in	mind,	Forest	Research,	the	government	research	agency	that	supports	the	UK	forestry	sector,	hascommitted	to	promote	a	culture	of	knowledge	exchange	and	interdisciplinarity	in	its	research.	This	mission	stemmedfrom	a	2012	external	impact	evaluation	which	recommended	directing	attention	toward	helping	researchersgenerate	and	record	impacts.	In	addition	to	programme	activities	such	as	training	workshops	and	small-groupconversations,	the	initiative	identified	an	additional	challenge:	that	of	demystifying	and	operationalising	impactevaluation	in	a	coherent	and	accessible	way	that	nonetheless	allows	for	diversity,	subtlety	and	uncertainty.The	initiative	identified	an	additional	challenge:	that	of	demystifying	and	operationalising	impactevaluation	in	a	coherent	and	accessible	way	that	nonetheless	allows	for	diversity,	subtlety	anduncertainty.A	new	evaluation	framework:	building	blocksTo	undertake	this	almost	contradictory	challenge,	we	developed	a	new	evaluation	framework	that	we	tested	acrossmultiple	projects	in	a	pilot	exercise.		Working	with	the	draft	framework	made	it	possible	to	produce	a	dozen	fullimpact	case	studies	from	initial	observations.	Analysis	of	this	exercise	indicated	that	the	components	of	theframework	were	sufficiently	clear	and	comprehensive	for	researchers	and	stakeholders	to	understand	and	use	inconstructing	a	range	of	very	different	narratives.We	envisioned	the	framework	as	a	set	of	building	blocks	which	could	be	combined	in	multiple	ways	to	tell	almostany	impact	story,	each	shedding	light	on	the	complex	interactions	between	research,	policy	and	practice.	In	fact,when	we	gave	a	demo	at	the	2019	UK	Knowledge	Mobilisation	Forum,	we	represented	each	component	withdifferent	wooden	tiles	(fitting	for	work	with	a	forest	research	agency!)	and	then	used	them	to	tell	two	contrastingimpact	case	study	narratives,	showing	how	we	could	manipulate	the	tiles	into	different	configurations	to	reveal	therelationships	between	them.	The	audience	seemed	to	appreciate	our	main	messages	of	dynamics	–	in	terms	ofmulti-directional	interactions	unfolding	in	varying	sequences	–	and	heterogeneity	–	in	terms	of	impact	types,	causesand	stakeholders			–	as	an	alternative	or	complement	to	conventional	linear	logic	models	that	tend	to	dominate	thefield	of	impact	evaluation.Impact of Social Sciences Blog: How to tell an impact story? The building blocks you need Page 1 of 4		Date originally posted: 2020-09-18Permalink: https://blogs.lse.ac.uk/impactofsocialsciences/2020/09/18/how-to-to-tell-an-impact-story-the-building-blocks-you-need/Blog homepage: https://blogs.lse.ac.uk/impactofsocialsciences/Impact	Evaluation	Framework	‘Building	Blocks’What	are	the	building	blocks?There	are:	five	types	of	impacts;	five	broad	categories	of	stakeholders;	and	eight	causal	factors,	along	with	a	set	ofover-arching	questions.Five	impact	typesFrom	previous	impact	evaluations	and	other	research,	we	had	found	that	providing	a	range	of	impact	types	toconsider	often	makes	people	feel	‘liberated’	and	‘authorised’	to	go	beyond	conventionally-sought	instrumentalchanges	when	searching	for	their	impacts.	The	five	impact	types	are:1)	Instrumental:	changes	to	plans,	decisions,	behaviours,	practices,	actions,	policies2)	Conceptual:	changes	to	knowledge,	awareness,	attitudes,	emotions3)	Capacity-building:	changes	to	skills	and	expertise4)	Enduring	connectivity:	changes	to	the	number	and	quality	of	relationships	andtrust5)	Culture/attitudes	towards	knowledge	exchange,	and	research	impact	itselfThere	is	no	fixed	hierarchy	or	linear	sequence	inherent	in	this	typology,	and	all	have	value	in	their	own	right,although	the	last	two,	process-based	impacts	may	make	other	sorts	of	impacts	more	likely.Impact of Social Sciences Blog: How to tell an impact story? The building blocks you need Page 2 of 4		Date originally posted: 2020-09-18Permalink: https://blogs.lse.ac.uk/impactofsocialsciences/2020/09/18/how-to-to-tell-an-impact-story-the-building-blocks-you-need/Blog homepage: https://blogs.lse.ac.uk/impactofsocialsciences/Asking	‘How	do	we	know?’	leads	towards	gathering	of	evidence,	including	recognition	of	telling	indicators	ofchange.We	envisioned	the	framework	as	a	set	of	building	blocks	which	could	be	combined	in	multiple	ways	totell	almost	any	impact	story,	each	shedding	light	on	the	complex	interactions	between	research,	policyand	practice	Five	categories	of	stakeholdersTo	stimulate	further	definition	(or	mapping)	for	a	particular	situation,	the	framework	offers	five	broad	categories	ofstakeholders,	which	of	course	can	be	subdivided	further,	and	may	influence	as	well	as	be	influenced	by	theresearch	process	and	its	outputs.1)Policy-makers:	government	agencies	and	regulatory	bodies,	local,	national	and	international2)Practitioners:	public,	private,	NGOs3)Communities:	of	places	or	interest,	general	public4)Researchers:	within	and	beyond	the	project	and	the	institution5)Other	Eight	causal	factorsBecause	impact	generation	is	a	dynamic	process,	it	is	important	to	ask:	‘Why/how	did	changes	occur?’.	Theframework	offers	eight	causal	factors:1)	Problem-framing:	Level	of	importance;	tractability	of	the	problem;	active	negotiation	of	research	questions;appropriateness	of	research	design.2)	Research	management:	Research	culture;	integration	between	disciplines	and	teams;	promotion	of	researchservices;	planning;	strategy.3)	Inputs:	Funding;	staff	capacity	and	turnover;	legacy	of	previous	work;	access	to	equipment	and	resources.4)	Outputs:	Quality	and	usefulness	of	content;	appropriate	format.5)	Dissemination:	Targeted	and	efficient	delivery	of	outputs	to	users	and	other	audiences.6)	Engagement:	Level	and	quality	of	interaction	with	users	and	other	stakeholders;	co-production	of	knowledge;collaboration	during	design,	dissemination	and	uptake	of	outputs.7)	Users:	Influence	of	knowledge	intermediaries,	e.g.	‘champions’	and	user	groups;	incentives	and	reinforcement	toencourage	uptake.8)	Context:	Societal,	political,	economic,	biophysical,	climate	and	geographical	factors.	Reflective	questionsThe	questions	underlying	the	framework	are:	‘Who	or	what	changed?	(And	how	do	we	know?);	Why	or	how	didchanges	occur?	and	What	lessons	can	be	learned?’	Impact of Social Sciences Blog: How to tell an impact story? The building blocks you need Page 3 of 4		Date originally posted: 2020-09-18Permalink: https://blogs.lse.ac.uk/impactofsocialsciences/2020/09/18/how-to-to-tell-an-impact-story-the-building-blocks-you-need/Blog homepage: https://blogs.lse.ac.uk/impactofsocialsciences/These	are	centred	around	two	questions	which	encourage	the	framework’s	users	to	step	back	and	think	about	whatlessons	have	been	learned	regarding	impact	identification	and	generation,	to	plan	actions	and	improve	futureefforts:1)	What	worked?	What	could	(or	should)	have	been	done	differently?2)	What	could	(or	should)	be	done	in	the	future?The	framework	can	be	used	to	stimulate	critical	reflection	by	individual	researchers	or	teams,	ideally	involving	end-users	and	other	stakeholders,	throughout	the	project	cycle,	at	the	planning,	mid-course	and	final	stages.We	tried	to	encourage	researchers	to	see	impact	as	complex,	developing	over	often	lengthy	periods	oftime,	in	a	dynamic	context	in	which	different	causal	factors,	stakeholders	and	impacts	themselvesinteract.From	simplicity	to	complexityWhen	assessing	impacts,	there	can	be	a	temptation	to	package	notions	of	impact	into	readily	countable	andcomparable	units	which	come	about	as	a	result	of	a	simple	linear	sequence	of	events.	Instead,	we	tried	toencourage	researchers	to	see	impact	as	complex,	developing	over	often	lengthy	periods	of	time,	in	a	dynamiccontext	in	which	different	causal	factors,	stakeholders	and	impacts	themselves	interact.	Ultimately,	impacts	arehighly	heterogenous	and	travel	different	developmental	pathways.	Their	stories	are	multi-dimensional	and	based	oncomplex	interactions.Forestry	was	an	ideal	field	in	which	to	develop	a	framework	of	this	kind,	as	it	draws	together	a	‘real	world’orientation	with	an	interdisciplinary	approach	spanning	the	natural	and	social	sciences.	However,	we	see	theframework	as	having	a	wider	application	and	being	flexible	enough	to	meet	the	needs	of	other	disciplines	andresearch	organisations.The	‘building	blocks’	(impact	types,	stakeholders	and	causal	factors)	we	identified	can	be	assembled	in	multiplecombinations	to	construct	nuanced	impact	narratives	in	a	way	that	is	helpful	for	accountability	(e.g.	for	funders),communication	(e.g.	for	stakeholders)	and	learning	(e.g.	for	researchers/managers).Whatever	their	organisation,	sector	or	country,	we	encourage	readers	to	try	it	out	and	judge	for	themselves!This	post	draws	on	the	authors’	co-authored	paper:David	M.	Edwards	and	Laura	R.	Meagher.	2019.	Forest	Policy	and	Economics.	A	framework	to	evaluate	theimpacts	of	research	on	policy	and	practice:	A	forestry	pilot	study.	Forest	Policy	and	Economics.https://doi.org/10.1016/j.forpol.2019.101975Note:	This	article	gives	the	views	of	the	author,	and	not	the	position	of	the	LSE	Impact	Blog,	nor	of	the	LondonSchool	of	Economics.	Please	review	our	comments	policy	if	you	have	any	concerns	on	posting	a	comment	below.Image	credit:	Author’s	own	Impact of Social Sciences Blog: How to tell an impact story? The building blocks you need Page 4 of 4		Date originally posted: 2020-09-18Permalink: https://blogs.lse.ac.uk/impactofsocialsciences/2020/09/18/how-to-to-tell-an-impact-story-the-building-blocks-you-need/Blog homepage: https://blogs.lse.ac.uk/impactofsocialsciences/

How to tell an impact story? the building blocks you need

http://eprints.lse.ac.uk/106858/1/impactofsocialsciences_2020_09_18_how_to_to_tell_an_impact_story_the.pdf

How to tell an impact story? the building blocks you need

Abstract

Similar works

Full text

Available Versions

LSE Research Online