Hanna, Donncha

O'Hare, Liam

This presentation makes the argument for using Path Analysis in the analysis of educational Randomised Controlled Trial evaluation data. Specifically, it suggests Path Analysis is useful for testing educational programme theory or finding out why a programme works (or doesn't work). Sometimes referred to as looking inside the 'black box' of educational program logic. The paper also provides an example of how the authors have done this with an after-school programme that showed negative effects on pupil behaviour outcomes

Queen's University Belfast Research Portal

Using Path Analysis with Educational Program RCT dataO'Hare, L., & Hanna, D. (2018). Using Path Analysis with Educational Program RCT data. Paper presented atGlobal Evidence and Implementation Summit 2018, Melbourne, Australia.Document Version:Publisher's PDF, also known as Version of recordQueen's University Belfast - Research Portal:Link to publication record in Queen's University Belfast Research PortalPublisher rightsCopyright 2018 The Authors.General rightsCopyright for the publications made accessible via the Queen's University Belfast Research Portal is retained by the author(s) and / or othercopyright owners and it is a condition of accessing these publications that users recognise and abide by the legal requirements associatedwith these rights.Take down policyThe Research Portal is Queen's institutional repository that provides access to Queen's research output. Every effort has been made toensure that content in the Research Portal does not infringe any person's rights, or applicable UK laws. If you discover content in theResearch Portal that you believe breaches copyright or violates any law, please contact openaccess@qub.ac.uk.Download date:10. Nov. 2018Dr. Liam O’Hare Dr. Donncha Hanna Centre for Evidence and Social Innovation Using Path Analysis with Educational Program RCT data RCTs	can	play	a	role	in	building	scientific	knowledge	and	useful	predictions	but	they	can	only	do	so	as	part	of	a	cumulative	program,	combining	with	other	methods,	including	conceptual	and	theoretical	development,	to	discover	not	“what	works,”	but	“why	things	work”.		Understanding	and	misunderstanding	randomized	controlled	trials	Angus	Deaton	and	Nancy	Cartwright	(2016)	Aim	of	this	paper		•  To	change	the	perception	that	RCT	data	can	only	be	used	to	find	IF	a	program	works.	•  That	RCT	data	can	be	analysed	in	different	ways	to	also	explore	WHY	programs	work	(or	do	not	work).	Perceptions of RCT Data Analysis 		CAUSE 	 	 	 	 	 	 	 	EFFECT			 	Really	good	at	finding	out	IF	an	educational	program	works…				Perceptions of RCT Data Analysis 		CAUSE 	 	 	 	 	 	 	 	EFFECT								 	…but	considered	not	good	at	finding	out	WHY	the	program	works.				Accepted ways to look inside the black box? 	•  Logic	model	(theoretical	picture	of	the	black	box	but	untested)	•  Process	evaluation	(focuses	on	black	box	but	generally	qualitative	and	subjective)	•  Implementation	analysis	(explores	some	elements	of	the	black	box	but	generally	focused	on	activities	rather	than	outcomes).	•  …but	in	an	RCT	we	usually	have	a	range	of	data	on	outcomes	and	implementation.	This	provides	quantitative,	context	specific	data.	•  Can	we	analyse	this	data	in	a	different	ways	to	investigate	the	black	box?	-	One	potential	method	is	Path	Analysis	-	Example	analysis	–	Mate-tricks	program	RCT	The Mate-Tricks Programme	•  A one year pro-social behaviour after-school programme •  Area of significant disadvantage in Dublin, Ireland  •  59 child only, 6 parent only (13%) and 3 family sessions (4%) •  RCT of 592 children (traditional analysis) - Outcome evaluation - Process Evaluation (Qualitative) - Implementation Evaluation (Quantitative) - All done separately 	Measures •  Range	of	Outcome	measures	(including):	-	Pro-social	behaviour	-	Anti-social	behaviour	-	Liberal	parenting	-	Supportive	parenting	-	Authoritarian	parenting	•  Implementation	Measures	-	Parental	Attendance	(Dosage)	-	Child	Attendance	(Dosage)	-	Child	satisfaction	with	the	program	-	Child	relationship	with	the	facilitator		Original RCT Effects Four	significant	adverse	effects	(from	child	self-report)		•  Anti-social	behavior	(PSB	Questionnaire)	ES	=	.18	•  Anti-social	behavior	(CBCL	Questionnaire)	ES	=	.20	•  Authoritarian	Parenting	(APQ)	ES	=	.20	•  Liberal	parenting	(APQ)	ES	=	.16	•  No	significant	positive	effects	Process evaluation findings •  The	views	and	observations	of	the	Mate-Tricks	(MT)	program	were	very	positive.	•  This	is	in	contrast	with	main	programme	effects.	Why?	•  Engaged	parents	and	their	children	took	part	in	the	process	evaluation	therefore	they	were	more	favourable	towards	the	programme	•  These	findings	might	suggest	that	Mate-Tricks	worked	successfully	for	engaged	families	•  A	program	for	these	families	alone	would	not	meet	the	original	aims	based	on	whole	community	disadvantage.	Original Implementation Findings •  MT	outcomes	were	influenced	by	a	range	of	factors		-	Parent	attendance,	Children	attendance,	Child	satisfaction	-	Overall	low	parent	attendance	was	the	most	significant	predictor	of	negative	outcomes	•  Program	satisfaction	also	influenced	outcomes	-	Satisfaction	with	programme	predicted	pro-social	behaviour	-	Relationship	with	facilitator	predicted	anti-social	behaviour	Original theories on what went wrong? •  Program	had	theory	of	change	issues	(no	logic	model)	-	Large	imbalance	of	child	and	parent	sessions	-	Difficulties	in	engaging	parents	-	Pattern	of	effects	were	parents	‘Giving	up’	(liberal)	or	‘Cracking	down’	(authoritarian)	and	children	‘Acting	out’	(anti-social)		-	Program	potentially	set	up	competitive	goal	structures	competitive	emotional	learning	rather	than	cooperative	emotional	learning	-	However,	these	are	untested	theories;	so	we	decided	to	explore	it	in	more	depth	with	Path	Analysis	Path Analysis	•  Subset	of	Structural	Equation	Modelling	•  Path	analysis	only	deals	with	observed	variables	•  Allows	the	specification	of	complex	a	prori	models	(e.g.,	direct,	indirect,	mediation,	moderation,	etc.)	reporting	magnitude	&	significance	of	paths	between	variables	•  Compares	the	relationships	in	the	model	with	the	relationships	that	exist	in	the	data;	reports	these	as	Goodness-of-Fit	Indices	•  Simple	example	from	MT		 	T1	Antisocial		Behaviour		Intervention	or		control	T2	Antisocial		Behaviour		χ2	(1,	n=592)	=	1.82,	p=.177;	RMSEA	=	.04;	CFI	=	.98;	TLI	=	.90	β=	.37;	p<.001	β=	.11;	p=.026		Replication of RCT results Liberal	parenting	gains	Authoritarian		Parenting	gains	Antisocial		Behaviour	gains	χ2	(1,	n=304)	=	14.40,	p<.001;	RMSEA	=	.21;	CFI	=	.69;	TLI	=	.83	β=	.22;	p=.004	β=	.40;	p<.001		Model 1 (Outcomes only) Parenting and behaviour Liberal	parenting	gains	Authoritarian		Parenting	gains	Antisocial		Behaviour	gains	χ2	(1,	n=304)	=	.69,	p=.406;	RMSEA	≈	.00;	CFI	≈	.99;	TLI	=	1.06	β=	.22;	p=.005	Parental		Dosage	β=	.38;	p<.001	β=	.11;	p=.115	β=	-.14;	p=.108	Model 2 (Outcomes & Implementation) Parenting, Dosage & behaviour How did we look inside the black box? 	•  Traditional	process	evaluation	-	Missed	lost	voices	on	negative	effects	•  Implementation	analysis	-	Showed	parental	engagement	was	a	problem	•  Path	Analysis	-	Drew	together	outcome	data	as	well	as	implementation	data	-	Tested	a	theory	on	WHY	the	programme	did	not	work	(looked	inside	the	black	box).	Parental	dis-engagement	led	to	competitive	emotional	learning	-	There	are	more	holistic	ways	to	analyse	the	data	provided	through	RCTs	to	allow	researchers	to	answer	IF	and	WHY	questions	about	educational	programs.	More Information  O’Hare,	L.,	Biggart,	A.,	Kerr,	K.,	&	Connolly,	P.	(2015).	A	randomized	controlled	trial	evaluation	of	an	after-school	prosocial	behavior	program	in	an	area	of	socioeconomic	disadvantage.	The	Elementary	School	Journal,	116(1),	1-29.		O’Hare,	L.,	Kerr,	K.,	Biggart,	A.,	&	Connolly,	P.	(2012).	Evaluation	of	the	effectiveness	of	the	Childhood	Development	Initiative’s'	Mate-Tricks'	pro-social	behaviour	after-school	programme.		O’Hare,	L.	(2014).	Did	Children's	Perceptions	of	an	after-school	Social	Learning	Program	Predict	Change	in	their	Behavior?.	Procedia-Social	and	Behavioral	Sciences,	116,	3786-3792.		Contact:	Dr	Liam	O’Hare	L.ohare@qub.ac.uk		Dr	Donncha	Hanna	donncha.hanna@qub.ac.uk		

Using Path Analysis with Educational Program RCT data

https://pure.qub.ac.uk/ws/files/159948562/GES_2018_LOH_DH_MT_.pdf

Using Path Analysis with Educational Program RCT data

Abstract

Similar works

Full text

Available Versions

Queen's University Belfast Research Portal