This article describes how Extension program impact was documented using a retrospective pretest. The method, employed with 35 economic development professionals involved in a traditional Extension educational program, illustrated change in knowledge, skills, attitudes, and behavior. Characteristics of this type of program evaluation are discussed in relation to its implementation

Davis, Gregory A

English

Clemson University: TigerPrints

The Journal of Extension Volume 41 Number 4 Article 23 8-1-2003 Using a Retrospective Pre-Post Questionnaire to Determine Program Impact Gregory A. Davis Ohio State University Extension, davis.1081@osu.edu This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License. Recommended Citation Davis, G. A. (2003). Using a Retrospective Pre-Post Questionnaire to Determine Program Impact. The Journal of Extension, 41(4), Article 23. https://tigerprints.clemson.edu/joe/vol41/iss4/23 This Tools of the Trade is brought to you for free and open access by the Conferences at TigerPrints. It has been accepted for inclusion in The Journal of Extension by an authorized editor of TigerPrints. For more information, please contact kokeefe@clemson.edu. 	 JOEHOME JOURNAL GUIDELINES ABOUT	JOE CONTACT NATIONAL	JOB	BANKCurrent	Issues Back	IssuesAugust	2003	//	Volume	41	//	Number	4	//	Tools	of	the	Trade	//	4TOT4Using	a	Retrospective	Pre-Post	Questionnaire	to	DetermineProgram	ImpactAbstractThis	article	describes	how	Extension	program	impact	was	documented	using	a	retrospectivepretest.	The	method,	employed	with	35	economic	development	professionals	involved	in	atraditional	Extension	educational	program,	illustrated	change	in	knowledge,	skills,	attitudes,	andbehavior.	Characteristics	of	this	type	of	program	evaluation	are	discussed	in	relation	to	itsimplementation.	IntroductionEvaluating	program	impact	is	important	for	all	Extension	educators	in	today's	political	economy.	Ifpassage	of	the	Government	Performance	and	Results	Act	of	1993	has	not	yet	placed	a	renewedemphasis	upon	Extension's	program	effectiveness	component	(Richardson,	Gamble,	&	Mustian,1998;	O'Neill,	1998),	most	certainly	shrinking	budgets	for	Extension	will.	Diem	(2003)	indicatedthat	documenting	such	impact	is	not	only	a	requirement	of	the	agencies	and	political	bodies	thatprovide	Extension	funding,	it	also	serves	as	a	way	to	build	and	maintain	credibility	as	well	as	justifyuse	of	limited	resources.	For	these	reasons	and	others,	evaluation	and	documentation	of	Extensionprogramming	impact	are	beginning	to	receive	increased	emphasis	in	Extension	work	(Arnold,2002).A	brief	analysis	of	recent	JOE	articles	on	the	topic	revealed	that	a	variety	of	methods	andtechniques	can	be	used	for	program	evaluation,	including:	Logic	Modeling,	children's	drawings,formal	qualitative	and	quantitative	methods,	and	the	retrospective	pretest	program	evaluation.This	article	describes	how	the	retrospective	pretest	methodology	was	used	to	determine	change	inknowledge,	skills,	and	attitudes	toward	organizational	strategic	planning	of	35	economicdevelopment	professionals	involved	in	a	traditional	Extension	educational	program.The	Retrospective	PretestOne	AdministrationDocumenting	changes	in	knowledge	and	behavior	can	be	done	simply	and	efficiently	using	theretrospective	pretest	evaluation	(Rockwell	&	Kohn,	1989;	Stevens	&	Lodl,	1999).	According	toRockwell	and	Kohn,	this	tool	"is	specifically	useful	for	evaluating	the	impact	of	Extension	programsby	asking	participants	to	report	actual	changes	in	behavior"	(in	Stevens	&	Lodl).	The	retrospectivepretest	design,	unlike	the	typical	pretest-posttest,	is	administered	only	once.	Because	of	timelimitations,	this	characteristic	made	using	the	method	more	appealing	to	my	audience	and	to	meas	the	administrator	of	the	instrument.	Only	a	few	minutes	were	required	to	complete	the	13-itemquestionnaire.Improved	AccuracyWith	the	retrospective	pretest,	participants	are	asked	to	share	the	knowledge	or	attitude	they	hadtoward	a	particular	subject	before	some	experience,	program,	or	treatment	and	after.	Whenparticipants	are	asked	to	respond	to	a	question	about	how	much	they	know	about	a	particularGregory	A.	DavisExtension	District	Specialist,	Community	DevelopmentOhio	State	University	ExtensionFindlay,	OhioInternet	Address:	davis.1081@osu.edusubject	after	they	have	some	basic	knowledge	of	the	subject	itself,	they	are	more	able	toaccurately	reflect	on	the	degree	of	change	in	knowledge	or	attitude	(Rockwell	&	Kohn,	1989).Furthermore,	respondents	oftentimes	overestimate	their	level	of	knowledge	on	a	particular	subjectwhen	using	the	traditional	pretest-posttest	(Pratt,	McGuigan,	&	Katzev,	2000).	With	theretrospective	pretest	methodology,	respondents	are	given	an	opportunity	to	learn	how	much	theyknow	about	a	subject	prior	to	responding	to	a	questionnaire.My	audience	indicated	that	they	had	some	experience	with	the	topic	prior	to	the	program.	Enablingthem	to	more	accurately	assess	their	baseline	level	of	understanding	after	the	program	providedthem	an	opportunity	to	better	illustrate	the	degree	of	change	as	a	result	of	the	program	andprovided	me	(and	ultimately	my	stakeholders)	with	more	meaningful	data.Using	the	Retrospective	Pretest	to	Measure	ChangeFor	this	evaluation	effort,	a	one-page	questionnaire	was	used	that	contained	four	backgroundquestions	designed	to	collect	basic	data	such	as:	role	played	in	economic	development;	number	ofyears	of	experience	in	these	roles;	population	of	the	community	on	which	these	efforts	arefocused;	and	frequency	of	formal	organizational	strategic	planning	processes	undertaken.	Thereverse	side	contained	the	retrospective	pretest.The	retrospective	pretest	was	designed	with	instructions	at	the	top,	an	example,	and	ninestatements.	The	statements	were	developed	using	the	learning	objectives	for	the	strategicplanning	workshop.	Participants	were	asked	to	indicate	their	level	of	agreement	with	eachstatement	before	and	after	the	workshop	using	a	six-point,	Likert-type	scale;	(1-strongly	disagreeand	6-strongly	agree).AdministrationParticipants	were	asked	to	complete	the	one-page	questionnaire	at	the	conclusion	of	theworkshop.	A	conscious	attempt	was	made	by	the	instructor	to	downplay	the	instrument,	and	therewas	no	verbal	instruction	provided	for	completing	the	two-part	questionnaire.	Participants	weresimply	asked	to	place	their	completed	questionnaire	on	a	table	at	the	back	of	the	room	as	theyexited.	Of	35	workshop	participants,	32	completed	questionnaires.Data	input/AnalysisQuestionnaire	data	were	analyzed	using	SPSS	10.1	to	determine	if	participation	in	the	workshopaffected	participant	knowledge,	awareness,	confidence,	and	attitude.	While	the	SPSS	software	isquite	capable	of	examining	the	degree	of	change	(among	numerous	other	data	analysisprocedures),	the	degree	of	change	was	not	examined.	Group	means	(before	and	after)	were	alsoexamined.Results	and	DiscussionThe	retrospective	pretest's	nine	workshop	indicators	revealed	that	participants	experienced	apositive	change	in	knowledge,	awareness,	confidence,	and	attitudes.	Eight	of	the	nine	indicatorsregistered	positive	change	for	at	least	one	third	of	the	respondents.	The	overall	mean	for	the	nineitems	increased	from	3.9	(before)	to	4.9	(after)	(Table	1).Table	1.Paired	t-Tests	for	Retrospective	Pretest	(n=31)Variable Mean sd. pI	have	a	basic	awareness	of	the	mechanics	of	strategic	planning.Pre 3.7 1.6 <.05Post 4.7 1.0I	know	what	the	key	components	of	strategic	planning	are.Pre 3.5 1.4 <.05Post 4.7 1.1I	think	I	could	facilitate	a	strategic	planning	process.Pre 3.3 1.5 <.05Post 4.5 1.1I	have	the	skills	necessary	to	facilitate	a	strategic	planning	process.Pre 3.5 1.4 <.05Post 4.4 1.2Strategic	planning	can	provide	direction	to	an	organization's	efforts.Pre 4.4 1.6 <.05Post 5.2 1.1I	would	like	to	try	facilitating	a	strategic	planning	process	at	some	point.Pre 3.6 1.6 <.05Post 4.6 1.3I	will	attempt	some	form	of	strategic	planning	process	in	the	future.Pre 4.0 1.8 <.05Post 5.0 1.1Thinking	strategically	is	a	worthwhile	practice.Pre 4.6 1.6 <.05Post 5.4 1.0Strategic	planning	is	an	ideal	way	to	guide	an	organization's	economic	development	efforts.Pre 4.4 1.7 <.05Post 5.3 1.0Conclusions	and	RecommendationsI	found	this	program	evaluation	tool	to	provide	rich	data	with	a	modest	investment	of	time,	relativeto	more	traditional	pretest-post	test	evaluative	measures.	Program	participants	had	little	difficultyunderstanding	and	completing	the	questionnaire.	Furthermore,	participants	were	able	to	completethe	instrument	in	a	timely	fashion,	yielding	very	useful	data	compared	to	other	evaluation	toolsrequiring	a	similar	investment	of	time.	The	data	gathered	were	relatively	easy	to	analyze	andcommunicate	a	change	in	knowledge,	awareness,	confidence,	and	attitudes	as	ably	as	other	morecomplex	and	involved	evaluative	measures.In	short,	I	found	the	retrospective	pretest	a	useful	tool	for	evaluating	this	traditional	Extensionprogram.	While	my	use	of	the	instrument	focused	primarily	on	immediate	impact,	the	tool	couldalso	be	used	for	demonstrating	intermediate	and	long-term	outcomes	of	Extension	programs.ReferencesArnold,	M.	E.	(2002).	Be	"Logical"	about	program	evaluation:	Begin	with	learning	assessment.Journal	of	Extension	[On-line],	40(3).	Available	at:	http://www.joe.org/joe/2002june/a4.htmlDiem,	K.	G.	(2003).	Program	development	in	a	political	world--It's	all	about	impact!	Journal	ofExtension	[On-line],	41(1).	Available	at:	http://www.joe.org/joe/2003february/a6.shtmlDiem,	K.	G.	(2002).	Using	research	methods	to	evaluate	your	Extension	program.	Journal	ofExtension	[On-line],	41(1).	Available	at:	http://www.joe.org/joe/2002december/a1.shtmlO'Neill,	B.	(1998).	Money	Talks:	Documenting	the	economic	impact	of	Extension	personal	financeprograms.	Journal	of	Extension	[On-line],	36(5).	Available	at:http://www.joe.org/joe/1998october/a2.htmlPratt,	C.	C.,	McGuigan,	W.	M.,	Katzev,	A.	R.	(2000).	Measuring	program	outcomes:	Usingretrospective	pretest	methodology.	American	Journal	of	Evaluation.	21(3).Richardson,	J.	G.,	Gamble,	K.	J.,	Mustian	R.	(1998).	Creation	of	a	web	based	accomplishmentreporting	system.	Journal	of	Extension		[On-line].	36(2).	Available	at:http://www.joe.org/joe/1998april/a1.htmlRockwell,	S.	K.,	&	Kohn,	H.	(1989).	Post-then-pre	evaluation.	Journal	of	Extension	[On-line].	27(2).Available	at:	http://www.joe.org/joe/1989summer/a5.htmlStevens,	G.	L.,	&	Lodl,	K.	A.	(1999).	Community	coalitions:	Identifying	changes	in	coalitionmembers	as	a	result	of	training.	Journal	of	Extension	[On-line].	37(2).	Available	at:http://www.joe.org/joe/1999april/rb2.htmlCopyright	©	by	Extension	Journal,	Inc.	ISSN	1077-5315.	Articles	appearing	in	the	Journal	become	the	property	of	theJournal.	 Single	 copies	 of	 articles	may	 be	 reproduced	 in	 electronic	 or	 print	 form	 for	 use	 in	 educational	 or	 trainingactivities.	Inclusion	of	articles	in	other	publications,	electronic	sources,	or	systematic	large-scale	distribution	may	bedone	only	with	prior	electronic	or	written	permission	of	the	Journal	Editorial	Office,	joe-ed@joe.org.If	you	have	difficulties	viewing	or	printing	this	page,	please	contact	JOE	Technical	Support©	Copyright	by	Extension	Journal,	Inc.	ISSN	1077-5315.	Copyright	Policy

Using a Retrospective Pre-Post Questionnaire to Determine Program Impact

https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=5037&context=joe

Using a Retrospective Pre-Post Questionnaire to Determine Program Impact

Abstract

Similar works

Full text

Available Versions

Clemson University: TigerPrints