Over the past 7 years, the University of Maine Cooperative Extension (UMCE) has conducted the Watershed Stewards Program (WSP), a 20-hour lake education and implementation program. To assess program effectiveness, we studied whether our program significantly improved program participant knowledge level over non-participants through quantitative and qualitative measures. An objective, 15-question test was administered to program participants and other people living in these lake watersheds. Stewards scored significantly (23%) higher on the objective test than those who had not been involved in the program. Program participants qualitatively demonstrated much more involvement with lake governance, implementation efforts, and related activities

Graham, Judith

Jemison, John M, Jr

Wilson, Laura

English

Clemson University: TigerPrints

The Journal of Extension Volume 42 Number 3 Article 15 6-1-2004 Effecting Land-Use Changes Through Education and Implementation: Assessing the Effectiveness of the Watershed Stewards Program John M. Jemison Jr University of Maine Cooperative Extension, jjemison@umext.maine.edu Laura Wilson University of Maine Cooperative Extension, lwilson@umext.maine.edu Judith Graham University of Maine Cooperative Extension, jgraham@umext.maine.edu This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License. Recommended Citation Jemison, J. M., Wilson, L., & Graham, J. (2004). Effecting Land-Use Changes Through Education and Implementation: Assessing the Effectiveness of the Watershed Stewards Program. The Journal of Extension, 42(3), Article 15. https://tigerprints.clemson.edu/joe/vol42/iss3/15 This Research in Brief is brought to you for free and open access by the Conferences at TigerPrints. It has been accepted for inclusion in The Journal of Extension by an authorized editor of TigerPrints. For more information, please contact kokeefe@clemson.edu. 	 JOEHOME JOURNAL GUIDELINES ABOUT	JOE CONTACT NATIONAL	JOB	BANKCurrent	Issues Back	IssuesJune	2004	//	Volume	42	//	Number	3	//	Research	in	Brief	//	3RIB40Effecting	Land-Use	Changes	Through	Education	andImplementation:	Assessing	the	Effectiveness	of	the	WatershedStewards	ProgramAbstractOver	the	past	7	years,	the	University	of	Maine	Cooperative	Extension	(UMCE)	has	conducted	theWatershed	Stewards	Program	(WSP),	a	20-hour	lake	education	and	implementation	program.	Toassess	program	effectiveness,	we	studied	whether	our	program	significantly	improved	programparticipant	knowledge	level	over	non-participants	through	quantitative	and	qualitativemeasures.	An	objective,	15-question	test	was	administered	to	program	participants	and	otherpeople	living	in	these	lake	watersheds.	Stewards	scored	significantly	(23%)	higher	on	theobjective	test	than	those	who	had	not	been	involved	in	the	program.	Program	participantsqualitatively	demonstrated	much	more	involvement	with	lake	governance,	implementationefforts,	and	related	activities.	IntroductionMaine	is	a	water-rich	state.	With	over	5,000	scenic	lakes	attracting	thousands	of	tourists	to	thestate	each	year,	it	is	essential	to	protect	this	resource.	In	1996,	the	University	of	MaineCooperative	Extension	(UMCE)	initiated	the	Watershed	Stewards	Program	to	educate	people	aboutthreats	to	Maine	lakes	and	actions	to	prevent	damage	to	lake	water	quality.	Designed	in	a	mannersimilar	to	the	Master	Gardener	Program	(Simonson	&	Pals,	1990),	participants	attended	20	hours	ofeducation	and	in	return	provided	20	hours	of	service	to	their	lake	watershed.Participants	attended	seven	(once	per	week)	2.5-hour	training	sessions.	The	program	relied	on	theassistance	of	our	state	partners,	including	the	Maine	Department	of	Environmental	Protection,Maine	Department	of	Agriculture,	Food	and	Rural	Resources,	and	Natural	Resources	ConservationService	personnel,	in	delivery	of	these	courses.	Non-point	source	pollution	is	such	a	threat	to	Mainelakes	(Chesters	&	Schierow,	1985),	and	course	topics	included:Non-point	source	pollution;Lake	biology	and	chemistry;Septic	systems	and	their	impacts	on	lake	water	quality;Development	methods	to	reduce	impacts	to	water	quality;Road	maintenance	methods;Water	quality	testing	methods	and	trends	in	lake	water	quality;Regulatory	issues;John	M.	Jemison,	Jr.Extension	Professor,	Water	Quality	and	Soil	Sciencejjemison@umext.maine.eduLaura	WilsonWater	Quality	Professionallwilson@umext.maine.eduJudith	GrahamAssociate	Extension	Professor,	Human	Developmentjgraham@umext.maine.eduUniversity	of	Maine	Cooperative	ExtensionOrono,	MaineLand-use	management:	buffers,	lawn	care,	and	gardening	methods;	andNEMO	(non-point	education	for	municipal	officials).One	additional	hands-on	field	training	was	conducted	to	teach	people	how	to	recognize	sites	thathave	the	potential	to	increase	soil	(and	phosphorus	(P))	loading	to	the	lake.	A	map	showing	thelake	watersheds	where	the	program	has	been	conducted	is	found	in	Figure	1.Figure	1.Watershed	Stewards	Program	Locations,	1997	-	2002Purpose	of	the	StudyHaving	conducted	the	course	for	a	7-year	period,	it	was	appropriate	to	assess	impacts	and	todetermine	areas	to	improve	or	redirect	our	program.	Program	evaluation	is	increasingly	afundamental	part	of	being	an	effective	Extension	educator	(Scarborough,	Killough,	Johnson,	&Farrington,	1997).	Given	tight	budgetary	times,	solid	evidence	of	program	impact	is	alsoincreasingly	important	(Van	den	Ban	&	Hawkins,	1996;	Shepard,	2002).	Documenting	solid	impactsand	program	successes	would	increase	our	opportunities	to	continue	program	funding.	Therefore,the	goals	of	this	project	were	to:1.	 Conduct	a	statistically	valid	sampling	of	participants	to	assess	program	impacts	and2.	 Compare	knowledge	levels,	awareness	of,	and	participation	in	lake	management	activitieswith	people	who	live	in	the	same	watersheds	but	who	have	not	participated	in	our	programs.Specific	Research	Questions1.	 Did	the	UMCE	water	quality	program	significantly	improve	the	knowledge	level	of	programparticipants	over	non-participants?;2.	 Did	participant	knowledge	remain	strong	over	time?;	and3.	 Were	program	participants	more	knowledgeable	and	actively	involved	in	lake	protectionefforts	than	non-program	participants	living	in	those	same	lake	watersheds?Methods:	Quantitative	AssessmentTo	answer	research	question	number	one,	a	short	15-question	test	was	developed.	This	test	wasfirst	piloted	on	a	small	group	of	lakefront	landowners.	The	test	was	sent	to	161	programparticipants	(stewards)	and	to	approximately	twice	that	number	(328)	of	people	on	the	same	lakeswho	did	not	participate	(non-stewards).	Anticipating	a	better	response	from	stewards,	an	attemptwas	made	to	increase	project	participation	from	non-stewards	by	doubling	the	number	of	non-stewards	who	received	the	test	and	by	offering	an	incentive	(framed	wildlife	photo)	forparticipation.	We	obtained	non-stewards	addresses	from	lake	association	mailing	lists.This	test	was	designed	to	test	knowledge	of	general	lake	biology,	land-use	impacts	on	waterquality,	and	similar	issues	that	have	been	covered	both	in	our	programs	and	lake-managementawareness	campaigns	of	our	partner	agencies.	Our	hope	was	to	determine	if	program	participantshad	a	significantly	higher	knowledge	level	than	other	lake	landowners.	The	top	score	possible	was65	points,	with	each	question	worth	five	points.	Most	questions	were	multiple	choice,	but	otherswere	fill	in	the	blank	or	yes/no	in	nature.	Some	questions	had	only	one	correct	response	possible,while	others	had	multiple	answers.	Partial	credit	was	possible.	The	entire	test	can	be	found	atwww.umaine.edu/waterquality.Typical	exam	questions	included	the	following:Typical	sources	of	phosphorus	to	Maine	lakes	include	(circle	all	that	apply):	a)	atmospheric	deposition,	b)	aquatic	plants,	c)	septic	systems,	d)	bottom	sediments,	e)	urbanrunoff,	and	f)	fertilizers.The	tool	used	to	measure	lake	clarity	is	called	a:	a)	piezometer,	b)	secchi	disk,	c)	yard	stick,	d)	GIS	(geographic	information	system)	e)	I	don'tknowIs	nitrogen	a	common	limiting	nutrient	in	Maine's	lakes?	___	yes		___	no	___	don't	know.There	was	22%	participation	(70	out	of	328)	with	non-program	participants	(non-stewards)	and39%	response	rate	(64	out	of	161)	with	program	participants	(stewards).	The	test	was	graded,	andthe	scores	were	subjected	to	a	T-test	analysis	to	assess	whether	the	populations	were	different(SAS,	2001).Qualitative	AssessmentStandard	testing	is	an	effective	means	to	assess	knowledge	differences,	but	it	is	difficult	todetermine	if	one	group	is	playing	a	more	active	role	in	lake	protection	efforts	or	working	moreclosely	with	neighbors	on	lake	issues.	It	can	also	be	argued	that	standard	objective	tests	do	notallow	one	to	further	tease	out	differences	in	knowledge	levels.To	more	completely	assess	group	differences	(stewards	from	non-stewards),	a	standard	scriptedtelephone	survey	was	designed	and	conducted.	This	was	an	eight-question	survey	with	multipleparts.	Fifty	people	from	each	group	were	randomly	identified	as	potential	interviewees.	Phonenumbers	of	non-stewards	were	obtained	from	lake	association	contact	lists.	The	first	25	peoplefrom	each	group	who	agreed	to	participate	were	interviewed.	An	individual	not	involved	in	theeducational	program	effort	conducted	the	surveys.Each	interview	took	approximately	15	minutes,	and	responses	were	entered	into	individualMicrosoft	Word	files.	Because	most	of	the	responses	were	subjective	in	nature,	data	were	analyzedusing	qualitative	data	analysis	software	called	NUDIST.	This	software	allows	one	to	search	throughmultiple	documents	and	identify	trends,	patterns,	and	similarity	of	responses.Typical	interview	questions	included	the	following:Who	do	you	think	is	responsible	for	protecting	your	lake?Have	you	personally	taken	action	to	protect	the	lake?If	so,	what	have	you	doneIf	you	haven't,	do	you	intend	to?How	many	hours	per	year	do	you	spend	on	lake	education/protection	activities?What	is	the	greatest	threat	to	your	lake?Major	Findings--Objective	TestingResults	indicate	that	stewards	had	significantly	higher	understanding	of	lake	biology,	knowledge	ofthreats	to	the	lake,	and	appropriate	steps	to	protect	lake	water	quality.	The	mean	score	forstewards	was	47,	and	for	non-stewards,	33.	Overall,	stewards	scored	23%	higher	on	the	objectivetest	than	those	that	did	not	been	involved	in	the	program.	This	difference	was	statisticallysignificant	(p<0.001)	(Figure	2).Figure	2.Bar	Chart	of	Test	ScoresThe	Watershed	Stewards	Program	is	offered	to	all	individuals,	regardless	of	socioeconomic	anddemographic	factors	and	educational	background.	In	order	to	test	whether	stewards	enter	theprogram	with	a	higher	knowledge	level	than	non-stewards	and	to	ensure	that	apparent	knowledgedifferences	were	gained	in	the	program,	the	test	was	administered	to	the	newest	WatershedStewards	class	(Pemaquid)	during	their	opening	session.	Their	class	average	(sample	size	of	12)was	33,	the	same	average	obtained	by	non-stewards	in	the	mailed	test.	Although	direct	pre-	andpost-test	measurements	were	not	performed,	the	new	group's	pre-test	scores	indicate	that	peopletaking	our	program	are	not	entering	with	a	higher	initial	knowledge	level.	This	indicates	that	thehigher	test	scores	by	stewards	reflected	knowledge	gained	in	the	program.To	determine	if	knowledge	gained	in	our	program	was	stable	over	time,	a	regression	analysis	wasperformed	on	program	data.	The	slope	of	the	regression	line	was	not	significantly	different	fromzero	(pr>(t)	=	0.754),	indicating	no	loss	of	participant	knowledge	over	time.	In	fact,	the	averagescores	from	the	first	program	delivered	in	1996	were	among	the	highest	(see	Figure	3).Figure	3.Average	Test	Scores	Over	TimeProgram	participants	scored	higher	than	non-stewards	did	on	each	question.	Some	interestingdifferences	between	stewards	responses	compared	to	their	counterparts	were	found.	For	instance,84%	of	stewards	recognized	that	additives	do	not	increase	the	life	of	a	septic	system	while	only46%	of	non-stewards	provided	the	correct	response.	Another	interesting	difference	was	that	98%of	stewards	knew	that	camp	roads	were	a	major	source	of	P	to	Maine	lakes,	a	fact	that	only	63%	ofnon-stewards	recognized.	Thus,	the	quantitative	test	was	instructive	in	determining	differencesbetween	trained	and	non-trained	lake	residents.Major	Findings--Phone	InterviewThe	interview	process	provided	detail	on	program	participants'	activities	and	allowed	someredirection	of	programming	efforts,	because	many	non-stewards	were	more	involved	or	moreknowledgeable	about	specific	areas	than	had	been	initially	considered.	Some	of	the	moreinteresting	responses	are	presented	later	in	this	article.Both	groups	were	asked	if	they	knew	about	efforts	made	in	their	lake	to	control	polluted	runoff	inthe	lake.	Sixty-eight	percent	of	non-stewards	were	aware	of	efforts	to	control	polluted	runoff	intheir	lake.	In	contrast,	98%	of	stewards	interviewed	were	aware	of	activities	to	protect	the	lakefrom	polluted	runoff.	The	quality	of	the	responses	that	stewards	gave	was	much	more	complete.	Ofthose	non-stewards	who	gave	detailed	responses,	only	three	actually	talked	about	activities	thatwould	actually	reduce	the	amount	of	polluted	runoff	reaching	their	lake.	Below	a	few	of	the	typesof	responses	stewards	gave	are	compared	to	those	of	non-stewards	(Table	1):Table	1.Group	Responses	to	Efforts	Made	to	Control	Polluted	Runoff	to	LakesNon-Stewards StewardsPlanting	a	barrier	to	prevent	road	runoff Abated	long-standing	road	bridgecombination	runoffWorking	on	erosion	control	committee Implementing	shoreland	buffers	andimproving	roadsContributing	money	to	install	culverts Working	on	an	on-going	319	projectimproving	culverts,	ditching,	and	plantingbuffers	Working	with	construction	companies	whoare	increasing	runoff	from	construction	Trying	to	get	rid	of	runoff	problems	Participated	in	field	studies	to	identifysources	of	polluted	runoff	Yes	--	road	rip-rapping	and	we	planted	newvegetation	Worked	on	demonstrations	for	erosioncontrolIn	another	interesting	question,	both	groups	were	asked	how	many	hours	per	year	they	regularlyspend	on	lake	education	and	protection	activities.	Stewards	are	clearly	more	active	in	the	lake(Table	2).Table	2	provides	the	best	evidence	that	the	program	has	motivated	people	to	take	action	in	theirlake	watershed.	For	the	most	part,	the	stewards	know	what	the	problem	is,	why	it	is	important,	andhow	to	correct	it.	Most	important,	they	are	doing	it.Table	2.Group	Responses	to	Hours	per	Year	Spent	on	Lake	Education	and	ProtectionActivitiesHours#	of	Non-Stewards#	ofStewardsLess	than	1016 310-201> 620-301 230-401 340-50-- 250-100-- 3More	than	100-- 2Other	(too	few,	not	necessary) 2 2When	both	groups	were	asked	what	were	the	greatest	threats	to	their	lake/Maine	lakes,	theresponses	were	reasonably	similar	(Table	3).Table	3.Greatest	Threat	to	Your	Lake/Maine	Lakes	(Number	of	Respondents)Non-Stewards StewardsJet	skis	(2) Invasive	aquatic	plants	(10)Bass Road	runoff	�	camp	roads	(4)Growth,	development,	clear-cutting,lack	of	government	intervention,	andrunoff.Phosphorus	(2)Invasive	plants/milfoil	(7) Nonpoint	source	pollution	(3)Runoff,	fertilizing	lawns	(5) ApathyAcid	rain Development	(3)Chicken	farms	in	area Ignorance	(2)Man Lack	of	clarityFertilizing	lawns Too	many	people	(2)Septic	systems Not	enough	shoreland	zoningDucks	and	flooding	Pollution	Not	sure	Don't	know	Phosphorus	This	strong	degree	of	similarity	is	likely	due	to	the	amount	of	information	about	aquatic	invasiveplants	that	has	been	in	the	Maine	news	and	due	to	the	new	state	legislation	passed	last	yearagainst	bringing	boats	into	the	state	carrying	any	vegetation.	The	major	difference	between	theresponses	of	the	two	groups	appears	to	be	that	the	stewards	viewed	the	whole	issue	ofdevelopment	as	the	major	problem.	Non-stewards	had	many	different	types	of	responses.The	question	that	really	differentiated	the	groups	was,	"Do	you	now,	or	have	you	ever	served	onyour	lake	association	board,	town	environmental	planning	board,	or	other	community	group	thatdeals	with	environmental	issues?"	In	summary,	less	than	25%	of	non-stewards	had	participated,while	92%	of	stewards	responded	favorably.	In	these	cases,	the	predominant	participation	was	inlake	association	roles,	town	selectmen,	and	other	related	positions.The	qualitative	assessment	was	useful	to	better	gain	an	understanding	of	what	average	lakelandowners	know	and	do	about	lake	water	quality.	With	just	a	single	test	approach,	theinvolvement	of	the	stewards	compared	to	other	lake	residents	would	not	have	been	appreciated.Another	discovery	that	would	have	been	missed	is	that	the	other	landowners	appear	to	have	aninterest	in	lake	protection,	even	though	they	don't	have	the	knowledge	and	skills	to	participatefully.ConclusionsThe	Watershed	Stewards	Program	has	been	successful.	By	almost	all	indications,	the	knowledgebase	of	stewards	has	improved,	they	know	which	actions	are	required	to	correct	problem	areas,and	they	are	taking	leadership	roles	in	their	local	lake	associations	and	town	boards	to	moreeffectively	improve	lake	water	quality.	State	partners	(Maine	Department	of	EnvironmentalProtection,	Conservation	District	staff,	and	many	others)	were	essential	to	help	make	this	programsuccessful.	Given	the	apparent	effectiveness	of	this	program,	educational	efforts	in	other	Mainelake	watersheds	will	occur.	Based	on	our	experiences,	this	two-tiered	approach	to	programevaluation	can	help	other	Extension	personnel	assess	their	programmatic	impacts.AcknowledgmentsWe	would	like	to	thank	Hilarie	Wilson,	who	conducted	our	phone	survey,	and	Dr.	Chris	Reberg-Horton,	for	his	statistical	assistance.ReferencesChesters,	G.,	&	Schierow,	L.	(1985).	A	primer	on	non-point	pollution.	Journal	of	Soil	and	WaterConservation	,	40(1),	pp.	9	-	13.QSR	NUD*IST	(1997).	Qualitative	solutions	and	research	PTY	ltd.SAS	Institute	Inc.	(2001).	The	SAS	system	for	windows	(Release	8.0).	Cary,	NC.Scarborough,	V.,	Killough,	S.,	Johnson,	D.,	&	Farrington,	J.	(Eds.).	(1997).	Farmer-led	extension.London:	Intermediate	Technology	Publications,	Ltd.Shepard,	R.	(2002).	Evaluating	Extension-based	water	resource	outreach	programs:	are	wemeeting	the	challenge?	Journal	of	Extension	[On-line]	40(1).	Available	at:http://joe.org/joe/2002february/a3.htmlSimonson,	D.	L.,	&	Pals,	D.	A.	(1990).	Master	Gardeners:	Views	from	the	Cabbage	Patch.	Journal	ofExtension	[On-line]	28(2).	Available	at:	http://www.joe.org/joe/1990summer/rb3.htmlVan	den	Ban,	A.	W.,	&	Hawkins,	H.	S.	(1996).	Agricultural	Extension	(2nd	ed.).	Cambridge,Massachusetts:	Blackwell	Science	Ltd.Copyright	©	by	Extension	Journal,	Inc.	ISSN	1077-5315.	Articles	appearing	in	the	Journal	become	the	property	of	theJournal.	 Single	 copies	 of	 articles	may	 be	 reproduced	 in	 electronic	 or	 print	 form	 for	 use	 in	 educational	 or	 trainingactivities.	Inclusion	of	articles	in	other	publications,	electronic	sources,	or	systematic	large-scale	distribution	may	bedone	only	with	prior	electronic	or	written	permission	of	the	Journal	Editorial	Office,	joe-ed@joe.org.If	you	have	difficulties	viewing	or	printing	this	page,	please	contact	JOE	Technical	Support©	Copyright	by	Extension	Journal,	Inc.	ISSN	1077-5315.	Copyright	Policy

Effecting Land-Use Changes Through Education and Implementation: Assessing the Effectiveness of the Watershed Stewards Program

https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=4824&context=joe

Effecting Land-Use Changes Through Education and Implementation: Assessing the Effectiveness of the Watershed Stewards Program

Abstract

Similar works

Full text

Available Versions

Clemson University: TigerPrints