Bond, William, MD, MS

Kane, Bryan G, MD

Miller, Andrew C, DO

Orlando, James P, EdD

Pasquale, Michael, MD

Rupp, Valerie A, RN, BSN

Umbrell, Cindy, RN, MSN

English

Lehigh Valley Health Network: LVHN Scholarly Works

Lehigh Valley Health NetworkLVHN Scholarly WorksDepartment of Emergency MedicineImpact Study of a Central Lines SimulationTraining Program Using Kirkpatrick’s Four-LevelEvaluation ModelJames P. Orlando EdDLehigh Valley Health NetworkAndrew C. Miller DOLehigh Valley Health Network, Andrew_C.Miller@lvhn.orgWilliam Bond MD, MSLehigh Valley Health Network, University of South Florida, william.bond@osfhealthcare.orgValerie A. Rupp RN, BSNLehigh Valley Health Network, Valerie.Rupp@lvhn.orgBryan G. Kane MDLehigh Valley Health Network, bryan.kane@lvhn.orgSee next page for additional authorsFollow this and additional works at: http://scholarlyworks.lvhn.org/emergency-medicinePart of the Emergency Medicine Commons, Medical Education Commons, Nursing Commons,and the Surgery CommonsThis Poster is brought to you for free and open access by LVHN Scholarly Works. It has been accepted for inclusion in LVHN Scholarly Works by anauthorized administrator. For more information, please contact LibraryServices@lvhn.org.Published In/Presented AtOrlando, J., Miller, A., Bond, W., Rupp, V., & Kane, B. (2010, March 4-7). Impact study of a central lines simulation training program usingkirkpatrick’s four-level evaluation model. Poster presented at: The 2010 ACGME Annual Educational Conference, Nashville, TN.AuthorsJames P. Orlando EdD; Andrew C. Miller DO; William Bond MD, MS; Valerie A. Rupp RN, BSN; Bryan G.Kane MD; Cindy Umbrell RN, MSN; and Michael Pasquale MDThis poster is available at LVHN Scholarly Works: http://scholarlyworks.lvhn.org/emergency-medicine/168Impact Study of a Central Lines Simulation Training Program Using Kirkpatrick’sFour-Level Evaluation ModelLehigh Valley Health Network, Allentown, PennsylvaniaBackground:James P. Orlando, Ed.D; Andrew Miller, DO; William Bond, MD, MS; Valerie Rupp, RN, BSN; Bryan Kane, MD; Cindy Umbrell, RN, MSN; Michael Pasquale, MDResults:•		The	breakneck	speed	of	medical	science	and	technology	far	outpaces	the	current	capabilities	of	our	healthcare	delivery	system	and	theprofessionals	who	work	in	them	to	prevent	medical	errors.•		Central	line-associated	infections	are	preventable	events	that	harm	patients	and	generate	significant	costs.•		Key	contributing	factors	to	these	adverse	events	include	insufficient	training,	lack	of	provider	collaboration	and	uneven	application	of	evidence-based	patient	safety	protocols.•		There	is	a	paucity	of	evidence	regarding	the	long-term	impact	of	simulation	training	on	patient	outcomes.•		Assessing	training	effectiveness	often	entails	using	the	four-level	model	developed	by	Donald	Kirkpatrick.		As	shown	in	Figure	1,	Kirkpatrick’s	model	is	based	upon	the	premise	that	training	programs	can	be	evalauted	across	four	levels:		 	 	 -	 Level	1	(reaction)	 	 	 -	 Level	2	(knowledge)	 	 	 -	 Level	3	(application)	 	 	 -	 	Level	4	(impact)•		According	to	this	model,	information	from	each	prior	level	serves	as	a	base	for	the	next	level’s	evaluation.	Thus,	each	successive	level	represents	a	more	precise	measure	of	the	effectiveness	of	the	training	program,	but	at	the	same	time	requires	a	more	rigorous	and	time-consuming	analysis.•	The	study	utilizes	a	mixed	methods,	quantitatiave/qualitative	approach	to:	 	 	 -	 (a)	determine	reliability	of	evaluation	instruments	 	 	 -	 (b)	understand	perceptions	of	resident	course	participants	and	nurses	 	 	 -	 (c)	determine	the	relationships	between	course	outcomes,	operator												practices,	and	patient	outcomes	 	 	 -	 	(d)	validate	the	accuracy	of	data	collected	on	the	procedural	checklist	 	 	 -	 (e)	determine	how	changes	in	training,	policies	and	protocols	impact	a												hospital’s	central	line-associated	infections	trend	rate.Objectives:The	study	evaluates	the	impact	of	central	lines	simulation	training	program	from	2006-2009	on	learner	and	patient	outcomes	within	the	framework	of	Kirkpatrick’s	four-level	evaluation	model.Methods:•		Level	1	course	evaluations	indicated	that	simulation,	small	group	teaching,	and	feedback	were	useful,	relevant	and	motivating	to	participants.•		Level	2	t-tests	and	ANCOVA	analyses	of	knowledge	scores	showed	significant	knowledge	gains	and	retention	within	and	between	cohorts.•		Level	3	chi-square	and	linear	regression	analyses	suggested	that	operators’	Institute	for	Healthcare	Improvement	(IHI)	central	lines	bundle	compliance	rate	predicted	their	complications	rate.•		Focus	group	data	suggested	that	having	a	nurse	in	the	room	had	an	unanticipated	effect	of	reducing	the	number	of	attempts	by	course	taking	residents	and	therefore,	lowering	complications	rate.•		Cronbach’s	Alpha	on	the	Level	3	procedural	checklist	instruments	showed	“good”	reliability.•		As	shown	in	Figure	2,	Level	4	time-series	analysis	suggested	that	central	lines	training	and	changes	in	policies	and	practices	had	significantly	reduced	the	hospital’s	central	line-associated	infection	trend	rate	since	April	2005.Significance:•		This	study	produced	evidence	suggesting	that	interprofessional	simulation	training	contributes	to	better	resident	adherence	with	IHI	Central	Lines	Bundle	and	lower	complication/infection	rates	than	if	the	course	did	not	exist.•		Performance	support	mechanisms	introduced	in	the	course	and	present	in	the	clinical	setting,	such	as	a	central	lines	checklist	and	peer	support,	reinforce	course	learnings	and	enable	skill	transfer.•		Kirkpatrick’s	Four-Level	Evaluation	Model	is	a	promising	framework	for	evaluating	the	impact	of	clinical	training	programs	on	resident	performance.Figure 2: LVHN Central Lines Acquired Bloodstream Infection RateApril	2005	-	December	2008Figure 1: Kirkpatrick’s Four Level Evaluation ModelImpact	Study	of	a	Central	Lines	Simulation	Training	Patient and Business Outcomes(i.e. Level 4: lower complication/infection ratesAnd financial impact)Feedback Loop #2Feedback Loop #1Competent Learners(i.e. Level 3: improved bundle complianceAnd insertion techniques)Quality of Curriculum(i.e. Level 1: improved design of central lines simulationCourse; per learners’ rating of course quality)Learner Outcomes(i.e. Level 2: improved skill, knowledge, and attitude ofCentral lines placement technique and bundle)Quality of Instruction(i.e. Level 1: improved teaching of central lines simulationCourse, per learners’ rating of quality teaching)Adapted diagram based on McLean et al 2008, MEDICAL TEACHERAnd Alliger et al, 1989, PERSONNEL PSYCHOLOGYCase NumberValue0.000.501.001.502.002.503.00Observed CLAB RatePredicted CLAB RateJune 06 – Cohort AJune 07 – Cohort BChecklist and OtherProtocols IntroducedJune 08 – Cohort COct 08 – Cohort DTime Series ModelP-value <.051 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45

Impact Study of a Central Lines Simulation Training Program Using Kirkpatrick’s Four-Level Evaluation Model

https://scholarlyworks.lvhn.org/cgi/viewcontent.cgi?article=1167&context=emergency-medicine

Impact Study of a Central Lines Simulation Training Program Using Kirkpatrick’s Four-Level Evaluation Model

Abstract

Similar works

Full text

Available Versions

Lehigh Valley Health Network: LVHN Scholarly Works