Fowler-Dixon, Sarah

Jones, Martha F

Leary, Michael

Pinkston, Carla

Velders, Jeanne

Walsh, Mandi

Digital Commons@Becker

Washington University School of MedicineDigital Commons@BeckerHRPO Publications Human Research Protection Office2013Effective evaluation for a human research educationprogramSarah Fowler-DixonWashington University School of Medicine in St. LouisMartha F. JonesWashington University School of Medicine in St. LouisMichael LearyWashington University School of Medicine in St. LouisCarla PinkstonWashington University School of Medicine in St. LouisJeanne VeldersWashington University School of Medicine in St. LouisSee next page for additional authorsFollow this and additional works at: http://digitalcommons.wustl.edu/hrpopubsThis Other is brought to you for free and open access by the Human Research Protection Office at Digital Commons@Becker. It has been accepted forinclusion in HRPO Publications by an authorized administrator of Digital Commons@Becker. For more information, please contactengeszer@wustl.edu.Recommended CitationFowler-Dixon, Sarah; Jones, Martha F.; Leary, Michael; Pinkston, Carla; Velders, Jeanne; and Walsh, Mandi, "Effective evaluation for ahuman research education program" (2013). HRPO Publications. Paper 7.http://digitalcommons.wustl.edu/hrpopubs/7AuthorsSarah Fowler-Dixon, Martha F. Jones, Michael Leary, Carla Pinkston, Jeanne Velders, and Mandi WalshThis other is available at Digital Commons@Becker: http://digitalcommons.wustl.edu/hrpopubs/7Effective Evaluation for a Human Research Education Program Authors: Sarah Fowler-Dixon, PhD, CIP; Martha F. Jones, MA, CIP; Michael Leary, MA, CIP;  Carla Pinkston; Jeanne Velders, JD, RN, CIP; and Mandi Walsh, MSW        Decrease in Non- Compliance Informed IRB Members More Efficient  Application Review Informed HRPO Staff Institutional Research Data Informed Research Community Published Findings/ Ed. Guidance GOALS Community Visibility PROGRAM        CITI Training New IRB Member Training IRB Member Education IRB Chair Education HRPO Staff Training myIRB Training        SWAT! (On Call/Office Hours) Study Initiation Visit IND/IDE Training HRPO On Demand Education Compliance Education HRPO Website/Blog Conferences (Intramural) Conferences (Extramural) Open Access Library Ethics Series Podcasts/Media Necessary Elements Brown Bag Series R.C.R. Credit Activities St. Louis IRB Consortium CEnR Task Force WU IRB History Project Research Toolkits Community Outreach Research Projects Publishing PROGRAM        CITI Training New IRB Member Training IRB Member Education IRB Chair Education HRPO Staff Training myIRB Training        SWAT! (On Call/Office Hours) Study Initiation Visit IND/IDE Training HRPO On Demand Education Compliance Education HRPO Website/Blog Conferences (Intramural) Conferences (Extramural) Open Access Library Ethics Series Podcasts/Media Necessary Elements Brown Bag Series R.C.R. Credit Activities St. Louis IRB Consortium CEnR Task Force WU IRB History Project Research Toolkits Community Outreach Research Projects Publishing Study Initiation Visits: These visits occur after IRB approval has been received, but prior to the implementation of the study. Using templates, staff discusses key compliance and administration issues relevant to the conduct of the given study. Studies are selected for the Study Initiation Visit based on risk of compliance infraction due to study design, complexity, novelty, lack of research team experience and/or previous experiences.  HRPO Digital Commons, http://digitalcommons.wustl.edu/hrpo/:  the HRPO open access collection makes resources available in a convenient, searchable format.  The open access collection is PubMed and Google searchable. These educational materials are useful for HUMAN RESEARCH PROTECTION PROGRAM (HRPP) staff development, IRB member training, the education of researchers and students, and raising community awareness. Materials are broken into four major categories: Conferences, Human Research Protection Program (HRPP), HRPO Podcasts and HRPO Publications. HRPO On Demand: This consists of a structured set of topics and presentations that can be given at any time. Current On Demand topics include: Biobanking, Investigational Devices, IRB 101, Research Databases, Recruitment, and Translational Research. Research Projects: To maintain quality and to contribute to generalizable knowledge concerning the conduct of human subjects research, HRPO participates or initiates research.  Examples include: participation in national or IRB surveys, criteria assessments for IRB meetings, program evaluation,  health literacy efforts, diversity and international initiatives. Problem/Issue Statement: Many Human Research Protection Programs (HRPPs) have responded to the growing complexity of federal requirements, guidance documents and human subjects review processes with educational programs of various sizes and formats.  Offering any type of educational program is costly and time-intensive making program evaluation an integral part.   Due to the fragmented nature of most HRPP educational efforts, robust program evaluations are not always in place and good national models are not available. Washington University has developed such a model for its large-scale education program that can be adopted by smaller programs, helping to increase effectiveness while saving time and costly mistakes.   Description of Program/Research: Using established qualitative and quantitative program evaluation models, the Education Team began to apply these principles to the comprehensive human research protection program in existence. Information garnered helped guide future efforts and assisted in determining which current offerings, most effective in helping reach goals, were to remain.  Program development took the following steps:   To determine if the HRPP education program is meeting its goals and aims, current data collection instruments were evaluated and subsequently enhanced to collect consistent data points across programs. Data points collected include: individuals’ first and last names, type of service, date, department/ affiliation, topic.  This allows us to track individuals’ involvement in the program, frequently used services, and current issues being raised which can then be shown in generated reports. The Education Team is currently working on correlating use of educational programming to IRB concerns related to time to approval, number of contingencies, and issues with non-compliance.  Evaluation such as this takes the cooperation of every individual involved in the process and requires dedicated staff time for tracking. The benefits are not only an effective education program tailored to the needs of the audience but also provides a mechanism by which institutional leadership can be apprised of educational needs and efforts being made to meet those needs.  This is also a system by which individuals can be held accountable for learning and knowledge related to compliance issues.   Additional Information: This program evaluation is unique in that it is being done on offerings designed for those engaged or interested in  human subjects research using adult education learning theory where little information regarding program effectiveness has been disseminated widely.  Results of these efforts will be made public via the Washington University Human Research Protection Program Digital Commons Open Access Library found at:  http://digitalcommons.wustl.edu/hrpo/  References: Cronbach LJ; Ambron SR ; Dornbusch SM ; Hess RD; R C Hornik RC; Phillips DC ; Walker DF ; Weiner; SS. Toward Reform of Program Evaluation. San Fransisco, CA, US: Jossey-Bass Publishers (1980); Patton, MQ. Qualitative Evaluation Methods. Thousand Oaks, CA: Sage Publications (1980); Posavac, Emil J.; Carey, Raymond G. Program Evaluation: Methods and Case Studies (5th ed.). Englewood Cliffs, NJ, US: Prentice-Hall, Inc. (1997).  1. Identifying Educational Programs: Twenty-seven (27) currently available educational outlets for the HRPP education program were identified. This process enabled us to see potential gaps in our educational programming, and subsequently created additional potential programs. (See Diagram 1)  2. Identifying Goals:  The initial assessment structure was completed by identifying intended program goals, based on objectives identified within each of the current 27 educational offerings. (See Diagram 2)  3. Individual educational goals were linked to broader more comprehensive outcomes/aims.  (See Diagram 3)  4.  Identifying Stakeholders impacted by the programs: Primary Stakeholders were identified as: Faculty Investigators, Research Staff, Student Researchers, Students, IRB Members, IRB Staff,  St. Louis Community, National, and International Audiences. (See Diagram 4)  OUTCOMES IRB/HRPO Staff Proficiency Expansion of Ethical and Regulatory Discourse Community Engagement and Awareness Awareness and Transparency for the Research Community Novel IRB Research And Publication Decrease in Non- Compliance Informed IRB Members More Efficient  Application Review Informed HRPO Staff Institutional Research Data Informed Research Community Published Findings/ Ed. Guidance GOALS Community Visibility PROGRAM        CITI Training New IRB Member Training IRB Member Education IRB Chair Education HRPO Staff Training myIRB Training        SWAT! (On Call/Office Hours) Study Initiation Visit IND/IDE Training HRPO On Demand Education Compliance Education HRPO Website/Blog Conferences (Intramural) Conferences (Extramural) Open Access Library Ethics Series Podcasts/Media Necessary Elements Brown Bag Series R.C.R. Credit Activities St. Louis IRB Consortium CEnR Task Force WU IRB History Project Research Toolkits Community Outreach Research Projects Publishing Key:           Implemented Program          Collaborative Program          Program In Progress The SWAT! (Staff With Answers Today!) Program offers a staff member “On Call” during all business hours and four total Office Hours sessions per week. In order to meet goals like “More Efficient IRB Application” and “Informed Research Community,” data from each individual SWAT! contact are collected (e.g. Name, Department, Issue Addressed, etc.). This allows us to correlate SWAT! consumer data with IRB application data, which enables us to identify potential educational topics, help us better understand the needs of our research community, and demonstrate the overall effectiveness of the program.   The below chart correlates  SWAT! consumers by department with IRB application rates for each month. As noted above, this specific measure enables us to demonstrate that we are partnering with key institutional stakeholders in an educational context to achieve more efficient application timelines and better overall compliance. HRPO -- Human Research Protection Office Diagram 3 Diagram 1 Diagram 4 Diagram 2 Number  of  Pageviews Number  of  Downloads  COLLECTION SUMMARY  10/1/2012 - 10/1/2013  Creation Date 10/1/12  Total Pages Viewed 7,379  Total Items Downloaded 2,832 Total Items 108  Countries Represented 36  Items Downloaded at Least Once 103      1400  1200  1000       959 Pages Viewed By Month   1170   901    1258    450 400 350 Items Downloaded By Month 423      335   800  600  400  200  0      392   664      389       283    253 841    591      349 300 250 200 150 100 50  2 0   213 242 237 248   213 252 232    177 260        The HRPO open access collection makes resources available in a convenient, searchable format.  The open access collection is PubMed and Google searchable. Google Analytics automates the collection of usage and visitor data. The below diagram represents basic usage statistics for the dates indicated. Additional visitor data enables us to demonstrate that this educational program is addressing the intended populations served (e.g. “International,” “National,” and “Research Staff.”)  For example, our visitor data indicates that the following representative entities from among 36 countries have accessed this open access collection and downloaded items:  • Washington University in St. Louis: 2,380 • IRBForum.org: 1,508 • US-Based Universities: 1,362 • Veteran’s Administration/Military: 249 • Federal (e.g. NIH and FDA): 208  Educational Program by Population ServedHRPO Website/BlogOpen Access LibraryPublishingPodcasts/MediaConferences (Intramural)CITI TrainingEthics SeriesHRPO On Demand EducationR.C.R. CreditNew IRB Member TrainingmyIRB TrainingSWAT! (On Call/Office Hours)WU IRB History ProjectNecessary ElementsCEnR Task ForceResearch ToolkitsResearch ProjectsIRB Member EducationStudy Initiation VisitCompliance EducationConferences (Extramural)Brown Bag SeriesIRB Chair EducationHRPO Staff TrainingIND/IDE TrainingCommunity OutreachSt. Louis IRB ConsortiumHRPO StaffIRB MembersFaculty InvestigatorsResearch StaffStudents as ResearchersStudents as StudentsWUSTLResearch ParticipantsSt. Louis Comm. NationalInternational11 11 11 10 8 7 7 6 6 5 5 5 5 4 4 4 4 3 3 3 3 3 2 2 2 2 1

Effective evaluation for a human research education program

http://digitalcommons.wustl.edu/cgi/viewcontent.cgi?article=1006&context=hrpopubs

Effective evaluation for a human research education program

Abstract

Similar works

Full text

Available Versions

Digital Commons@Becker