Validation Evidence from using Generalizability Theory in a Basic-Science Course: Reliability of Course-Grades From Multiple Examinations

Description of the Problem: Reliability is critical validation evidence on which to base high-stakes decision-making. Many times, one exam in a didactic course may not be acceptably reliable on its own. But how much might multiple exams add when combined together? The Innovation: To improve validation evidence towards high-stakes decision-making, Generalizability Theory (G-Theory) can combine reliabilities from multiple exams into one composite-reliability (G_String IV software). Further, G-Theory decision-studies can illustrate changes in course-grade reliability, depending on the number of exams and exam-items. Critical Analysis: 101 first-year PharmD students took two midterm-exams and one final-exam in a pharmaceutics course. Individually, Exam1 had 50MCQ (KR-20=0.69), Exam2 had 43MCQ (KR-20=0.65), and Exam3 had 67MCQ (KR-20=0.67). After combining exam occasions using G-Theory, the composite-reliability was 0.71 for overall course-grades—better than any exam alone. Remarkably, increased numbers of exam occasions showed fewer items per exam were needed, and fewer items over all exams, to obtain an acceptable composite-reliability. Acceptable reliability could be achieved with different combinations of number of MCQs on each exam and number of exam occasions. Implications: G-Theory provided reliability critical validation evidence towards high-stakes decision-making. Final course-grades appeared quite reliable after combining multiple course exams—though this reliability could and should be improved. Notably, more exam occasions allowed fewer items per exam and fewer items over all the exams. Thus, one added benefit of more exam occasions for educators is developing fewer items per exam and fewer items over all exams

Use of a Team-Based Video Simulation to Complement a Lecture in Motivational Interviewing to Develop Students’ Initial Attitudes and Skills

Background: Motivational interviewing (MI) is increasingly recognized for its patient-centered approach to clinician-patient communication and often effective in evoking appropriate changes in patient behavior.  Doctor of Pharmacy (PharmD) programs are increasingly incorporating MI education; however, doing so represents a challenge regarding availability of both time and labor capital. Case Description: This study reports on the use of a 90-minute software-based simulation and tutorial coupled with a 90-minute lecture in a 3-hour course session. In a subsequent session consisting of several brief interactions with standardized patients (SPs), students reflected upon their strengths and challenges as a result of attempting to apply the MI principles they learned during the educational intervention. Case Themes: Students’ responses to a questionnaire delivered both before and after the simulation and lecture, showed improvements in several attitudinal components related to use of MI, as well as developing self-efficacies in deploying patient-centered communication strategies. A post-intervention survey without students’ opportunity to study/prepare saw gains in student knowledge about MI. Case Impact: The measurements employed to determine student’s attitude and knowledge showed good to very good internal consistency reliably based on calculated Cronbach’s alpha and KR-20. Student reflections indicated their understanding of MI principles, even though they fell short of implementing them in large part during their encounters with SPs. Conclusion: Use of a video simulation undergone by teams of PharmD students coupled with a brief lecture might be an effective and efficient way of building a foundation for MI competency among PharmD students where instructors might lack labor capital and significant space in the curriculum

A Comparison of the Knowledge of Alzheimer’s Disease among Community Pharmacists Based on Regional Practice Setting Using the Alzheimer’s Disease Knowledge Scale (ADKS)

Alzheimer’s Disease (AD) is the most prevalent type of dementia and has a high prevalence in the aging population. Due to the requirements for care, it is valuable for members of the healthcare system to be knowledgeable about AD and its treatment. Community pharmacists are particularly valuable in assisting in AD care because of their accessibility to the general public and they are a source for dispensing AD medications. The purpose of this study is to assess the level of knowledge of AD among Ohio community pharmacists based on regional practice settings using the Alzheimer’s Disease Knowledge Scale (ADKS). A standardized, 30-question assessment tool known as the Alzheimer’s Disease Knowledge Scale exists to measure the level of knowledge of AD among a given population. Studies on AD knowledge have been conducted using this tool, and the use of this scale identifies major misconceptions about AD that could be addressed in education initiatives for both the general public and for health care professionals. This study implemented this tool to compare the knowledge of community pharmacists that serve rural areas to those that serve urban areas. The Ohio Board of Pharmacy provided a list of emails of community pharmacists to send an anonymous survey including both demographic questions and the ADKS. Statistical analyses including independent t-tests and descriptive statistics were conducted using SPSS to determine the significance of the data. This study found no statistically significant difference between urban versus rural community pharmacists. However, in the future we hope to identify specific questions that correlate to areas of AD where continuing education may be helpful in improving patient outcomes by enabling caregivers and healthcare providers with a more accurate understanding of AD

Educational Approach to Increase Student Pharmacist Confidence with Medicare

Objective: To describe a didactic, case-based approach to teach student pharmacists about Medicare enabling them to consider the spectrum of coverage options, as well as both medical and drug needs, when assessing insurance plans. Innovation: Education on Medicare often focuses on Part D.   However, the growing popularity of Medicare Advantage plans requires students to have a more comprehensive understanding of Medicare.  To address this gap, a didactic, case-based approach was developed where students received four 90-minute lectures on Medicare and then applied the information through a patient case.   Data was collected on student-reported confidence with Medicare, and attitudes towards the importance of understanding insurance in improving patient care.  Surveys were conducted at baseline, after the didactic lectures (interim survey), and upon project completion (final survey).  Key Findings: Confidence with Medicare increased between the baseline and interim survey (all p ≤ 0.001).  Additional gains were seen after completing the project in helping patients navigate financial assistance programs (p ≤ 0.001) and selecting specific plans (p ≤ 0.05).  After the interim survey, students more strongly agreed with statements related to knowledge of insurance as an opportunity to help advocate for patients (p ≤ 0.001), and the need to consider Medicare coverage when making medication recommendations (p ≤ 0.05) Conclusion: The combination of didactic lectures and a case-based exercise positively impacted self-reported student confidence in their knowledge, as well as attitude towards Medicare.   Increasing student confidence in Medicare and willingness to engage beneficiaries in conversations about insurance coverage may be a strategy to improve patient care and health outcomes.

Providing Validation Evidence for a Clinical-Science Module: Improving Testing Reliability with Quizzes

Description of the Problem: High-stakes decision-making should have sound validation evidence; reliability is vital towards this. A short exam may not be very reliable on its own within didactic courses, and so supplementing it with quizzes might help. But how much? This study’s objective was to understand how much reliability (for the overall module-grades) could be gained by adding quiz data to traditional exam data in a clinical-science module. The Innovation: In didactic coursework, quizzes are a common instructional strategy. However, individual contexts/instructors can vary quiz use formatively and/or summatively. Second-year PharmD students took a clinical-science course, wherein a 5-week module focused on cardiovascular therapeutics. Generalizability Theory (G-Theory) combined seven quizzes leading to an exam into one module-level reliability, based on a model where students were crossed with items nested in eight fixed testing occasions (mGENOVA used). Furthermore, G-Theory decision-studies were planned to illustrate changes in module-grade reliability, where the number of quiz-items and relative-weighting of quizzes were altered. Critical Analysis: One-hundred students took seven quizzes and one exam. Individually, the exam had 32 multiple-choice questions (MCQ) (KR-20 reliability=0.67), while quizzes had a total of 50MCQ (5-9MCQ each) with most individual quiz KR-20s less than or equal to 0.54. After combining the quizzes and exam using G-Theory, estimated reliability of module-grades was 0.73; improved from the exam alone. Doubling the quiz-weight, from the syllabus’ 18% quizzes and 82% exam, increased the composite-reliability of module-grades to 0.77. Reliability of 0.80 was achieved with equal-weight for quizzes and exam. Next Steps: Expectedly, more items lent to higher reliability. However, using quizzes predominantly formatively had little impact on reliability, while using quizzes more summatively (i.e., increasing their relative-weight in module-grade) improved reliability further. Thus, depending on use, quizzes can add to a course’s rigor

Initial Validation Evidence for Clinical Case Presentations by Student Pharmacists

Objective: There is a paucity of validation evidence for assessing clinical case-presentations by Doctor of Pharmacy (PharmD) students.  Within Kane’s Framework for Validation, evidence for inferences of scoring and generalization should be generated first.  Thus, our objectives were to characterize and improve scoring, as well as build initial generalization evidence, in order to provide validation evidence for performance-based assessment of clinical case-presentations. Design: Third-year PharmD students worked up patient-cases from a local hospital.  Students orally presented and defended their therapeutic care-plan to pharmacist preceptors (evaluators) and fellow students.  Evaluators scored each presentation using an 11-item instrument with a 6-point rating-scale.  In addition, evaluators scored a global-item with a 4-point rating-scale.  Rasch Measurement was used for scoring analysis, while Generalizability Theory was used for generalization analysis. Findings: Thirty students each presented five cases that were evaluated by 15 preceptors using an 11-item instrument.  Using Rasch Measurement, the 11-item instrument’s 6-point rating-scale did not work; it only worked once collapsed to a 4-point rating-scale.  This revised 11-item instrument also showed redundancy.  Alternatively, the global-item performed reasonably on its own.  Using multivariate Generalizability Theory, the g-coefficient (reliability) for the series of five case-presentations was 0.76 with the 11-item instrument, and 0.78 with the global-item.  Reliability was largely dependent on multiple case-presentations and, to a lesser extent, the number of evaluators per case-presentation.  Conclusions: Our pilot results confirm that scoring should be simple (scale and instrument).  More specifically, the longer 11-item instrument measured but had redundancy, whereas the single global-item provided measurement over multiple case-presentations.  Further, acceptable reliability can be balanced between more/fewer case-presentations and using more/fewer evaluators

Evaluation of an Outpatient Pharmacist Consult Service at a Large Academic Medical Center

Objective: To evaluate a novel outpatient pharmacist consult service in a large academic medical center. Setting: Four outpatient pharmacies that are part of a large academic medical center Methods: An outpatient pharmacist consult order was created and embedded in the electronic medical record (EMR). Medical center providers utilized this consult order when identifying patients in need of specific services provided by outpatient pharmacists. Descriptive data about each individual consult was collected including number completed, type of service, and duration. Rate of accepted pharmacy recommendations and patient cost savings were also evaluated. A survey was administered at the completion of the study period to assess provider and pharmacist satisfaction with the service.  Patient demographic information was collected for those who had a documented completed consult. Results: A total of 193 consults were completed: 137 immunizations, 37 care affordability, 15 education, 3 polypharmacy and 1 OTC recommendation. 89% of completed consults took pharmacists 20 minutes or less to complete. Of completed care affordability consults (n=31), 55% of patients saved between $100 -$500 per medication fill. Of providers who completed a survey and utilized the service (n=12), 83.3% were extremely satisfied and 16.7% were satisfied with it. The provider acceptance rate of pharmacist’s recommendations was 74%. Conclusion: Implementation of an outpatient pharmacist consult service provided an alternative method for the utilization of pharmacist provided MTM services in outpatient pharmacies at a large academic medical center. The service was well received by both providers and pharmacists

Moving beyond Cronbach’s Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education

Background: When available, empirical evidence should help guide decision-making. Following each administration of a learning assessment, data becomes available for analysis. For learning assessments, Kane’s Framework for Validation can helpfully categorize evidence by inference (i.e., scoring, generalization, extrapolation, implications). Especially for test-scores used within a high-stakes setting, generalization evidence is critical. While reporting Cronbach’s alpha, inter-rater reliability, and other reliability coefficients for a single measurement error are somewhat common in pharmacy education, dealing with multiple concurrent sources of measurement error within complex learning assessments is not. Performance-based assessments (e.g., OSCEs) that use raters, are inherently complex learning assessments. Primer: Generalizability Theory (G-Theory) can account for multiple sources of measurement error. G-Theory is a powerful tool that can provide a composite reliability (i.e., generalization evidence) for more complex learning assessments, including performance-based assessments. It can also help educators explore ways to make a learning assessment more rigorous if needed, as well as suggest ways to better allocate resources (e.g., staffing, space, fiscal). A brief review of G-Theory is discussed herein focused on pharmacy education. Moving Forward: G-Theory has been common and useful in medical education, though has been used rarely in pharmacy education. Given the similarities in assessment methods among health-professions, G-Theory should prove helpful in pharmacy education as well. Within this Journal and accompanying this Idea Paper, there are multiple reports that demonstrate use of G-Theory in pharmacy education

Motivating Students to Engage in Preparation for Flipped Classrooms by Using Embedded Quizzes in Pre-class Videos

Background: The success of flipped classrooms is dependent upon students’ preparation prior to class, the lack of which is the most common challenge associated with this teaching methodology. To mitigate this limitation, it is important to develop and assess methods of engaging learners during pre-class activities. Objective: To determine if quiz delivery method (embedded throughout versus after pre-class videos) affects students’ knowledge retention, grades, and video viewing behavior. Methods:  Participating students were randomized to take quizzes, either during pre-class videos via Panopto™ (EQV) or after pre-class videos in the traditional manner via Moodle™ (TMM).  Outcomes assessed included students’ knowledge retention, scores on pre- and post-class quizzes, and pre-class video viewing behavior (total views and minutes viewed per student) during a three-week period.  Having experienced both quizzing modalities during the semester, the perceptions from students in the EQV group were surveyed. Results: Baseline assessment results of both groups (n = 27 per group) were comparable with a median score of 33% (IQR: 17, 50) in both groups.  Performance was also similar on knowledge retention [TMM: 67% (50, 83) vs. EQV: 83% (50, 83)], pre-class quiz scores [TMM: 90% (87, 97) vs. EQV: 93% (90, 95)], and post-class quiz scores [TMM: 93% (80, 100) vs. EQV: 87% (80, 100)], while students in the EQV group had more total views [10 (8, 12)] vs. [5 (2, 11)] and minutes viewed [71 (36, 108) vs. 35 (15, 81)]. Results from the perception survey administered to students in the EQV group (74.1% response rate) indicated a preference for embedded quizzes overall (58%) and for class preparation (75%) when compared with post-video quizzes. Conclusion: Students’ knowledge retention and performance were similar in both EQV and TMM groups, though students in the EQV group were more engaged with videos and most of them preferred this quiz delivery.  Using embedded quizzes for formative rather than summative assessment might be an appropriate mechanism to encourage students’ viewing of pre-class videos and their preparation for flipped classes

Validation Evidence using Generalizability Theory for an Objective Structured Clinical Examination

Objectives: Performance-based assessments, including objective structured clinical examinations (OSCEs), are essential learning assessments within pharmacy education. Because important educational decisions can follow from performance-based assessment results, pharmacy colleges/schools should demonstrate acceptable rigor in validation of their learning assessments. Though G-Theory has rarely been reported in pharmacy education, it would behoove pharmacy educators to, using G-Theory, produce evidence demonstrating reliability as a part of their OSCE validation process. This investigation demonstrates the use of G-Theory to describes reliability for an OSCE, as well as to show methods for enhancement of the OSCE’s reliability. Innovation: To evaluate practice-readiness in the semester before final-year rotations, third-year PharmD students took an OSCE. This OSCE included 14 stations over three weeks. Each week had four or five stations; one or two stations were scored by faculty-raters while three stations required students’ written responses. All stations were scored 1-4. For G-Theory analyses, we used G_Strings and then mGENOVA.  Critical Analysis: Ninety-seven students completed the OSCE; stations were scored independently. First, univariate G-Theory design of students crossed with stations nested in weeks (p x s:w) was used. The total-score g-coefficient (reliability) for this OSCE was 0.72. Variance components for test parameters were identified. Of note, students accounted for only some OSCE score variation. Second, a multivariate G-Theory design of students crossed with stations (p· x s°) was used. This further analysis revealed which week(s) were weakest for the reliability of test-scores from this learning assessment. Moreover, decision-studies showed how reliability could change depending on the number of stations each week. For a g-coefficient >0.80, seven stations per week were needed. Additionally, targets for improvements were identified. Implications: In test validation, evidence of reliability is vital for the inference of generalization; G-Theory provided this for our OSCE. Results indicated that the reliability of scores was mediocre and could be improved with more stations. Revision of problematic stations could help reliability as well. Within this need for more stations, one practical insight was to administer those stations over multiple weeks/occasions (instead of all stations in one occasion)