Background: Paramedic programs use formative assessments to determine cognitive competency. Understanding the number of failed formative units as a probability of passing the summative exam will allow programs to set additional benchmarks. The purpose of this study was to determine whether failure in formative exams determines success on a summative exam.
Methods: Formative and summative scores from 2011 – 2016 for paramedic students with accounts in Fisdap™, an Internet-based administrative database, were retrospectively reviewed for the following criteria: provided consent for research, completed all six formative (unit) examinations, and completed a summative (comprehensive) examination. Analyses were performed with Pearson correlations and linear regression. Results: A total of 1,406 student records were included based on inclusion criteria. Correlation with each formative and the summative examination were all significant, p \u3c 0.001: Cardiology 0.597; Airway 0.571; Medical 0.571; Trauma 0.566; Ob/Pediatrics 0.549; Operations 0.495. The cardiology exam was shown to have a moderate correlation on summative performance, whereas the operations exam had the weakest correlation. The number of formative examination failures was a significant predictor of the probability of passing the summative examination, t(1405) = –31.02, p \u3c 0.001. Zero failed unit examinations yielded a 100% probability of passing. Three failed formative exams yielded a 60.4% probability. Four failed attempts yielded a 44.8% probability. Failure of all six formative exams yielded a 13.4% probability of passing the Paramedic Readiness Exam Version 3. Conclusion: Not all formative examinations hold the same predictive power on the probability of passing a summative examination. Each had their own correlation value. Students who did not fail formative examinations have a 100% likelihood of passing the summative examination

Batt, Alan M

Berry, Jennifer C.

Dinsch, James

Fentress, Tom

Leggio, William J, Jr

Vosper, Marilee L

Walsh, Kelly

English

NSU Works

Internet Journal of Allied Health Sciencesand PracticeVolume 15 | Number 4 Article 2October 2017Correlation of formative assessments as the meansof predicting summative performance in paramedicstudentsWilliam J. Leggio JrCreighton University, williamleggio@creighton.eduAlan M. BattFanshawe College, ON, Canada, alan@batt.meJennifer C. BerryFisdap, jberry@fisdap.netTom FentressMethodist Hospitals, tomfentress@gmail.comMarilee L. VosperAmerican Ambulance, mvosper@americanambulance.comSee next page for additional authorsFollow this and additional works at: https://nsuworks.nova.edu/ijahspPart of the Medical Education CommonsThis Manuscript is brought to you for free and open access by the College of Health Care Sciences at NSUWorks. It has been accepted for inclusion inInternet Journal of Allied Health Sciences and Practice by an authorized editor of NSUWorks. For more information, please contactnsuworks@nova.edu.Recommended CitationLeggio WJ, Batt AM, Berry JC, Fentress T, Vosper ML, Walsh K, et al. Correlation of formative assessments as the means of predictingsummative performance in paramedic students. The Internet Journal of Allied Health Sciences and Practice. 2017 Oct 03;15(4),Article 2.Correlation of formative assessments as the means of predicting summative performance in paramedicstudentsBackground: Paramedic programs use formative assessments to determine cognitive competency.Understanding the number of failed formative units as a probability of passing the summative exam willallow programs to set additional benchmarks. The purpose of this study was to determine whether failure informative exams determines success on a summative exam.Methods: Formative and summative scores from 2011 – 2016 for paramedic students with accounts inFisdap™, an Internet-based administrative database, were retrospectively reviewed for the following criteria:provided consent for research, completed all six formative (unit) examinations, and completed a summative(comprehensive) examination. Analyses were performed with Pearson correlations and linear regression.Results: A total of 1,406 student records were included based on inclusion criteria. Correlation with eachformative and the summative examination were all significant, p < 0.001: Cardiology 0.597; Airway 0.571;Medical 0.571; Trauma 0.566; Ob/Pediatrics 0.549; Operations 0.495. The cardiology exam was shown to havea moderate correlation on summative performance, whereas the operations exam had the weakest correlation.The number of formative examination failures was a significant predictor of the probability of passing thesummative examination, t(1405) = –31.02, p < 0.001. Zero failed unit examinations yielded a 100% probabilityof passing. Three failed formative exams yielded a 60.4% probability. Four failed attempts yielded a 44.8%probability. Failure of all six formative exams yielded a 13.4% probability of passing the Paramedic ReadinessExam Version 3. Conclusion: Not all formative examinations hold the same predictive power on theprobability of passing a summative examination. Each had their own correlation value. Students who did notfail formative examinations have a 100% likelihood of passing the summative examination.Author Bio(s)William Leggio, EdD, NRP is assistant professor and paramedic program coordinator at Creighton Universityin Omaha, Nebraska.Alan Batt, MSc, CCP, is faculty in the paramedic programs at Fanshawe College, Ontario, CanadaJennifer C. Berry, BA, NREMT is a Product Owner at Fisdap in St. Paul, Minnesota.Tom Fentress, BS, NRP is EMS Coordinator at Methodist Hospitals in Gary, Indiana.Marilee L. Vosper, M.Ed., NRP is Community Education Manager at American Ambulance in Fresno,CaliforniaKelly Walsh, BSN, RN is EMS Academy Program Director at Advance Medical Transport of Central Illinois inPeoria, Illinois.James Dinsch, MS, NRP is Department Chair and Program Director of Emergency Medical Services Academyat Indian River State College in Fort Pierce, FloridaAcknowledgements1. This study was conducted during the Fisdap Research Summit in February, 2016. 2. The findings of thisstudy were presented as an abstract poster and oral presentation during the annual Prehpospital Care ResearchForum at the National Association of EMS Educators conference in August, 2016.This manuscript is available in Internet Journal of Allied Health Sciences and Practice:https://nsuworks.nova.edu/ijahsp/vol15/iss4/2AuthorsWilliam J. Leggio Jr, Alan M. Batt, Jennifer C. Berry, Tom Fentress, Marilee L. Vosper, Kelly Walsh, and JamesDinschThis manuscript is available in Internet Journal of Allied Health Sciences and Practice:https://nsuworks.nova.edu/ijahsp/vol15/iss4/2 © The Internet Journal of Allied Health Sciences and Practice, 2017  Dedicated to allied health professional practice and education  Vol. 15 No. 4 ISSN 1540-580X  Correlation of Formative Assessments as the Means of Predicting Summative Performance in Paramedic Students      William J. Leggio Jr, EdD, NRP1      Alan M. Batt, MSc, CCP2      Jennifer Berry, BA, NREMT3      Tom Fentress, BS, NRP4      Marilee L. Vosper, M.Ed., NRP5      Kelly Walsh, BSN, RN6      James Dinsch, MS, NRP7  1. Creighton University 2. Fanshawe College 3. Fisdap 4. Methodist Hospitals 5. American Ambulance 6. Advance Medical Transport 7. Indian River State College  United States/Canada ABSTRACT Background: Paramedic programs use formative assessments to determine cognitive competency. Understanding the number of failed formative units as a probability of passing the summative exam will allow programs to set additional benchmarks. The purpose of this study was to determine whether failure in formative exams determines success on a summative exam.  Methods: Formative and summative scores from 2011 – 2016 for paramedic students with accounts in Fisdap™, an Internet-based administrative database, were retrospectively reviewed for the following criteria: provided consent for research, completed all six formative (unit) examinations, and completed a summative (comprehensive) examination. Analyses were performed with Pearson correlations and linear regression. Results: A total of 1,406 student records were included based on inclusion criteria. Correlation with each formative and the summative examination were all significant, p < 0.001: Cardiology 0.597; Airway 0.571; Medical 0.571; Trauma 0.566; Ob/Pediatrics 0.549; Operations 0.495. The cardiology exam was shown to have a moderate correlation on summative performance, whereas the operations exam had the weakest correlation. The number of formative examination failures was a significant predictor of the probability of passing the summative examination, t(1405) = –31.02, p < 0.001. Zero failed unit examinations yielded a 100% probability of passing. Three failed formative exams yielded a 60.4% probability. Four failed attempts yielded a 44.8% probability. Failure of all six formative exams yielded a 13.4% probability of passing the Paramedic Readiness Exam Version 3.  Conclusion: Not all formative examinations hold the same predictive power on the probability of passing a summative examination. Each had their own correlation value. Students who did not fail formative examinations have a 100% likelihood of passing the summative examination.  INTRODUCTION Emergency medical services (EMS) providers are expected to deliver care in the out-of-hospital environment 24-hours a day, seven days a week.1 In most systems and services, paramedics represent the highest level of care by providing advanced life support (ALS), interventions, and medications. Paramedics complete rigorous training to develop competency in psychomotor, cognitive, and affective domains.1 Training programs assess cognitive competency within the field of EMS by using formative assessments of units, such as cardiology, medical emergencies, and pediatrics, followed by a final summative assessment of Correlation of Formative Assessments as the Means of Predicting Summative Performance in Paramedic Students 2                                                                                                                                                                                                                                                              © The Internet Journal of Allied Health Sciences and Practice, 2017 the comprehensive program. After paramedic students have demonstrated competence with a passing score through their training program, they are often required to pass a national certifying or state licensing summative assessment of cognitive knowledge before obtaining their credentials to provide care and entering the practice of paramedicine.   In medical education and throughout the curriculum, the role of formative assessment is to evaluate the learner and make specific suggestions for improvement when needed.2 In contrast, a summative assessment evaluates the learner to help instructors form judgments or make decisions on the following key areas: achievement, motivation to maintain performance, certification of performance for others, and grades.2 Regardless of an assessment being formative or summative, any evaluation has an effect on the learner, curriculum, faculty, and stakeholders.2 If an evaluation has methodological rigor, then it could certainly identify progress or achievement made by the learner. However, if an evaluation does not have methodological rigor, then much consideration must be given to how it may result in false conclusions and potentially cause more harm than good because of improper interpretation.2   Paramedic programs use multiple formative assessments to determine cognitive competency. Logically, a student who does not achieve a predetermined score on one or more formative unit examinations may face academic counseling or discipline that could include dismissal. It is the program’s responsibility that students have a baseline of knowledge in all topics rather than a lot in one topic and little in another topic. This is part of minimum competencies. A formative assessment can identify areas for improvement and highlight potential learning opportunities to the student.3 The aim of this study was to assess whether a student could successfully pass a summative assessment after failing one or more formative assessments during a paramedic program.  METHODS Subjects This retrospective study reviewed prospectively collected data from the Field Internship Student Data Acquisition Project (Fisdap™), which is a national database used by paramedic programs. The Institutional Review Board at Inver Hills Community College provided approval for prospective data into Fisdap™. One feature offered by Fisdap is standardized formative unit exams and summative comprehensive multiple-choice exams. Unit and comprehensive exam scores for paramedic student accounts with Fisdap™ were reviewed based on the following inclusion criteria: the student provided consent for research, completed all six-unit exams (Airway, Cardiology, Medical, Obstetrics and Pediatrics, Operations, and Trauma), and completed the comprehensive Paramedic Readiness Exam Version 3 (PRE-3).   Statistical Analysis The data collected for study were submitted between January 2011 and February 2016. Collected data were analyzed using a linear regression model to examine the relationship between number of unit exam failures and the probability of passing the PRE-3. In this model, the dependent variable was pass-fail status of PRE-3 and the independent variables were the number of unit exam failures, range from zero to six.   RESULTS A total of 1,406 student records were included in this study based on inclusion criteria. Correlations with each formative and the summative PRE-3 examination were all significant and are reported in Table 1. It should be noted that these correlations are at best moderate for most and are weak in a few exams. The Cardiology (0.597), Airway (0.571), and Medical (0.571) unit exams were found to have the most significant correlation to PRE-3 score.   Table 1. Probability of Passing PRE-3 Unit Exam (n=338) Pearson Correlation:  Unit Exam to Summative Overall p value Cardiology 0.597 < 0.001 Airway 0.571 < 0.001 Medical 0.571 < 0.001 Trauma 0.566 < 0.001 Ob/Pediatrics 0.549 < 0.001 Operations 0.495 < 0.001    Correlation of Formative Assessments as the Means of Predicting Summative Performance in Paramedic Students 3                                                                                                                                                                                                                                                              © The Internet Journal of Allied Health Sciences and Practice, 2017 Depicted in Figure 1 is the probability of passing the summative PRE-3 exam based on the number of failed formative unit exams (p < 0.001). The results revealed a steady decline on the probability of successfully passing the summative exam with each failed unit exam (Range: 0.082 to 0.157, Mode: 0.157, Mean: 0.144, SD: ± 0.03).  Zero failed unit examinations yielded a 100% probably of passing. Three failed formative exams yielded a 60.4% probability of passing. Four failed attempts yielded a 44.8% probability. Failure of all six formative exams yielded a 13.4% probability of passing the PRE-3.     Figure 1   DISCUSSION Formative assessment is generally defined as a process undertaken by educators as an integral part of the teaching and learning process to modify and enhance learning and understanding.3 As such, the results of this “assessment” are provided as feedback to students to modify their learning activities or to reinforce learning points. This is an active process, one whereby the learner and educator trade information to improve and enhance the learning process. Thus, feedback is a key element in formative assessment and the purpose of formative assessment is therefore to “close the gap.” To close any gap, the learner must: 1) Possess a concept of the standard (e.g., goal or reference level) being aimed for, 2) Compare the actual (or current) level of performance with the standard, and 3) Engage in appropriate action that leads to some closure of the gap.3   In contrast, a summative assessment in EMS is concerned with summarizing the learning achievement status of a student and is generally intended for purposes of certification.3 Successfully passing the PRE-3 is closely correlated with successfully passing the paramedic-level National Registry of Emergency Medical Technicians Cognitive Examination (NREMTCE), which is a national summative certification exam.5 The PRE-3 has a positive predictive value (PPV) of 96% for passing the NREMTCE at the suggested Angoff cut score of 73%.5 The PPV is defined as the percentage of students who passed the PRE 3 at or above the stated cut score and then passed the NREMTCE exam on their first attempt.  The results in this study demonstrate that a student who passes all formative unit exams and has a 100% probability of successfully passing the PRE-3 summative exam, and thus has a high probability of passing the NREMTCE. The results revealed a steady decline on the probability of successfully passing the summative exam with each failed unit exam These findings support a notion that a student can fail multiple formative exams within a given limit and still be successful on a summative exam.   These findings hold practical implications for EMS education and training programs in the United States because the data indicated that not all formative exams correlate the same on the probability of successfully passing a summative examination. Of the six formative examinations and their correlation to successfully passing the PRE-3 summative examination, the cardiology unit exam held a moderate correlation at best and operations the least. This indicates that the correlation of a failed unit, or Correlation of Formative Assessments as the Means of Predicting Summative Performance in Paramedic Students 4                                                                                                                                                                                                                                                              © The Internet Journal of Allied Health Sciences and Practice, 2017 exams, to the probability of successfully passing a summative exam should be considered when deciding whether to dismiss the student or retain them and improve their identified formative deficits.  The decision to dismiss or retain a student should also include consideration of the number of leads on ALS calls that a student has or has not completed. Previous research discovered the number of ALS leads the student completed was the strongest internship predictor of passing the NREMTCE.4 These separate findings on factors affecting success on summative exams warrant additional investigation to discover if any correlations exist between both formative exam performance and the number of student ALS runs.   Limitations This study was limited to retrospective data on scores obtained from first attempt at paramedic formative and summative exams submitted to Fisdap™. Not all paramedic programs in the United States use the Fisdap™ database and testing products. An additional limitation is the sequence, use, and timing of these exams within the students training was not determined.    CONCLUSIONS The discoveries and discussion in this study has practical implications for EMS education and training programs in the United States. This study has revealed how a student could fail up to three formative exams and still have a probable success rate of approximately 60% on a PRE-3 summative exam that is a strong predictor of performance in the NREMTCE. Of the six formative exams, the cardiology formative exam had the greatest correlation on the probability of being successful on a PRE-3 summative exam and the operations formative exam had the least, although moderate to weak at best. Programs can utilize the probability of passing a summative exam, such as the PRE-3, after failing a certain number of formative exams in their policies pertaining to student success, retention, and program dismissal. Additionally, this study may encourage programs to perform their own quality initiatives related to their formative exams scores, exam statistical analysis, and performance on summative exams.  References 1. Studnek JR, Fernandez AR, Shimberg B, Garifo M, Correll M. The association between emergency medical services field performance assessed by high-fidelity simulation and the cognitive knowledge of practicing paramedics. Academic Emergency Medicine. 2011;18(11):1177-85. doi:10.111/j.1553-2712.2011.01208.x [PMID: 22092899] 2. Lipsett PA, Kern DE. Step 6: Evaluation and feedback. In Kern DE,  Thomas PA, Hughes MT (Eds.), Curriculum development for medical education: A six-step approach. 2nd Edition. Baltimore, Maryland: The John Hopkins University Press; 2006. 3. Saddler DE. Formative assessment and the design of instructional systems. Instructional Science. 1989;18:119-44. 4. Salzman J, Dillingham J, Kobersteen J, Kaye K, Page D. Effect of paramedic student internship on performance on the National Registry Written Exam. Prehospital Emergency Care. 2008;12(2):212-6. doi:10.1080/10903120801906879 [PMID: 18379919] 5. Fisdap. Paramedic Readiness Exam 3: Statistics and other facts. Unpublished. 2011.     

Correlation of formative assessments as the means of predicting summative performance in paramedic students

https://nsuworks.nova.edu/cgi/viewcontent.cgi?article=1672&amp;context=ijahsp

Correlation of formative assessments as the means of predicting summative performance in paramedic students

Abstract

Similar works

Full text

Available Versions

NSU Works