Composite Scores and Decision-Making in Undergraduate Medical Education

Composite scores are frequently used in medical education to reflect aggregate information about a student’s performance. Combining assessment scores into composites has been shown to be a successful practice in traditional medical education models and is normally driven by the educational system in place. The recent paradigm shift to competency-based medical education has been associated with many implications for assessment. A major challenge emerged about the procedure of combining assessment information in competency-based models, and the validity of decision-making based on composites. In this study, we examined validity evidence associated with traditional composite scores and consequential decision-making, and that associated with reformulated composites based on the competency framework. Furthermore, a third decision model about students’ academic progress was built from deliberations among education experts. All assessment data about third-year medical students were collected, in addition to scores on International exams and information about residency placement. Our results showed that the reliability of composite scores is adequate for the scope of their use, irrespective of the medical education system that drove their formulation. However, associations were more meaningful and interpretable in the decision model based on the competency framework, in comparison to the traditional model. The three models yielded an absolute agreement in 67.4% of cases, and a re-classification of students’ academic status in the rest. Correlations with external criteria (performance on International exams and residency placement) demonstrated that decisions ensuing from the three models are supported by consequential validity evidence, and that the second model, using competency-guided composite scores, provided a better classification accuracy, especially in the borderline spectrum of performance. Finally, our findings suggest that the use of composite scores is associated with defensible decisions about student advancement irrespective of the medical education model. However, decision models differ with their ability to address the challenge of identifying struggling students. Although the advancement of competency-based medical education had implications over assessment, formulating composite scores as measures of competencies is feasible and seems to yield better classification decisions

Critical pathway for deceased organ donors, Lebanon 2017–2019 (NOD-Lb).

Critical pathway for deceased organ donors, Lebanon 2017–2019 (NOD-Lb).</p

DAS criteria and alerts received by NOD-Lb for donors and non-donors.

DAS criteria and alerts received by NOD-Lb for donors and non-donors.</p

Characteristics of donors and non-donors.

BackgroundOrgan donation shortage and in particular organ procurement is an international concern as the gap between the number of donors and recipients is steadily growing. Organ procurement is a chain of steps with donor identification and referral (ID&R) as the very first link in this chain. Failure of this step hinders the progress in the organ transplantation program.ObjectivesOur study was conducted to evaluate and highlight the gap between the national system and the practice at the identification and referral (ID&R) step of the organ procurement chain in a single tertiary-care academic health center in Beirut: the Lebanese American University Medical Center–Rizk Hospital (LAUMC-RH), and to appraise the literature for challenges at this step and for possible interventions for improvement based on the international experience.Materials and methodsThis retrospective study was a descriptive case series of ICU and ED deceased patients at a single tertiary-care university hospital in Beirut. Patients’ characteristics were collected from medical records for all patients who died between 2017 and 2019 while in the ICU or the ED and shared with the National Organization for Organ and Tissue Donation and Transplantation (NOD-Lb), for each subject separately, to decide on the donor status. All data collected from the patient cohort was analyzed using R version 3.6.1. Wilcoxon signed-rank test, chi-squared, and fisher-exact tests were used to compare differences in clinical characteristics in terms of donor status when appropriate.ResultsThis study served as 3 years audit of a single hospital experience, and it demonstrates failure to make any referrals to NOD-Lb and zero actual organ and tissue donations over the study period. The review of 295 deceased subjects’ charts demonstrates 295 missed alerts to NOD-Lb and the overall missing of 5 organ and tissue donors and 24 cornea donors assuming the organ procurement chain of steps will continue uninterrupted after ID&R.ConclusionThe data gathered suggests the presence of an inefficient identification and referral system that is translated into a complete failure of reporting to NOD-Lb from LAUMC-RH. A systematic evidence-based approach to evaluate for the most cost-effective intervention to increase identification and referral rates is needed with a serious effort to examine and account for any inefficient implantation.</div

Algorithm of brain death diagnosis in adult patients.

Algorithm of brain death diagnosis in adult patients.</p

Frequency distribution of donor status.

BackgroundOrgan donation shortage and in particular organ procurement is an international concern as the gap between the number of donors and recipients is steadily growing. Organ procurement is a chain of steps with donor identification and referral (ID&R) as the very first link in this chain. Failure of this step hinders the progress in the organ transplantation program.ObjectivesOur study was conducted to evaluate and highlight the gap between the national system and the practice at the identification and referral (ID&R) step of the organ procurement chain in a single tertiary-care academic health center in Beirut: the Lebanese American University Medical Center–Rizk Hospital (LAUMC-RH), and to appraise the literature for challenges at this step and for possible interventions for improvement based on the international experience.Materials and methodsThis retrospective study was a descriptive case series of ICU and ED deceased patients at a single tertiary-care university hospital in Beirut. Patients’ characteristics were collected from medical records for all patients who died between 2017 and 2019 while in the ICU or the ED and shared with the National Organization for Organ and Tissue Donation and Transplantation (NOD-Lb), for each subject separately, to decide on the donor status. All data collected from the patient cohort was analyzed using R version 3.6.1. Wilcoxon signed-rank test, chi-squared, and fisher-exact tests were used to compare differences in clinical characteristics in terms of donor status when appropriate.ResultsThis study served as 3 years audit of a single hospital experience, and it demonstrates failure to make any referrals to NOD-Lb and zero actual organ and tissue donations over the study period. The review of 295 deceased subjects’ charts demonstrates 295 missed alerts to NOD-Lb and the overall missing of 5 organ and tissue donors and 24 cornea donors assuming the organ procurement chain of steps will continue uninterrupted after ID&R.ConclusionThe data gathered suggests the presence of an inefficient identification and referral system that is translated into a complete failure of reporting to NOD-Lb from LAUMC-RH. A systematic evidence-based approach to evaluate for the most cost-effective intervention to increase identification and referral rates is needed with a serious effort to examine and account for any inefficient implantation.</div

Donor status by year.

BackgroundOrgan donation shortage and in particular organ procurement is an international concern as the gap between the number of donors and recipients is steadily growing. Organ procurement is a chain of steps with donor identification and referral (ID&R) as the very first link in this chain. Failure of this step hinders the progress in the organ transplantation program.ObjectivesOur study was conducted to evaluate and highlight the gap between the national system and the practice at the identification and referral (ID&R) step of the organ procurement chain in a single tertiary-care academic health center in Beirut: the Lebanese American University Medical Center–Rizk Hospital (LAUMC-RH), and to appraise the literature for challenges at this step and for possible interventions for improvement based on the international experience.Materials and methodsThis retrospective study was a descriptive case series of ICU and ED deceased patients at a single tertiary-care university hospital in Beirut. Patients’ characteristics were collected from medical records for all patients who died between 2017 and 2019 while in the ICU or the ED and shared with the National Organization for Organ and Tissue Donation and Transplantation (NOD-Lb), for each subject separately, to decide on the donor status. All data collected from the patient cohort was analyzed using R version 3.6.1. Wilcoxon signed-rank test, chi-squared, and fisher-exact tests were used to compare differences in clinical characteristics in terms of donor status when appropriate.ResultsThis study served as 3 years audit of a single hospital experience, and it demonstrates failure to make any referrals to NOD-Lb and zero actual organ and tissue donations over the study period. The review of 295 deceased subjects’ charts demonstrates 295 missed alerts to NOD-Lb and the overall missing of 5 organ and tissue donors and 24 cornea donors assuming the organ procurement chain of steps will continue uninterrupted after ID&R.ConclusionThe data gathered suggests the presence of an inefficient identification and referral system that is translated into a complete failure of reporting to NOD-Lb from LAUMC-RH. A systematic evidence-based approach to evaluate for the most cost-effective intervention to increase identification and referral rates is needed with a serious effort to examine and account for any inefficient implantation.</div