Analysis of Orthopaedic In-Training Examination Trauma Questions: 2017 to 2021

INTRODUCTION: The Orthopaedic In-Training Examination (OITE) is a multiple-choice examination developed by the American Academy of Orthopaedic Surgeons annually since 1963 to assess orthopaedic residents\u27 knowledge. This study\u27s purpose is to analyze the 2017 to 2021 OITE trauma questions to aid orthopaedic residents preparing for the examination. METHODS: The 2017 to 2021 OITEs on American Academy of Orthopaedic Surgeons\u27 ResStudy were retrospectively reviewed to identify trauma questions. Question topic, references, and images were analyzed. Two independent reviewers classified each question by taxonomy. RESULTS: Trauma represented 16.6% (204/1,229) of OITE questions. Forty-nine percent of trauma questions included images (100/204), 87.0% (87/100) of which contained radiographs. Each question averaged 2.4 references, of which 94.9% were peer-reviewed articles and 46.8% were published within 5 years of the respective OITE. The most common taxonomic classification was T1 (46.1%), followed by T3 (37.7%) and T2 (16.2%). DISCUSSION: Trauma represents a notable portion of the OITE. Prior OITE trauma analyses were published greater than 10 years ago. Since then, there has been an increase in questions with images and requiring higher cognitive processing. The Journal of Orthopaedic Trauma (24.7%), Journal of the American Academy of Orthopaedic Surgeons (10.1%), and Journal of Bone and Joint Surgery, American Volume (9.3%) remain the most cited sources

An Assessment of the Performance of Different Chatbots on Shoulder and Elbow Questions

Background/Objectives: The utility of artificial intelligence (AI) in medical education has recently garnered significant interest, with several studies exploring its applications across various educational domains; however, its role in orthopedic education, particularly in shoulder and elbow surgery, remains scarcely studied. This study aims to evaluate the performance of multiple AI models in answering shoulder- and elbow-related questions from the AAOS ResStudy question bank. Methods: A total of 50 shoulder- and elbow-related questions from the AAOS ResStudy question bank were selected for the study. Questions were categorized according to anatomical location, topic, concept, and difficulty. Each question, along with the possible multiple-choice answers, was provided to each chatbot. The performance of each chatbot was recorded and analyzed to identify significant differences between the chatbots\u27 performances across various categories. Results: The overall average performance of all chatbots was 60.4%. There were significant differences in the performances of different chatbots (p = 0.034): GPT-4o performed best, answering 74% of the questions correctly. AAOS members outperformed all chatbots, with an average accuracy of 79.4%. There were no significant differences in performance between shoulder and elbow questions (p = 0.931). Topic-wise, chatbots did worse on questions relating to Adhesive Capsulitis than those relating to Instability (p = 0.013), Nerve Injuries (p = 0.002), and Arthroplasty (p = 0.028). Concept-wise, the best performance was seen in Diagnosis (71.4%), but there were no significant differences in scores between different chatbots. Difficulty analysis revealed that chatbots performed significantly better on easy questions (68.5%) compared to moderate (45.4%; p = 0.04) and hard questions (40.0%; p = 0.012). Conclusions: AI chatbots show promise as supplementary tools in medical education and clinical decision-making, but their limitations necessitate cautious and complementary use alongside expert human judgment

The Number of Shoulder and Elbow Questions on the Orthopedic In-Training Examination Is Increasing with Greater Emphasis on Critical Thinking over Recall

Background: It is critical for orthopedic surgery residents and residency programs to have a current understanding of the content and resources utilized by the Orthopedic In-Training Examination (OITE) to continuously guide study and educational efforts. This study presents an updated analysis of the shoulder and elbow section of the OITE. Methods: All OITE questions, answers, and references from 2013 to 2019 were reviewed. The number of shoulder and elbow questions per year was recorded, and questions were analyzed for topic, imaging modalities, cognitive taxonomy, and references. We compared our data to the results of a previous study that analyzed shoulder and elbow OITE questions from 2002 to 2007 to examine trends and changes in this domain overtime. Results: There were 177 shoulder and elbow questions (126 shoulder, 71.2%; 51 elbow, 28.8%) of 1863 OITE questions (9.5%) over a 7-year period. The most commonly tested topics included degenerative joint disease/stiffness/arthroplasty (31.6%), anatomy/biomechanics (16.9%), instability/athletic injury (15.3%), trauma (14.7%), and rotator cuff (13.6%). Half of all questions involved clinical management decisions (49.7%). A total of 417 references were cited from 56 different sources, the most common of which were the Journal of Shoulder and Elbow Surgery (23.3%), Journal of the American Academy of Orthopaedic Surgeons (20.4%), and Journal of Bone and Joint Surgery (American Volume) (16%). The average time lag from article publication to OITE reference was 7.7 years. Compared with a prior analysis from 2002 to 2007, there was a significant increase in the number of shoulder and elbow questions on the OITE (5.5% to 9.5%; P \u3c.001). Recent exams incorporated more complex multistep treatment questions (4.4% vs. 49.7%; P \u3c.001) and fewer recall questions (42.2% vs. 22%; P \u3c.001). There was a significant increase in the use of imaging modalities (53.3% vs. 79.1%; P \u3c.001). No significant differences in the distribution of question topics were found. Conclusions: The percentage of shoulder and elbow questions on the OITE has nearly doubled over the past decade with greater emphasis on critical thinking (eg, clinical management decisions) over recall of facts. These findings should prompt educators to direct didactic efforts (eg, morning conferences and journal club) toward case-based learning to foster critical thinking and clinical reasoning skills

Comparison of Sports Medicine Questions on the Orthopaedic In-Training Examination Between 2009 and 2012 and 2017 and 2020 Reveals an Increasing Number of References

Purpose To provide an updated analysis of the sports medicine section of the Orthopedic In-Training Examination (OITE). Methods A cross-sectional review of OITE sports medicine questions from 2009 to 2012 and 2017-2020 was performed. Subtopics, taxonomy, references, and use of imaging modalities were recorded and changes between the time periods were analyzed. Results The most tested sports medicine subtopics included ACL (12.6%), rotator cuff (10.5%), and throwing injuries to the shoulder (7.4%) in the early subset, while ACL (10%), rotator cuff (6.25%), shoulder instability (6.25%), and throwing injuries to the elbow (6.25%) were the most common in the later subset. The American Journal of Sports Medicine (28.3%) was the most cited journal referenced from 2009 to 2012, while The Journal of the American Academy of Orthopaedic Surgeons (17.5%) was most referenced in questions from 2017 to 2020. The number of references per question increased from the early to the late subset (P \u3c .001). There was a trend toward an increased taxonomy type one questions (P = .114), while type 2 questions had a decreased trend (P = .263) when comparing the new subset to the early group. Conclusion When comparing sports medicine OITE questions from 2009 to 2012 and 2017 to 2020, there was an increase in the number of references per question. Subtopics, taxonomy, lag time, and use of imaging modalities did not show statistically significant changes. Clinical Relevance This study provides a detailed analysis of the sports medicine section of the OITE, which can be used by residents and program directors to direct their preparation for the annual examination. The results of this study may help examining boards align their examinations and provide a benchmark for future studies

Analysis of OITE by Subsection from 2017 - 2022 with Comparison to ABOS Part 1 Certifying Examination

Introduction: In light of recent efforts of the American Academy of Orthopaedic Surgeons (AAOS) and the American Board of Orthopaedic Surgeons (ABOS) to align the Orthopaedic In-Training Examination (OITE) with the Part I Certifying Exam, this study assessed each section of recent OITEs and compared content between the OITE exams and the ABOS part 1 exam outline. Methods: A cross-sectional review was performed using questions from 2017 – 2022 OITE exams. Question domains, subtopics, journal references, publication lag time, imaging modalities, and question taxonomy were reviewed for each exam section. The AAOS ResStudy online module was used to extract question data from recent years. The breakdown of topics of OITE questions was compared to the published blueprint for the ABOS Part 1 certifying examination. Results: Over 1500 questions were reviewed and analyzed. The top content domains on the OITE from 2017-2022 were trauma (13.6%), basic science (12.2%), and pediatrics (11.8%). The most cited in references from OITE questions were: Journal of American Academy of Orthopaedic Surgery (21.7%), Journal of Bone and Joint Surgery (19.4%), and Journal of Orthopaedic Trauma (10.3%). The breakdown of topics of OITE questions was compared to the published blueprint for the ABOS Part 1 certifying examination. There were four content domains (spine, basic science, oncology, foot and ankle) that fell in line with the proportions suggested on the ABOS Part 1 blueprint, while the other six fell outside of the range of proportions. Discussion/Conclusion: In a comprehensive review of recent OITE examinations, the most common content categories were trauma, basic science, and pediatrics. Four of the ten content categories constituted a proportion of OITE questions that fell within the range provided in the ABOS Part 1 blueprint