This Guide to Statistics and Methods gives an overview of artificial intelligence techniques and tools in surgical education research

Hashimoto, Daniel A.

Schwartz, Todd A.

Varas, Julian

English

Carolina Digital Repository

growing interest in applying AImethods to surgical education research. While the potential for AIto be used in surgical education is high, current infrastructure andpractices on data capture, storage, labeling, and analysis leave muchto be desired to allow rigorous research using AI methods.1 This prac-tical guide provides an overview of important considerations in usingAI techniques and tools in surgical education research and offers aframework to ensure that appropriately designed studies meet ex-pectations for methodologic rigor, particularly regarding use ofhigh-quality data that reflect the phenomenon of interest (Box).Using the MethodologyThe wide-ranging potential of AI means that it can be used in manydifferent types of surgical education research, from analysis ofquantitative data such as test scores, qualitative data such as writ-ten feedback, or visual data such as operative video.2 The broadrange of AI applications requires multidisciplinary collaborationwith statisticians and computer or data scientists to ensure rigor-ous investigation and appropriate interpretation of results. As withany area of research, it is critical first to outline a clear question forinvestigation before considering which methods are appropriateto address it, including whether to use AI techniques. Artificialintelligence can be advantageous with large data sets with com-plex relationships or with data that do not lend themselves asnaturally to conventional statistical analyses (ie, images, video,and natural language). As much of the current attention to AI forsurgical education has been on ML, we focus our guide on MLmethods.While ML approaches encompass techniques that are consid-ered traditional statistics, such as linear regression, they also offermethods to analyze complex relationships within data. Ensemblelearning methods leverage the outputs of multiple models to im-prove performance, while deep learning offers approaches to non-tabular data such as images and text. Other common methods in-clude tree-based approaches, K-nearest neighbors, logistic and pe-nalized regression, and K-means clustering. Improvements inhardware and access to large data sets have begun to unlock the po-tential to analyze unique combinations of data (eg, combining videoanalysis with clinical outcomes to assess learning curves).Most ML problems are either classification or regression prob-lems, and the choice of methods should be driven by the researchquestion. Depending on the problem and type of data, supervised,unsupervised, or reinforcement learning approaches can be used;however, many current investigations adopt supervised or semisu-pervised learning.3 A researcher investigating whether videos oflaparoscopic performance can predict whether a resident passes theFundamentals of Laparoscopic Surgery examination may use su-pervised learning for classification of performance into passor fail. Natural language processing of personal statements fromresidency applicants could predict the likelihood of matchinginto a program. Alternatively, ML tools can be used to facilitatedecision-making4 and actions (such as case logging5) by trainees, orstudies can compare groups who do or do not use such tools.While ML is computationally intensive and time consuming, ac-cess to powerful computing resources, including graphical process-ing units, can accelerate these analyses. Cloud computing, while ex-pensive at an hourly rate, reduces the cost of capital hardwareacquisition. Although commercial statistical programs support someML functionality, ML methods are currently often performed usingR (R Project for Statistical Computing) or Python (Python SoftwareFoundation) software with open-source libraries such as PyTorch,Keras, and TensorFlow facilitating deep learning techniques. Auto-mated ML tools are offered by cloud vendors, allowing researchersto use pretrained models in a no-code environment. However, weadvise that these tools are best used for simple problems with well-designed and maintained data sets, as they do not yet offer theability to fine-tune all elements of a model and are limited topublicly available models.Artificial intelligence (AI) is the study of machine intelligence as it relates to perceiving and inferring data, typically with the goal of ap-proximating human performance on tasks. Modern growth in com-puting power and access to data has raised interest in AI, leading to investigation of its applications to surgery. Machine learning (ML),a subfield of AI, refers to thestudy of methods and algo- rithms that use data to improve  task performance. With the growth of AI in different appli-cations of medicine, there isBox. Summary1. Collaborate with a machine learning practitioner from start tofinish. Identifying expertise for a particular application can behelpful to align goals and complementary knowledge betweenan education researcher and a computational scientist.2. Clearly define your research question.3. Determine appropriateness of using machine learning andwhether you have a sufficient sample size for a machinelearning approach.4. Fully explore your data to ensure it adequately representsyour phenomenon of interest.5. Appropriately interpret your results within the context of yourdata. Assess for sources of bias within your data that couldimpact generalizability. Understand the limitations ofyour machine learning approach. Assess the validity ofthe output relative to its intended task.Practical Guide to Machine Learning and Artificial Intelligence in Surgical Education ResearchDaniel A. Hashimoto, MD; Julian Varas, MD; Todd A. Schwartz, DrPHIntroductionrithms, further stressing the importance of partnering with statis-tical and computational experts when planning and conductingthe research. The range and type of algorithms available are evolv-ing rapidly. However, the core principles of analytic approachesremain static. The research question should drive the analysis andselection of an appropriately sized data set. A small data set can re-sult in overfitting or underperformance, while an overly large onecan consume excessive resources.When training ML models, it is important that data from the test-ing or validation sets do not contaminate the training data. For ex-ample, in computer vision studies, it is common to extract hun-dreds of thousands of frames from a data set of a few hundred videos.If data are split at the frame level, frames from 1 video may end upin both the training and the testing data sets, providing the modelwith a biased advantage in evaluating the testing set. In such a case,data should be split into training and testing sets at the video level.Finally, evaluation metrics for algorithms are a key consider-ation. While many investigators are familiar with terms such as ac-curacy and area under the receiver operating characteristic curve,these should not be viewed as one-size-fits-all metrics. Metric se-lection should be driven by the research question, the intendedapplication of the result, and the nature of the data.6Where to Find More InformationJAMA has an overview of ML methodology in the clinical literaturethat is applicable to surgical education.3 Textbooks offer a broad in-troduction to concepts to AI and ML for clinicians.7 Informatics andengineering conferences such as those by the American Medical In-formatics Association and the Medical Image Computing and Com-puter Assisted Intervention Society offer insights into state-of-the-art models that are being developed for application in medicine.ARTICLE INFORMATIONAuthor Affiliations: Department of Surgery,University of Pennsylvania Perelman School ofMedicine, Philadelphia (Hashimoto); Department ofComputer and Information Science, University ofPennsylvania School of Engineering and AppliedScience, Philadelphia (Hashimoto); Centro deCirugía Experimental y Simulación, PontificaUniversidad Católica, Santiago, Chile (Varas);Department of Biostatistics, Gillings School ofGlobal Public Health, University of North Carolina atChapel Hill (Schwartz); Statistical Editor, JAMASurgery (Schwartz).Corresponding Author: Daniel A. Hashimoto, MD,University of Pennsylvania, 3400 Spruce St,4 Silverstein, Philadelphia, PA 19104(daniel.hashimoto@pennmedicine.upenn.edu).Published Online: January 3, 2024.doi:10.1001/jamasurg.2023.6687Conflict of Interest Disclosures: Dr Hashimotoreported receiving grants from OlympusCorporation to Massachusetts General Hospital;personal fees from Johnson & Johnson, ActivSurgical, and Verily Life Science outside thesubmitted work; having patents pending forConcept Graph Neural Networks for Surgery,Surgical Phase Recognition with SufficientStatistical Model, and Surgical Decision SupportUsing a Decision Theoretic Model, all owned byMassachusetts General Hospital. Dr Varas reportedbeing CEO and founder of Training Competence,official spinoff from the Catholic University of Chile,outside the submitted work. No other disclosureswere reported.REFERENCES1. Maier-Hein L, Eisenmann M, Sarikaya D, et al.Surgical data science—from concepts towardclinical translation. Med Image Anal. 2022;76(102306):102306. doi:10.1016/j.media.2021.1023062. Ward TM, Mascagni P, Madani A, Padoy N,Perretta S, Hashimoto DA. Surgical data science andartificial intelligence for surgical education. J SurgOncol. 2021;124(2):221-230. doi:10.1002/jso.264963. Liu Y, Chen PC, Krause J, Peng L. How to readarticles that use machine learning: users’ guides tothe medical literature. JAMA. 2019;322(18):1806-1816. doi:10.1001/jama.2019.164894. Mascagni P, Alapatt D, Sestini L, et al.Computer vision in surgery: from potential toclinical value. NPJ Digit Med. 2022;5(1):163.doi:10.1038/s41746-022-00707-55. Thanawala R, Jesneck J, Shelton J, Rhee R,Seymour NE. Overcoming systems factors in caselogging with artificial intelligence tools. J Surg Educ.2022;79(4):1024-1030. doi:10.1016/j.jsurg.2022.01.0136. Maier-Hein L, Reinke A, Godau P, et al. Metricsreloaded: Pitfalls and recommendations for imageanalysis validation. arXiv. Preprint published onlineJune 3, 2022. https://arxiv.org/abs/2206.016537. Hashimoto DA, Rosman G, Meireles OR. ArtificialIntelligence in Surgery: Understanding the Role of AIin Surgical Practice. McGraw-Hill Education; 2021.It is critical to understand that ML methods have limitations. It is important to fully explore and understand the data to ensure they appropriately represent the phenomenon of interest. Sources of bias can permeate any data and lead to biased results. It is crucial to use high-quality annotations or labels for data, particularly in su-pervised learning and especially if multiple annotators have la-beled the data. This is analogous to requiring rater training to achieve high interrater reliability in assessment studies. Regarding algo-rithm selection, many of these methods are referred to as black box approaches that automatically select data features that are most con-tributory to high performance output without providing easily in-terpretable results. Thus, while tools such as saliency and attention maps generated by algorithms may help with interpretation, the lack of true transparency and interpretability may negatively impact the trustworthiness of these methods. Investigators may further intro-duce their own bias during results interpretation to evaluate algo-rithmic performance.Statistical ConsiderationsThe reporting of AI research should follow the same standards of rigor as other research approaches. There are AI-specific versions of re-porting guidelines under development to help guide reporting, including the Standards for Reporting of Diagnostic Accuracy (STARD)–AI (diagnostic models), Transparent Reporting of a Multi-variable Prediction Model of Individual Prognosis or Diagnosis (TRIPOD)–AI (prediction models), Standard Protocol Items: Recom-mendations for Interventional Trials (SPIRIT)–AI (interventional trials), and Consolidated Standards of Reporting Trials (CONSORT)–AI (clinical trials) guidelines.The selection of a particular method should depend on the ques-tion and availability of data. There are no one-size-fits-all algo-

Practical Guide to Machine Learning and Artificial Intelligence in Surgical Education Research

http://cdr.lib.unc.edu/downloads/pg15bv50j

Practical Guide to Machine Learning and Artificial Intelligence in Surgical Education Research

Abstract

Similar works

Full text

Available Versions

Carolina Digital Repository