QU-BraTS: MICCAI BraTS 2020 challenge on quantifying uncertainty in brain tumor segmentation -- analysis of ranking metrics and benchmarking results

Deep learning (DL) models have provided the state-of-the-art performance in a wide variety of medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder the translation of DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties, could enable clinical review of the most uncertain regions, thereby building trust and paving the way towards clinical translation. Recently, a number of uncertainty estimation methods have been introduced for DL medical image segmentation tasks. Developing metrics to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a metric developed during the BraTS 2019-2020 task on uncertainty quantification (QU-BraTS), and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This metric (1) rewards uncertainty estimates that produce high confidence in correct assertions, and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentages of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QUBraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, and hence highlight the need for uncertainty quantification in medical image analyses. Finally, in favor of transparency and reproducibility our evaluation code is made publicly available at https://github.com/RagMeh11/QU-BraTSResearch reported in this publication was partly supported by the Informatics Technology for Cancer Research (ITCR) program of the National Cancer Institute (NCI) of the National Institutes of Health (NIH), under award numbers NIH/NCI/ITCR:U01CA242871 and NIH/NCI/ITCR:U24CA189523. It was also partly supported by the National Institute of Neurological Disorders and Stroke (NINDS) of the NIH, under award number NIH/NINDS:R01NS042645.Document signat per 92 autors/autores: Raghav Mehta1 , Angelos Filos2 , Ujjwal Baid3,4,5 , Chiharu Sako3,4 , Richard McKinley6 , Michael Rebsamen6 , Katrin D¨atwyler6,53, Raphael Meier54, Piotr Radojewski6 , Gowtham Krishnan Murugesan7 , Sahil Nalawade7 , Chandan Ganesh7 , Ben Wagner7 , Fang F. Yu7 , Baowei Fei8 , Ananth J. Madhuranthakam7,9 , Joseph A. Maldjian7,9 , Laura Daza10, Catalina Gómez10, Pablo Arbeláez10, Chengliang Dai11, Shuo Wang11, Hadrien Raynaud11, Yuanhan Mo11, Elsa Angelini12, Yike Guo11, Wenjia Bai11,13, Subhashis Banerjee14,15,16, Linmin Pei17, Murat AK17, Sarahi Rosas-González18, Illyess Zemmoura18,52, Clovis Tauber18 , Minh H. Vu19, Tufve Nyholm19, Tommy L¨ofstedt20, Laura Mora Ballestar21, Veronica Vilaplana21, Hugh McHugh22,23, Gonzalo Maso Talou24, Alan Wang22,24, Jay Patel25,26, Ken Chang25,26, Katharina Hoebel25,26, Mishka Gidwani25, Nishanth Arun25, Sharut Gupta25 , Mehak Aggarwal25, Praveer Singh25, Elizabeth R. Gerstner25, Jayashree Kalpathy-Cramer25 , Nicolas Boutry27, Alexis Huard27, Lasitha Vidyaratne28, Md Monibor Rahman28, Khan M. Iftekharuddin28, Joseph Chazalon29, Elodie Puybareau29, Guillaume Tochon29, Jun Ma30 , Mariano Cabezas31, Xavier Llado31, Arnau Oliver31, Liliana Valencia31, Sergi Valverde31 , Mehdi Amian32, Mohammadreza Soltaninejad33, Andriy Myronenko34, Ali Hatamizadeh34 , Xue Feng35, Quan Dou35, Nicholas Tustison36, Craig Meyer35,36, Nisarg A. Shah37, Sanjay Talbar38, Marc-Andr Weber39, Abhishek Mahajan48, Andras Jakab47, Roland Wiest6,46 Hassan M. Fathallah-Shaykh45, Arash Nazeri40, Mikhail Milchenko140,44, Daniel Marcus40,44 , Aikaterini Kotrotsou43, Rivka Colen43, John Freymann41,42, Justin Kirby41,42, Christos Davatzikos3,4 , Bjoern Menze49,50, Spyridon Bakas∗3,4,5 , Yarin Gal∗2 , Tal Arbel∗1,51 // 1Centre for Intelligent Machines (CIM), McGill University, Montreal, QC, Canada, 2Oxford Applied and Theoretical Machine Learning (OATML) Group, University of Oxford, Oxford, England, 3Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA, 4Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA, 5Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA, 6Support Center for Advanced Neuroimaging (SCAN), University Institute of Diagnostic and Interventional Neuroradiology, University of Bern, Inselspital, Bern University Hospital, Bern, Switzerland, 7Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX, USA, 8Department of Bioengineering, University of Texas at Dallas, Texas, USA, 9Advanced Imaging Research Center, University of Texas Southwestern Medical Center, Dallas, TX, USA, 10Universidad de los Andes, Bogotá, Colombia, 11Data Science Institute, Imperial College London, London, UK, 12NIHR Imperial BRC, ITMAT Data Science Group, Imperial College London, London, UK, 13Department of Brain Sciences, Imperial College London, London, UK, 14Machine Intelligence Unit, Indian Statistical Institute, Kolkata, India, 15Department of CSE, University of Calcutta, Kolkata, India, 16 Division of Visual Information and Interaction (Vi2), Department of Information Technology, Uppsala University, Uppsala, Sweden, 17Department of Diagnostic Radiology, The University of Pittsburgh Medical Center, Pittsburgh, PA, USA, 18UMR U1253 iBrain, Université de Tours, Inserm, Tours, France, 19Department of Radiation Sciences, Ume˚a University, Ume˚a, Sweden, 20Department of Computing Science, Ume˚a University, Ume˚a, Sweden, 21Signal Theory and Communications Department, Universitat Politècnica de Catalunya, BarcelonaTech, Barcelona, Spain, 22Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand, 23Radiology Department, Auckland City Hospital, Auckland, New Zealand, 24Auckland Bioengineering Institute, University of Auckland, New Zealand, 25Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, MA, USA, 26Massachusetts Institute of Technology, Cambridge, MA, USA, 27EPITA Research and Development Laboratory (LRDE), France, 28Vision Lab, Electrical and Computer Engineering, Old Dominion University, Norfolk, VA 23529, USA, 29EPITA Research and Development Laboratory (LRDE), Le Kremlin-Bicˆetre, France, 30School of Science, Nanjing University of Science and Technology, 31Research Institute of Computer Vision and Robotics, University of Girona, Spain, 32Department of Electrical and Computer Engineering, University of Tehran, Iran, 33School of Computer Science, University of Nottingham, UK, 34NVIDIA, Santa Clara, CA, US, 35Biomedical Engineering, University of Virginia, Charlottesville, USA, 36Radiology and Medical Imaging, University of Virginia, Charlottesville, USA, 37Department of Electrical Engineering, Indian Institute of Technology - Jodhpur, Jodhpur, India, 38SGGS ©2021 Mehta et al.. License: CC-BY 4.0. arXiv:2112.10074v1 [eess.IV] 19 Dec 2021 Mehta et al. Institute of Engineering and Technology, Nanded, India, 39Institute of Diagnostic and Interventional Radiology, Pediatric Radiology and Neuroradiology, University Medical Center, 40Department of Radiology, Washington University, St. Louis, MO, USA, 41Leidos Biomedical Research, Inc, Frederick National Laboratory for Cancer Research, Frederick, MD, USA, 42Cancer Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA, 43Department of Diagnostic Radiology, University of Texas MD Anderson Cancer Center, Houston, TX, USA, 44Neuroimaging Informatics and Analysis Center, Washington University, St. Louis, MO, USA, 45Department of Neurology, The University of Alabama at Birmingham, Birmingham, AL, USA, 46Institute for Surgical Technology and Biomechanics, University of Bern, Bern, Switzerland, 47Center for MR-Research, University Children’s Hospital Zurich, Zurich, Switzerland, 48Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, India, 49Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland, 50Department of Informatics, Technical University of Munich, Munich, Germany, 51MILA - Quebec Artificial Intelligence Institute, Montreal, QC, Canada, 52Neurosurgery department, CHRU de Tours, Tours, France, 53 Human Performance Lab, Schulthess Clinic, Zurich, Switzerland, 54 armasuisse S+T, Thun, Switzerland.Preprin

2017 Commencement for Sidney Kimmel Medical College, Jefferson Graduate School of Biomedical Sciences, and Jefferson School of Population Health

Processional Trumpet Voluntary, JOHN STANLEY The Jefferson Processional, BURLE MARX Organist, THE REVEREND R. BRUCE TODD Opening Proclamation RICHARD W. HEVNER, Chair, Board of Trustees, Thomas Jefferson University and Jefferson Health Presentation of Colors U.S. Armed Forces Career Center, Philadelphia The National Anthem Convocation and Remarks STEPHEN K. KLASKO, MD, MBA President and CEO, Thomas Jefferson University and Jefferson Health President\u27s Award PRESIDENT KLASKO (HAROLD AND LYNNE HONICKMAN) Conferring of Honorary Degrees PRESIDENT KLASKO (CAROLINE KIMMEL, Doctor of Science; SIDNEY KIMMEL, Doctor of Science; DONATO J . TRAMUTO, Doctor of Science) Conferring of Degrees in Course (President Klasko) Jefferson College of Biomedical Sciences Doctor of Philosophy Master of Science Presented by GERALD B. GRUNWALD, PH, Dean, Jefferson College of Biomedical Sciences Jefferson College of Population Health Doctor of Philosophy Master of Public Health Master of Science Presented by DAVID B. NASH, MD, MBA, Dean, Jefferson College of Population Health Sidney Kimmel Medical College Doctor of Medicine Presented by MARK L. TYKOCINSKI, MD, Provost and Executive Vice President, Thomas Jefferson University, Anthony F. and Gertrude M. DePalma Dean, Sidney Kimmel Medical College Oath of Hippocrates JOSEPH F. MAJDAN, MD, FACP, Associate Professor of Medicine Recessional Pomp and Circumstance, ELGAR (REVEREND TODD

2017 Commencement for Sidney Kimmel Medical College, Jefferson Graduate School of Biomedical Sciences, and Jefferson School of Population Health

Processional Trumpet Voluntary, JOHN STANLEY The Jefferson Processional, BURLE MARX Organist, THE REVEREND R. BRUCE TODD Opening Proclamation RICHARD W. HEVNER, Chair, Board of Trustees, Thomas Jefferson University and Jefferson Health Presentation of Colors U.S. Armed Forces Career Center, Philadelphia The National Anthem Convocation and Remarks STEPHEN K. KLASKO, MD, MBA President and CEO, Thomas Jefferson University and Jefferson Health President\u27s Award PRESIDENT KLASKO (HAROLD AND LYNNE HONICKMAN) Conferring of Honorary Degrees PRESIDENT KLASKO (CAROLINE KIMMEL, Doctor of Science; SIDNEY KIMMEL, Doctor of Science; DONATO J . TRAMUTO, Doctor of Science) Conferring of Degrees in Course (President Klasko) Jefferson College of Biomedical Sciences Doctor of Philosophy Master of Science Presented by GERALD B. GRUNWALD, PH, Dean, Jefferson College of Biomedical Sciences Jefferson College of Population Health Doctor of Philosophy Master of Public Health Master of Science Presented by DAVID B. NASH, MD, MBA, Dean, Jefferson College of Population Health Sidney Kimmel Medical College Doctor of Medicine Presented by MARK L. TYKOCINSKI, MD, Provost and Executive Vice President, Thomas Jefferson University, Anthony F. and Gertrude M. DePalma Dean, Sidney Kimmel Medical College Oath of Hippocrates JOSEPH F. MAJDAN, MD, FACP, Associate Professor of Medicine Recessional Pomp and Circumstance, ELGAR (REVEREND TODD

2017 Commencement for Sidney Kimmel Medical College, Jefferson Graduate School of Biomedical Sciences, and Jefferson School of Population Health

Processional Trumpet Voluntary, JOHN STANLEY The Jefferson Processional, BURLE MARX Organist, THE REVEREND R. BRUCE TODD Opening Proclamation RICHARD W. HEVNER, Chair, Board of Trustees, Thomas Jefferson University and Jefferson Health Presentation of Colors U.S. Armed Forces Career Center, Philadelphia The National Anthem Convocation and Remarks STEPHEN K. KLASKO, MD, MBA President and CEO, Thomas Jefferson University and Jefferson Health President\u27s Award PRESIDENT KLASKO (HAROLD AND LYNNE HONICKMAN) Conferring of Honorary Degrees PRESIDENT KLASKO (CAROLINE KIMMEL, Doctor of Science; SIDNEY KIMMEL, Doctor of Science; DONATO J . TRAMUTO, Doctor of Science) Conferring of Degrees in Course (President Klasko) Jefferson College of Biomedical Sciences Doctor of Philosophy Master of Science Presented by GERALD B. GRUNWALD, PH, Dean, Jefferson College of Biomedical Sciences Jefferson College of Population Health Doctor of Philosophy Master of Public Health Master of Science Presented by DAVID B. NASH, MD, MBA, Dean, Jefferson College of Population Health Sidney Kimmel Medical College Doctor of Medicine Presented by MARK L. TYKOCINSKI, MD, Provost and Executive Vice President, Thomas Jefferson University, Anthony F. and Gertrude M. DePalma Dean, Sidney Kimmel Medical College Oath of Hippocrates JOSEPH F. MAJDAN, MD, FACP, Associate Professor of Medicine Recessional Pomp and Circumstance, ELGAR (REVEREND TODD

Retraction notice to “Prediction of cardiovascular events in statin-treated patients by lipid and non-lipid biomarkers” [J Am Coll Cardiol 2011;57:63–69]

Benoit J. Arsenault, PhD,* Philip Barter, MD, PhD,† David A. DeMicco, PharmD,‡ Weihang Bao, PhD,‡ Gregory M. Preston, PhD,‡ John C. LaRosa, MD,§ Scott M. Grundy, MD, PhD,‖ Prakash Deedwania, MD, PhD,¶ Heiner Greten, MD,# Nanette K. Wenger, MD,** James Shepherd, MD,†† David D. Waters, MD,‡‡ John J. P. Kastelein, MD, PhD,* for the TNT Study InvestigatorsFrom the *Department of Vascular Medicine, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands; †The Heart Research Institute, Sydney, Australia; ‡Pfizer, Inc., New York, New York; §State University of New York Health Science Center, Brooklyn, New York; ‖University of Texas Southwestern Medical Center, Dallas, Texas; ¶Veterans Affairs Central California Healthcare System and University of California San Francisco School of Medicine, San Francisco, California; #Universitätsklinikum, Eppendorf, Germany; **Emory University School of Medicine, Atlanta, Georgia; ††University of Glasgow, Glasgow, United Kingdom; and the ‡‡San Francisco General Hospital, San Francisco, CaliforniaThis article has been retracted: please see Elsevier Policy on Article Withdrawal (http://www.elsevier.com/locate/withdrawalpolicy).This article has been retracted at the request of the author.The main findings of the TNT Trial were published in 2005 (1). Since that time, members of the Steering Committee and other investigators have published 32 papers based upon additional analyses of TNT. The data for these papers were derived from analyses of the TNT clinical database, managed by Pfizer. The clinical database has been crosschecked many times and the data in it is valid. During the trial, blood samples were drawn from the patients at regular intervals for subsequent analysis. We performed a nested case-control study that included 507 patients who experienced a CV event and 1,020 control patients in the main biomarker analysis, and 496 patients who experienced a CV event and 1,117 control patients in the PCSK9 analysis. The biomarker database was separate from the clinical database. An anonymization code was run in 2007 to link patients from one database to the other.In late 2012, the TNT frozen blood samples were integrated into a large automated biobank that includes samples from other Pfizer trials. At that time discrepancies were noted among the samples, indicating that an error had occurred when the samples were anonymized in 2007. Further investigation revealed that the code created to manually anonymize the data was accidentally run twice. During the first run, anonymized subject identifiers were successfully assigned to both biosamples and clinical data. However, after this first run had passed quality control checks, the anonymization code was re-run inadvertently, replacing the first correct set of identifiers with a random and incorrect set. We do not understand how or why the code was re-run. The study team, who were blinded as to patient identity, thus reported on mismatched clinical and biomarker data. The investigators of the biomarkers study were puzzled that none of the 18 biomarkers were predictive of cardiovascular events. However we were reassured because on-treatment LDL-cholesterol, HDL-cholesterol and triglyceride levels were all strongly predictive of events, and we reported this in the paper. These lipid levels were part of the clinical database, and thus were not subject to the error that occurred with the biomarkers. In the PCSK9 analysis, PCSK9 levels were predictive of events in the atorvastatin 10-mg group (p = 0.039) but not in the 80-mg group. This finding, which we now realize is totally spurious, was not unexpected and raised no red flags. Similarly, the failure of vitamin D levels to predict events, as reported in the AHJ paper, was not surprising.Since the error was discovered, we have created a new anonymized clinical and biomarker database by restoring the original set of anonymized identifiers. We are currently reanalyzing the data according to our original study plans. However, the nested case-control feature of the original study design has been lost because the patient selection for biomarker sampling was random. Only approximately one tenth of patients now had an event, compared to one third in the original study design. Thus, the power to detect a difference in the level of a biomarker between patients with and without events has been attenuated.All authors of these manuscripts are anguished to have made this mistake and publishing incorrect information

Correction - Summer 2022 Issue: Changing Lives Through Research

An article last Summer incorrectly stated that the Jefferson Molecular Profiling of Pancreatic Cancer (JMP PaC) Program was developed by a handful of researchers in the Department of Surgery. The program was spearheaded by Hien Dang, PhD, Director of the Division of Surgical Research, in collaboration with surgeons Charles J. Yeo, MD, Wilbur Bowne, MD, and Harish Lavu, MD. Senior leadership also includes representatives from non-surgical specialties: Babar Bashir, MD, and Jennifer M. Johnson, MD, PhD (Medical Oncology); Wei Jiang, MD, PhD (Pathology); and Christopher McNair, PhD, Associate Director for Data Science at Sidney Kimmel Cancer Center

2022 Summer Experience Program Abstracts

The mission of the MD Anderson Summer Experience Program is to eliminate cancer, through training of high School, undergraduates and first year medical students to make a lasting impact on training the next generation of scientist and physicians through research and education. The MD Anderson Summer Experience is an umbrella program that comprises 18 summer academic programs at MD Anderson. The summer research program is an 8-10 week program that offers hands-on experience in biomedical, translational or clinical research. In this program, students are matched with a mentor from MD Anderson’s research or clinical faculty. Participants work alongside the mentor in a lab or clinic, on projects designed by faculty to reflect current research. Workshops and lectures provide opportunities to connect with faculty, residents, postdoctoral and clinical fellows, and other participants. Through the program, students assess goals related to careers in oncology research and patient care. The program culminates with a symposium in which participants present talks and posters on their research projects to peers and faculty.https://openworks.mdanderson.org/sumexp22/1061/thumbnail.jp

Interview with Katie McGready

An oral history interview with Mary Catherine Bussey Boice (Katie) McGready, the first medical librarian in the Texas Medical Center, The University of Texas MD Anderson Cancer Center, Houston, Texas. Mary Catherine Bussey Boice “Katie” McGready became the first medical librarian in the Texas Medical Center when Dr. Ernst W. Bertner asked to her to begin the medical library at what was then the fledgling MD Anderson Cancer Center. She was among the first employees at the Cancer Center, then located at “The Oaks,” the estate of the late Colonel James A. Baker at 2310 Baldwin Street in Houston. Born in rural Timpson, Texas, she was the only daughter with four brothers – two older and one younger. Her journey from running the soda fountain in her father’s drugstore to several jobs in the nascent Texas Medical Center reflects not only the early days there but also the lives of young women in Texas in the early 20th century. She married twice – first to Dr. Edward Henry “Ned” Boice, whom she met when she was learning medical librarian duties at The University of Texas Medical Branch in Galveston. They had five children – Betsy McPhaden of Seattle, Bill Boice of Atlanta, Peggy Boice of Houston, Cathy Bacon of Houston and Jim Boice of Austin. Later, as a widow, she married Frances Cornelius “Mac” McGready, which whom she lived many happy years before his death in 2009