Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation

Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult when using deep learning-based models, which are the state-of-the-art in many imaging applications. The current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. Scalable solutions often rely on classical techniques, such as dropout, during inference or training ensembles of identical models with different random seeds to obtain a posterior distribution. In this paper, we show that these approaches fail to approximate the classification probability. On the contrary, we propose a scalable and intuitive framework to calibrate ensembles of deep learning models to produce uncertainty quantification measurements that approximate the classification probability. On unseen test data, we demonstrate improved calibration, sensitivity (in two out of three cases) and precision when being compared with the standard approaches. We further motivate the usage of our method in active learning, creating pseudo-labels to learn from unlabeled images and human-machine collaboration

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

Abstract: Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts

Augmented Radiology in High Grade Serous Ovarian Carcinoma

Radiological imaging is at the centre of how modern medical care is delivered and provides a unique perspective on disease: it is non-invasively attained, gives 3-dimensional spatially resolved information about disease, and when images are captured sequentially, allows for temporal comparison. Unlocking the information contained within these images, however, relies on access to a radiologist. Their interpretation is time-consuming, largely qualitative, dependent on their level of training and experience, and prone to subjectivity. Demand for radiological imaging outstrips the supply of radiologists needed to interpret them. As healthcare providers struggle to bridge this shortfall, artificial intelligence (AI) tools may provide a solution. Many organisations, including the US Food and Drug Administration (FDA), the UK Medicines and Healthcare products Regulatory Agency (MHRA), the Royal College of Radiologists, and the European Society of Radiology, are convinced that AI-based tools will transform how imaging departments function [1 ]. Such tools are already showing great promise in the research setting, however transition to clinical practice remains largely limited to a small number of paired down decision-aid tools [2]. For AI-based image analysis tools to become truly integrated into clinical care and deliver on their undoubted promise the trust and acceptance of two crucial stakeholder groups: clinicians and patients - is required. Currently, clinicians remain largely distinct from the development and validation of AI-based medical image analysis tools, save for a passive role in the generation of training and testing datasets. Many groups developing them do not have clinician input, which can result in systemic errors or false assumptions obvious to a clinician being overlooked [ 3 ]. The process of evaluating the performance of these tools also happens without clinician input, relying on quantitative metrics which do not necessarily align with utility in a clinical setting. The results of these quantitative evaluations can also be opaque to clinician insight. These factors contribute to a lack of clinician trust in their performance. Consensus is building that greater active clinician involvement in development and performance validation of AI-based medical image analysis tools is required for clinicians to accept oversight responsibility for there use in the clinic, and expedite their translation into clinical practice [1]. Their involvement would also align with the principles of "Good Machine Learning Practice for Medical Device Development: Guiding Principles" [4 ]. How a clinician should formally be involved in these processes, however, is currently unclear. For any successfully validated tool to have full clinical adoption it must be acceptable and trusted by patients [5]. There are numerous examples of promising new technologies which have struggled or been rejected due to lack of public acceptance [6 –10]. How patients will respond to the idea of an AI-based tool being involved in their care, particularly when its judgement informs decisions about treatment, remains largely unknown. This PhD examines 1) the issues of clinician involvement in the development and validation of AI based medical image analysis tools, 2) our patients’ attitude towards them, and 3) the unique insight they may give by deepening our understanding of disease. These topics will be considered in the context of ovarian cancer, a complex, multi-site disease which is frequently diagnosed at late stage, and for which limited improvement in survival outcomes has been made in recent decades. Chapter 3 considers the clinical utility of a newly developed AI-based segmentation tool, which has modest performance by standard quantitative metrics. In the process of establishing its clinical utility, a novel physician led framework for evaluating AI based segmentations is proposed, which assesses the utility of an AI segmentation tool in a clinical context both in an independent and assisting role. Chapter 4 examines the perceptions of patients investigated for ovarian cancer towards having an AI based tool involved in their care. Themes such as trust, privacy, reliability, and accountability are considered in the context of the patient’s own diagnostic journey. The study highlights issues that patients feel should be considered when developing tools which contribute to their care, and makes recommendations on how to pursue patient-focused AI development. With the successful validation of its performance in Chapter 3, Chapter 5 will leverage the segmentation tool to unravel the different spatial distributions of ovarian cancer across a large multi-centre dataset acquired during this PhD, with the aim to better understand the significance of different volume-of-disease distribution patterns in ovarian cancer.Wellcome Trust Innovator Award [RG98755} National Institute for Health and Care Research (NIHR) Invention for Innovation (i4i) Award [NIHR206092

Radioproteomics in patients with ovarian cancer.

Radioproteomics is the integration of proteomics, the systematic study of the protein expression of an organism, with radiomics, the extraction and analysis of large numbers of quantitative features from medical images. This article examines this developing field, and it's application in high grade serous ovarian carcinoma. Seminal proteomic studies in the area of ovarian cancer, such as the PROVAR and CPTA studies are discussed, along side recent research, such as that highlighting the central role of methyltransferase nicotinamide N-methyltransferase as the metabolic regulation of cancer progression in the tumour stroma. Finally, this article considers a novel, hypothesis generating approach to integrate CT-based qualitative and radiomic features with proteomic analysis, and the future direction of the field. Combined advances in radiomic, proteomic and genomic analysis has the potential to signal the age of true precision medicine, where treatment is centered specifically on the molecular profile of the tumour, rather than based on empirical knowledge, thus altering the course of a disease that has the highest mortality of all cancers of the female reproductive system

Radiomics and radiogenomics in ovarian cancer: a literature review.

Ovarian cancer remains one of the most lethal gynecological cancers in the world despite extensive progress in the areas of chemotherapy and surgery. Many studies have postulated that this is because of the profound heterogeneity that underpins response to therapy and prognosis. Standard imaging evaluation using CT or MRI does not take into account this tumoral heterogeneity especially in advanced stages with peritoneal carcinomatosis. As such, newly emergent fields in the assessment of tumor heterogeneity have been proposed using radiomics to evaluate the whole tumor burden heterogeneity as opposed to single biopsy sampling. This review provides an overview of radiomics, radiogenomics, and proteomics and examines the use of these newly emergent fields in assessing tumor heterogeneity and its implications in ovarian cancer

Radiomic and Volumetric Measurements as Clinical Trial Endpoints-A Comprehensive Review.

Clinical trials for oncology drug development have long relied on surrogate outcome biomarkers that assess changes in tumor burden to accelerate drug registration (i.e., Response Evaluation Criteria in Solid Tumors version 1.1 (RECIST v1.1) criteria). Drug-induced reduction in tumor size represents an imperfect surrogate marker for drug activity and yet a radiologically determined objective response rate is a widely used endpoint for Phase 2 trials. With the addition of therapies targeting complex biological systems such as immune system and DNA damage repair pathways, incorporation of integrative response and outcome biomarkers may add more predictive value. We performed a review of the relevant literature in four representative tumor types (breast cancer, rectal cancer, lung cancer and glioblastoma) to assess the preparedness of volumetric and radiomics metrics as clinical trial endpoints. We identified three key areas-segmentation, validation and data sharing strategies-where concerted efforts are required to enable progress of volumetric- and radiomics-based clinical trial endpoints for wider clinical implementation

Artificial intelligence for early detection of renal cancer in computed tomography: A review

Renal cancer is responsible for over 100,000 yearly deaths and is principally discovered in computed tomography (CT) scans of the abdomen. CT screening would likely increase the rate of early renal cancer detection, and improve general survival rates, but it is expected to have a prohibitively high financial cost. Given recent advances in artificial intelligence (AI), it may be possible to reduce the cost of CT analysis and enable CT screening by automating the radiological tasks that constitute the early renal cancer detection pipeline. This review seeks to facilitate further interdisciplinary research in early renal cancer detection by summarising our current knowledge across AI, radiology, and oncology and suggesting useful directions for future novel work. Initially, this review discusses existing approaches in automated renal cancer diagnosis, and methods across broader AI research, to summarise the existing state of AI cancer analysis. Then, this review matches these methods to the unique constraints of early renal cancer detection and proposes promising directions for future research that may enable AI-based early renal cancer detection via CT screening. The primary targets of this review are clinicians with an interest in AI and data scientists with an interest in the early detection of cancer

Artificial intelligence for early detection of renal cancer in computed tomography: A review

Abstract Renal cancer is responsible for over 100,000 yearly deaths and is principally discovered in computed tomography (CT) scans of the abdomen. CT screening would likely increase the rate of early renal cancer detection, and improve general survival rates, but it is expected to have a prohibitively high financial cost. Given recent advances in artificial intelligence (AI), it may be possible to reduce the cost of CT analysis and enable CT screening by automating the radiological tasks that constitute the early renal cancer detection pipeline. This review seeks to facilitate further interdisciplinary research in early renal cancer detection by summarising our current knowledge across AI, radiology, and oncology and suggesting useful directions for future novel work. Initially, this review discusses existing approaches in automated renal cancer diagnosis, and methods across broader AI research, to summarise the existing state of AI cancer analysis. Then, this review matches these methods to the unique constraints of early renal cancer detection and proposes promising directions for future research that may enable AI-based early renal cancer detection via CT screening. The primary targets of this review are clinicians with an interest in AI and data scientists with an interest in the early detection of cancer.ACED - International Alliance for Cancer Early Detection, CRUK National Cancer Imaging Translational Accelerator (NCITA), Wellcome Trust Innovator Award, UK 297 [215733/Z/19/Z] and the National Institute of Health Research (NIHR) Cambridge Biomedical Research Centr

Calibrating ensembles for scalable uncertainty quantification in deep learning-based medical image segmentation

Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult when using deep learning-based models, which are the state-of-the-art in many imaging applications. The current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. Scalable solutions often rely on classical techniques, such as dropout, during inference or training ensembles of identical models with different random seeds to obtain a posterior distribution. In this paper, we present the following contributions. First, we show that the classical approaches fail to approximate the classification probability. Second, we propose a scalable and intuitive framework for uncertainty quantification in medical image segmentation that yields measurements that approximate the classification probability. Third, we suggest the usage of k-fold cross-validation to overcome the need for held out calibration data. Lastly, we motivate the adoption of our method in active learning, creating pseudo-labels to learn from unlabeled images and human-machine collaboration