Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data

Knee osteoarthritis (OA) is the most common musculoskeletal disease without a cure, and current treatment options are limited to symptomatic relief. Prediction of OA progression is a very challenging and timely issue, and it could, if resolved, accelerate the disease modifying drug development and ultimately help to prevent millions of total joint replacement surgeries performed annually. Here, we present a multi-modal machine learning-based OA progression prediction model that utilizes raw radiographic data, clinical examination results and previous medical history of the patient. We validated this approach on an independent test set of 3,918 knee images from 2,129 subjects. Our method yielded area under the ROC curve (AUC) of 0.79 (0.78-0.81) and Average Precision (AP) of 0.68 (0.66-0.70). In contrast, a reference approach, based on logistic regression, yielded AUC of 0.75 (0.74-0.77) and AP of 0.62 (0.60-0.64). The proposed method could significantly improve the subject selection process for OA drug-development trials and help the development of personalized therapeutic plans

Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis

Breast cancer is one of the main causes of cancer death worldwide. Early diagnostics significantly increases the chances of correct treatment and survival, but this process is tedious and often leads to a disagreement between pathologists. Computer-aided diagnosis systems showed potential for improving the diagnostic accuracy. In this work, we develop the computational approach based on deep convolution neural networks for breast cancer histology image classification. Hematoxylin and eosin stained breast histology microscopy image dataset is provided as a part of the ICIAR 2018 Grand Challenge on Breast Cancer Histology Images. Our approach utilizes several deep neural network architectures and gradient boosted trees classifier. For 4-class classification task, we report 87.2% accuracy. For 2-class classification task to detect carcinomas we report 93.8% accuracy, AUC 97.3%, and sensitivity/specificity 96.5/88.0% at the high-sensitivity operating point. To our knowledge, this approach outperforms other common methods in automated histopathological image classification. The source code for our approach is made publicly available at https://github.com/alexander-rakhlin/ICIAR2018Comment: 8 pages, 4 figure

The KNee OsteoArthritis Prediction (KNOAP2020) challenge:An image analysis challenge to predict incident symptomatic radiographic knee osteoarthritis from MRI and X-ray images

Objectives: The KNee OsteoArthritis Prediction (KNOAP2020) challenge was organized to objectively compare methods for the prediction of incident symptomatic radiographic knee osteoarthritis within 78 months on a test set with blinded ground truth. Design: The challenge participants were free to use any available data sources to train their models. A test set of 423 knees from the Prevention of Knee Osteoarthritis in Overweight Females (PROOF) study consisting of magnetic resonance imaging (MRI) and X-ray image data along with clinical risk factors at baseline was made available to all challenge participants. The ground truth outcomes, i.e., which knees developed incident symptomatic radiographic knee osteoarthritis (according to the combined ACR criteria) within 78 months, were not provided to the participants. To assess the performance of the submitted models, we used the area under the receiver operating characteristic curve (ROCAUC) and balanced accuracy (BACC). Results: Seven teams submitted 23 entries in total. A majority of the algorithms were trained on data from the Osteoarthritis Initiative. The model with the highest ROCAUC (0.64 (95% confidence interval (CI): 0.57–0.70)) used deep learning to extract information from X-ray images combined with clinical variables. The model with the highest BACC (0.59 (95% CI: 0.52–0.65)) ensembled three different models that used automatically extracted X-ray and MRI features along with clinical variables. Conclusion: The KNOAP2020 challenge established a benchmark for predicting incident symptomatic radiographic knee osteoarthritis. Accurate prediction of incident symptomatic radiographic knee osteoarthritis is a complex and still unsolved problem requiring additional investigation.</p

Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging

Using medical images to evaluate disease severity and change over time is a routine and important task in clinical decision making. Grading systems are often used, but are unreliable as domain experts disagree on disease severity category thresholds. These discrete categories also do not reflect the underlying continuous spectrum of disease severity. To address these issues, we developed a convolutional Siamese neural network approach to evaluate disease severity at single time points and change between longitudinal patient visits on a continuous spectrum. We demonstrate this in two medical imaging domains: retinopathy of prematurity (ROP) in retinal photographs and osteoarthritis in knee radiographs. Our patient cohorts consist of 4861 images from 870 patients in the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) cohort study and 10,012 images from 3021 patients in the Multicenter Osteoarthritis Study (MOST), both of which feature longitudinal imaging data. Multiple expert clinician raters ranked 100 retinal images and 100 knee radiographs from excluded test sets for severity of ROP and osteoarthritis, respectively. The Siamese neural network output for each image in comparison to a pool of normal reference images correlates with disease severity rank (ρ = 0.87 for ROP and ρ = 0.89 for osteoarthritis), both within and between the clinical grading categories. Thus, this output can represent the continuous spectrum of disease severity at any single time point. The difference in these outputs can be used to show change over time. Alternatively, paired images from the same patient at two time points can be directly compared using the Siamese neural network, resulting in an additional continuous measure of change between images. Importantly, our approach does not require manual localization of the pathology of interest and requires only a binary label for training (same versus different). The location of disease and site of change detected by the algorithm can be visualized using an occlusion sensitivity map-based approach. For a longitudinal binary change detection task, our Siamese neural networks achieve test set receiving operator characteristic area under the curves (AUCs) of up to 0.90 in evaluating ROP or knee osteoarthritis change, depending on the change detection strategy. The overall performance on this binary task is similar compared to a conventional convolutional deep-neural network trained for multi-class classification. Our results demonstrate that convolutional Siamese neural networks can be a powerful tool for evaluating the continuous spectrum of disease severity and change in medical imaging

Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior authors: Paul F. J\"ager, Lena Maier-Hei

Deep learning for knee osteoarthritis diagnosis and progression prediction from plain radiographs and clinical data

Abstract Osteoarthritis (OA) is the most common musculoskeletal disorder in the world, affecting hand, hip, and knee joints. At the final stage, OA leads to joint replacement, causing an immense burden at the individual and societal levels. Multiple risk factors that can lead to OA are known; however, the etiology of OA and the underlying mechanisms of OA progression are not currently known. OA is currently diagnosed by a clinical examination and, when necessary, confirmed by imaging — a radiographic evaluation. However, these conventional tools are not sensitive to detect the early stages of OA, which makes the development of preventive measures for further disease progression difficult. Therefore, there is a need for other methods that could allow for the early diagnosis of OA. As such, computer vision-based techniques provide quantitative biomarkers that allow for an automatic and systematic assessment of OA severity from images. In recent years, the rapid development of computer vision and machine learning methods have merged into a new field — deep learning (DL). DL allows for one to formulate the problems of computer vision and other fields in a machine learning fashion. In the medical field, DL has made a tremendous impact and allowed to approach for human-level decision-making accuracy in diagnostic and prognostic tasks compared with the traditional computer vision-based methods. The focus of this thesis is on the development of DL-based methods for fully automatic knee OA severity diagnosis and the prediction of its progression. Multiple new methods for localizing the region of interest, landmark localization, knee OA severity assessment, and OA progression prediction are proposed. The results exceeded the state-of-the-art or formed completely new benchmarks for the evaluation of diagnostic and predictive model performance in OA. The main conclusion is that DL yields excellent performance in the diagnostics of OA and in the prediction of its progression. All the source codes of all the developed methods and the annotations for some of the datasets have been made publicly available.Tiivistelmä Nivelrikko on maailman yleisin käden, lonkan ja polven niveliin vaikuttava liikuntaelinsairaus. Viimekädessä nivelrikko johtaa tekonivelleikkauksiin, aiheuttaen merkittävää rasitetta niin yksilö- kuin yhteiskunnallisella tasolla. Monia nivelrikolle altistavia tekijöitä on jo tunnistettu, mutta kaikkia nivelrikon syitä ja vaikutusmekanismeja nivelrikon etenemisessä ei tunneta. Nivelrikko diagnosoidaan kliinisellä tutkimuksella ja vahvistetaan/varmistetaan tarvittaessa tehtävällä kuvantamistutkimuksella — tekemällä radiografinen arviointi. Nämä perinteiset työkalut eivät kuitenkaan ole riittävän herkkiä nivelrikon varhaisten vaiheiden havaitsemiseen, ja tämä hankaloittaa sairauden kehittymistä ehkäisevien toimenpiteiden kehittämistä. Näistä syistä johtuen tarvitaan muita menetelmiä, jotka mahdollistavat nivelrikon varhaisen diagnosoinnin. Konenäkömenetelmät sellaisenaan tuottavat kvantitatiivisia biologisia indikaattoreita jotka mahdollistavat automaattisen ja järjestelmällisen nivelrikon vakavuusarvion tekemisen kuvamateriaalista. Viime vuosina konenäkö- ja koneoppimismenetelmien nopea kehitys on synnyttänyt uuden syväoppimisen haaran. Syväoppiminen mahdollistaa konenäkö- ja muiden ongelmien määrittelyn koneoppimisongelman tavoin. Verrattuna perinteisiin lääketieteessä käytettyihin tietokonenäkömenetelmiin, syväoppiminen on mahdollistanut ihmisen suorituskykyä lähestyvät toteutukset lääketieteen diagnostisissa ja prognostisissa tehtävissä ja niiden vaikutus alan kehitykselle on ollut merkittävä. Tämän väitöskirja keskittyy kehittämään syväoppimismenetelmiä täysautomaattiseen polven nivelrikon vakavuuden diagnosointiin ja taudin kehittymisen ennustamiseen. Työssä ehdotetaan/esitetään useita uusia menetelmiä kohdealueen paikallistamiseen, maamerkkien paikallistamiseen, polven nivelrikon vakavuuden arviointiin ja nivelrikon etenemisen ennustamiseen. Työn tulokset ylittävät viimeisintä tekniikkaa edustavat ratkaisut tai muodostavat täysin uuden mittarin diagnostisten ja ennustavien menetelmien suorituskyvyn evaluoinnille nivelrikon kontekstissa. Työn keskeisimpänä johtopäätöksenä esitetään, että syväoppimisella on mahdollista saavuttaa erittäin hyvä suorituskyky nivelrikon diagnosoinnissa ja sen etenemisen ennustamisessa. Kaikki työssä kehitetyt menetelmät lähdekoodeineen sekä annotoinnit osalle tutkimuksessa käytetyistä aineistoista on saatettu avoimesti saataville

Automatic grading of individual knee osteoarthritis features in plain radiographs using deep convolutional neural networks

Abstract Knee osteoarthritis (OA) is the most common musculoskeletal disease in the world. In primary healthcare, knee OA is diagnosed using clinical examination and radiographic assessment. Osteoarthritis Research Society International (OARSI) atlas of OA radiographic features allows performing independent assessment of knee osteophytes, joint space narrowing and other knee features. This provides a fine-grained OA severity assessment of the knee, compared to the gold standard and most commonly used Kellgren–Lawrence (KL) composite score. In this study, we developed an automatic method to predict KL and OARSI grades from knee radiographs. Our method is based on Deep Learning and leverages an ensemble of residual networks with 50 layers. We used transfer learning from ImageNet with a fine-tuning on the Osteoarthritis Initiative (OAI) dataset. An independent testing of our model was performed on the Multicenter Osteoarthritis Study (MOST) dataset. Our method yielded Cohen’s kappa coefficients of 0.82 for KL-grade and 0.79, 0.84, 0.94, 0.83, 0.84 and 0.90 for femoral osteophytes, tibial osteophytes and joint space narrowing for lateral and medial compartments, respectively. Furthermore, our method yielded area under the ROC curve of 0.98 and average precision of 0.98 for detecting the presence of radiographic OA, which is better than the current state-of-the-art

Kneel:knee anatomical landmark localization using hourglass networks

Abstract This paper addresses the challenge of localization of anatomical landmarks in knee X-ray images at different stages of osteoarthritis (OA). Landmark localization can be viewed as regression problem, where the landmark position is directly predicted by using the region of interest or even full-size images leading to large memory footprint, especially in case of high resolution medical images. In this work, we propose an efficient deep neural networks framework with an hourglass architecture utilizing a soft-argmax layer to directly predict normalized coordinates of the landmark points. We provide an extensive evaluation of different regularization techniques and various loss functions to understand their influence on the localization performance. Furthermore, we introduce the concept of transfer learning from low-budget annotations, and experimentally demonstrate that such approach is improving the accuracy of landmark localization. Compared to the prior methods, we validate our model on two datasets that are independent from the train data and assess the performance of the method for different stages of OA severity. The proposed approach demonstrates better generalization performance compared to the current state-of-the-art

Deep semi-supervised active learning for knee osteoarthritis severity grading

Abstract This paper tackles the problem of developing active learning (AL) methods in the context of knee osteoarthritis (OA) diagnosis from X-ray images. OA is known to be a huge burden for society, and its associated costs are constantly rising. Automatic diagnostic methods can potentially reduce these costs, and Deep Learning (DL) methodology may be its key enabler. To date, there have been numerous studies on knee OA severity grading using DL, and all but one of them assume a large annotated dataset available for model development. In contrast, our study shows one can develop a knee OA severity grading model using AL from as little as 50 samples randomly chosen from a pool of unlabeled data. The main insight of this work is that the performance of AL improves when the model developer leverages the consistency regularization technique, commonly applied in semi-supervised learning

Breast Tumor Cellularity Assessment Using Deep Neural Networks

© 2019 IEEE. Breast cancer is one of the main causes of death worldwide. Histopathological cellularity assessment of residual tumors in post-surgical tissues is used to analyze a tumor's response to a therapy. Correct cellularity assessment increases the chances of getting an appropriate treatment and facilitates the patient's survival. In current clinical practice, tumor cellularity is manually estimated by pathologists; this process is tedious and prone to errors or low agreement rates between assessors. In this work, we evaluated three strong novel Deep Learning-based approaches for automatic assessment of tumor cellularity from post-treated breast surgical specimens stained with hematoxylin and eosin. We validated the proposed methods on the BreastPathQ SPIE challenge dataset that consisted of 2395 image patches selected from whole slide images acquired from 64 patients. Compared to expert pathologist scoring, our best performing method yielded the Cohen's kappa coefficient of 0.69 (vs. 0.42 previously known in literature) and the intra-class correlation coefficient of 0.89 (vs. 0.83). Our results suggest that Deep Learning-based methods have a significant potential to alleviate the burden on pathologists, enhance the diagnostic workflow, and, thereby, facilitate better clinical outcomes in breast cancer treatment