MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.Comment: CVPR2017 Ora

Cancer diagnosis using deep learning: A bibliographic review

In this paper, we first describe the basics of the field of cancer diagnosis, which includes steps of cancer diagnosis followed by the typical classification methods used by doctors, providing a historical idea of cancer classification techniques to the readers. These methods include Asymmetry, Border, Color and Diameter (ABCD) method, seven-point detection method, Menzies method, and pattern analysis. They are used regularly by doctors for cancer diagnosis, although they are not considered very efficient for obtaining better performance. Moreover, considering all types of audience, the basic evaluation criteria are also discussed. The criteria include the receiver operating characteristic curve (ROC curve), Area under the ROC curve (AUC), F1 score, accuracy, specificity, sensitivity, precision, dice-coefficient, average accuracy, and Jaccard index. Previously used methods are considered inefficient, asking for better and smarter methods for cancer diagnosis. Artificial intelligence and cancer diagnosis are gaining attention as a way to define better diagnostic tools. In particular, deep neural networks can be successfully used for intelligent image analysis. The basic framework of how this machine learning works on medical imaging is provided in this study, i.e., pre-processing, image segmentation and post-processing. The second part of this manuscript describes the different deep learning techniques, such as convolutional neural networks (CNNs), generative adversarial models (GANs), deep autoencoders (DANs), restricted Boltzmann’s machine (RBM), stacked autoencoders (SAE), convolutional autoencoders (CAE), recurrent neural networks (RNNs), long short-term memory (LTSM), multi-scale convolutional neural network (M-CNN), multi-instance learning convolutional neural network (MIL-CNN). For each technique, we provide Python codes, to allow interested readers to experiment with the cited algorithms on their own diagnostic problems. The third part of this manuscript compiles the successfully applied deep learning models for different types of cancers. Considering the length of the manuscript, we restrict ourselves to the discussion of breast cancer, lung cancer, brain cancer, and skin cancer. The purpose of this bibliographic review is to provide researchers opting to work in implementing deep learning and artificial neural networks for cancer diagnosis a knowledge from scratch of the state-of-the-art achievements

Addressing current challenges in cancer immunotherapy with mathematical and computational modeling

The goal of cancer immunotherapy is to boost a patient's immune response to a tumor. Yet, the design of an effective immunotherapy is complicated by various factors, including a potentially immunosuppressive tumor microenvironment, immune-modulating effects of conventional treatments, and therapy-related toxicities. These complexities can be incorporated into mathematical and computational models of cancer immunotherapy that can then be used to aid in rational therapy design. In this review, we survey modeling approaches under the umbrella of the major challenges facing immunotherapy development, which encompass tumor classification, optimal treatment scheduling, and combination therapy design. Although overlapping, each challenge has presented unique opportunities for modelers to make contributions using analytical and numerical analysis of model outcomes, as well as optimization algorithms. We discuss several examples of models that have grown in complexity as more biological information has become available, showcasing how model development is a dynamic process interlinked with the rapid advances in tumor-immune biology. We conclude the review with recommendations for modelers both with respect to methodology and biological direction that might help keep modelers at the forefront of cancer immunotherapy development.Comment: Accepted for publication in the Journal of the Royal Society Interfac

Deep Learning Models to Characterize Smooth Muscle Fibers in Hematoxylin and Eosin Stained Histopathological Images of the Urinary Bladder

Muscularis propria (MP) and muscularis mucosa (MM), two types of smooth muscle fibers in the urinary bladder, are major benchmarks in staging bladder cancer to distinguish between muscle-invasive (MP invasion) and non-muscle-invasive (MM invasion) diseases. While patients with non-muscle-invasive tumor can be treated conservatively involving transurethral resection (TUR) only, more aggressive treatment options, such as removal of the entire bladder, known as radical cystectomy (RC) which may severely degrade the quality of patient’s life, are often required in those with muscle-invasive tumor. Hence, given two types of image datasets, hematoxylin & eosin-stained histopathological images from RC and TUR specimens, we propose the first deep learning-based method for efficient characterization of MP. The proposed method is intended to aid the pathologists as a decision support system by facilitating accurate staging of bladder cancer. In this work, we aim to semantically segment the TUR images into MP and non-MP regions using two different approaches, patch-to-label and pixel-to-label. We evaluate four different state-of-the-art CNN-based models (VGG16, ResNet18, SqueezeNet, and MobileNetV2) and semantic segmentation-based models (U-Net, MA-Net, DeepLabv3+, and FPN) and compare their performance metrics at the pixel-level. The SqueezeNet model (mean Jaccard Index: 95.44%, mean dice coefficient: 97.66%) in patch-to-label approach and the MA-Net model (mean Jaccard Index: 96.64%, mean dice coefficient: 98.29%) in pixel-to-label approach are the best among tested models. Although pixel-to-label approach is marginally better than the patch-to-label approach based on evaluation metrics, the latter is computationally efficient using least trainable parameters

A Survey on Deep Learning in Medical Image Analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.Comment: Revised survey includes expanded discussion section and reworked introductory section on common deep architectures. Added missed papers from before Feb 1st 201

Studies on Category Prediction of Ovarian Cancers Based on Magnetic Resonance Images

Ovarian cancer is the gynecological malignant tumor with low early diagnosis rate and high mortality. Ovarian epithelial cancer (OEC) is the most common subtype of ovarian cancer. Pathologically, OEC is divided into two subtypes: Type I and Type II. These two subtypes of OEC have different biological characteristics and treatment response. Therefore, it is important to accurately categorize these two groups of patients and provide the reference for clinicians in designing treatment plans. In the current magnetic resonance (MR) examination, the diagnoses given by the radiologists are largely based on individual judgment and not sufficiently accurate. Because of the low accuracy of the results and the risk of suffering Type II OEC, most patients will undertake the fine-needle aspiration, which may cause harm to patients’ bodies. Therefore, there is need for the method for OEC subtype classification based on MR images. This thesis proposes the automatic diagnosis system of ovarian cancer based on the combination of deep learning and radiomics. The method utilizes four common useful sequences for ovarian cancer diagnosis: sagittal fat-suppressed T2WI (Sag-fs-T2WI), coronal T2WI (Cor-T2WI), axial T1WI (Axi-T1WI), and apparent diffusion coefficient map (ADC) to establish a multi-sequence diagnostic model. The system starts with the segmentation of the ovarian tumors, and then obtains the radiomic features from lesion parts together with the network features. Selected Features are used to build model to predict the malignancy of ovarian cancers, the subtype of OEC and the survival condition. Bi-atten-ResUnet is proposed in this thesis as the segmentation model. The network is established on the basis of U-Net with adopting Residual block and non-local attention module. It preserves the classic encoder/decoder architecture in the U-Net network. The encoder part is reconstructed by the pretrained ResNet to make use of transfer learning knowledge, and bi-non-local attention modules are added to the decoder part on each level. The application of these techniques enhances the network’s performance in segmentation tasks. The model achieves 0.918, 0.905, 0.831, and 0.820 Dice coefficient respectively in segmenting on four MR sequences. After the segmentation work, the thesis proposes a diagnostic model with three steps: quantitative description feature extraction, feature selection, and establishment of prediction models. First, radiomic features and network features are obtained. Then iterative sparse representation (ISR) method is adopted as the feature selection to reduce the redundancy and correlation. The selected features are used to establish a predictive model, and support vector machine (SVM) is used as the classifier. The model achieves an AUC of 0.967 in distinguishing between benign and malignant ovarian tumors. For discriminating Type I and Type II OEC, the model yields an AUC of 0.823. In the survival prediction, patients categorized in high risk group are more likely to have poor prognosis with hazard ratio 4.169

Automated Grading of Bladder Cancer using Deep Learning

PhD thesis in Information technologyUrothelial carcinoma is the most common type of bladder cancer and is among the cancer types with the highest recurrence rate and lifetime treatment cost per patient. Diagnosed patients are stratified into risk groups, mainly based on the histological grade and stage. However, it is well known that correct grading of bladder cancer suffers from intra- and interobserver variability and inconsistent reproducibility between pathologists, potentially leading to under- or overtreatment of the patients. The economic burden, unnecessary patient suffering, and additional load on the health care system illustrate the importance of developing new tools to aid pathologists. With the introduction of digital pathology, large amounts of data have been made available in the form of digital histological whole-slide images (WSI). However, despite the massive amount of data, annotations for the given data are lacking. Another potential problem is that the tissue samples of urothelial carcinoma contain a mixture of damaged tissue, blood, stroma, muscle, and urothelium, where it is mainly the urothelium tissue that is diagnostically relevant for grading. A method for tissue segmentation is investigated, where the aim is to segment WSIs into the six tissue classes: urothelium, stroma, muscle, damaged tissue, blood, and background. Several methods based on convolutional neural networks (CNN) for tile-wise classification are proposed. Both single-scale and multiscale models were explored to see if including more magnification levels would improve the performance. Different techniques, such as unsupervised learning, semi-supervised learning, and domain adaptation techniques, are explored to mitigate the challenge of missing large quantities of annotated data. It is necessary to extract tiles from the WSI since it is intractable to process the entire WSI at full resolution at once. We have proposed a method to parameterize and automate the task of extracting tiles from different scales with a region of interest (ROI) defined at one of the scales. The method is reproducible and easy to describe by reporting the parameters. A pipeline for automated diagnostic grading is proposed, called TRIgrade. First, the tissue segmentation method is utilized to find the diagnostically relevant urothelium tissue. Then, the parameterized tile extraction method is used to extract tiles from the urothelium regions at three magnification levels from 300 WSIs. The extracted tiles form the training, validation, and test data used to train and test a diagnostic model. The final system outputs a segmented tissue image showing all the tissue regions in the WSI, a WHO grade heatmap indicating low- and high-grade carcinoma regions, and finally, a slide-level WHO grade prediction. The proposed TRIgrade pipeline correctly graded 45 of 50 WSIs, achieving an accuracy of 90%