Deep Learning in Cardiology

The medical field is creating large amount of data that physicians are unable to decipher and use efficiently. Moreover, rule-based expert systems are inefficient in solving complicated medical tasks or for creating insights using big data. Deep learning has emerged as a more accurate and effective technology in a wide range of medical problems such as diagnosis, prediction and intervention. Deep learning is a representation learning method that consists of layers that transform the data non-linearly, thus, revealing hierarchical relationships and structures. In this review we survey deep learning application papers that use structured data, signal and imaging modalities from cardiology. We discuss the advantages and limitations of applying deep learning in cardiology that also apply in medicine in general, while proposing certain directions as the most viable for clinical use.Comment: 27 pages, 2 figures, 10 table

Reasoning with Uncertainty in Deep Learning for Safer Medical Image Computing

Deep learning is now ubiquitous in the research field of medical image computing. As such technologies progress towards clinical translation, the question of safety becomes critical. Once deployed, machine learning systems unavoidably face situations where the correct decision or prediction is ambiguous. However, the current methods disproportionately rely on deterministic algorithms, lacking a mechanism to represent and manipulate uncertainty. In safety-critical applications such as medical imaging, reasoning under uncertainty is crucial for developing a reliable decision making system. Probabilistic machine learning provides a natural framework to quantify the degree of uncertainty over different variables of interest, be it the prediction, the model parameters and structures, or the underlying data (images and labels). Probability distributions are used to represent all the uncertain unobserved quantities in a model and how they relate to the data, and probability theory is used as a language to compute and manipulate these distributions. In this thesis, we explore probabilistic modelling as a framework to integrate uncertainty information into deep learning models, and demonstrate its utility in various high-dimensional medical imaging applications. In the process, we make several fundamental enhancements to current methods. We categorise our contributions into three groups according to the types of uncertainties being modelled: (i) predictive; (ii) structural and (iii) human uncertainty. Firstly, we discuss the importance of quantifying predictive uncertainty and understanding its sources for developing a risk-averse and transparent medical image enhancement application. We demonstrate how a measure of predictive uncertainty can be used as a proxy for the predictive accuracy in the absence of ground-truths. Furthermore, assuming the structure of the model is flexible enough for the task, we introduce a way to decompose the predictive uncertainty into its orthogonal sources i.e. aleatoric and parameter uncertainty. We show the potential utility of such decoupling in providing a quantitative “explanations” into the model performance. Secondly, we introduce our recent attempts at learning model structures directly from data. One work proposes a method based on variational inference to learn a posterior distribution over connectivity structures within a neural network architecture for multi-task learning, and share some preliminary results in the MR-only radiotherapy planning application. Another work explores how the training algorithm of decision trees could be extended to grow the architecture of a neural network to adapt to the given availability of data and the complexity of the task. Lastly, we develop methods to model the “measurement noise” (e.g., biases and skill levels) of human annotators, and integrate this information into the learning process of the neural network classifier. In particular, we show that explicitly modelling the uncertainty involved in the annotation process not only leads to an improvement in robustness to label noise, but also yields useful insights into the patterns of errors that characterise individual experts

Doctor of Philosophy

dissertationMachine learning is the science of building predictive models from data that automatically improve based on past experience. To learn these models, traditional learning algorithms require labeled data. They also require that the entire dataset fits in the memory of a single machine. Labeled data are available or can be acquired for small and moderately sized datasets but curating large datasets can be prohibitively expensive. Similarly, massive datasets are usually too huge to fit into the memory of a single machine. An alternative is to distribute the dataset over multiple machines. Distributed learning, however, poses new challenges as most existing machine learning techniques are inherently sequential. Additionally, these distributed approaches have to be designed keeping in mind various resource limitations of real-world settings, prime among them being intermachine communication. With the advent of big datasets machine learning algorithms are facing new challenges. Their design is no longer limited to minimizing some loss function but, additionally, needs to consider other resources that are critical when learning at scale. In this thesis, we explore different models and measures for learning with limited resources that have a budget. What budgetary constraints are posed by modern datasets? Can we reuse or combine existing machine learning paradigms to address these challenges at scale? How does the cost metrics change when we shift to distributed models for learning? These are some of the questions that have been investigated in this thesis. The answers to these questions hold the key to addressing some of the challenges faced when learning on massive datasets. In the first part of this thesis, we present three different budgeted scenarios that deal with scarcity of labeled data and limited computational resources. The goal is to leverage transfer information from related domains to learn under budgetary constraints. Our proposed techniques comprise semisupervised transfer, online transfer and active transfer. In the second part of this thesis, we study distributed learning with limited communication. We present initial sampling based results, as well as, propose communication protocols for learning distributed linear classifiers

Discovering multi-purpose modules through deep multitask learning

Machine learning scientists aim to discover techniques that can be applied across diverse sets of problems. Such techniques need to exploit regularities that are shared across tasks. This begs the question: What shared regularity is not yet being exploited? Complex tasks may share structure that is difficult for humans to discover. The goal of deep multitask learning is to discover and exploit this structure automatically by training a joint model across tasks. To this end, this dissertation introduces a deep multitask learning framework for collecting generic functional modules that are used in different ways to solve different problems. Within this framework, a progression of systems is developed based on assembling shared modules into task models and leveraging the complementary advantages of gradient descent and evolutionary optimization. In experiments, these systems confirm that modular sharing improves performance across a range of application areas, including general video game playing, computer vision, natural language processing, and genomics; yielding state-of-the-art results in several cases. The conclusion is that multi-purpose modules discovered by deep multitask learning can exceed those developed by humans in performance and generality.Computer Science

Multi-Task Deep Learning Model for Classification of Dental Implant Brand and Treatment Stage Using Dental Panoramic Radiograph Images

It is necessary to accurately identify dental implant brands and the stage of treatment to ensure efficient care. Thus, the purpose of this study was to use multi-task deep learning to investigate a classifier that categorizes implant brands and treatment stages from dental panoramic radiographic images. For objective labeling, 9767 dental implant images of 12 implant brands and treatment stages were obtained from the digital panoramic radiographs of patients who underwent procedures at Kagawa Prefectural Central Hospital, Japan, between 2005 and 2020. Five deep convolutional neural network (CNN) models (ResNet18, 34, 50, 101 and 152) were evaluated. The accuracy, precision, recall, specificity, F1 score, and area under the curve score were calculated for each CNN. We also compared the multi-task and single-task accuracies of brand classification and implant treatment stage classification. Our analysis revealed that the larger the number of parameters and the deeper the network, the better the performance for both classifications. Multi-tasking significantly improved brand classification on all performance indicators, except recall, and significantly improved all metrics in treatment phase classification. Using CNNs conferred high validity in the classification of dental implant brands and treatment stages. Furthermore, multi-task learning facilitated analysis accuracy

Parieto-Occipital Alpha and Low-Beta EEG Power Reflect Sense of Agency

The sense of agency (SoA) is part of psychophysiological modules related to the self. Disturbed SoA is found in several clinical conditions, hence understanding the neural correlates of the SoA is useful for the diagnosis and determining the proper treatment strategies. Although there are several neuroimaging studies on SoA, it is desirable to translate the knowledge to more accessible and inexpensive EEG-based biomarkers for the sake of applicability. However, SoA has not been widely investigated using EEG. To address this issue, we designed an EEG experiment on healthy adults (n = 15) to determine the sensitivity of EEG on the SoA paradigm using hand movement with parametrically delayed visual feedback. We calculated the power spectral density over the traditional EEG frequency bands for ten delay conditions relative to no delay condition. Independent component analysis and equivalent current dipole modeling were applied to address artifact rejection, volume conduction, and source localization to determine the effect of interest. The results revealed that the alpha and low-beta EEG power increased in the parieto-occipital regions in proportion to the reduced SoA reported by the subjects. We conclude that the parieto-occipital alpha and low-beta EEG power reflect the sense of agency

Diagnóstico automático de melanoma mediante técnicas modernas de aprendizaje automático

The incidence and mortality rates of skin cancer remain a huge concern in many countries. According to the latest statistics about melanoma skin cancer, only in the Unites States, 7,650 deaths are expected in 2022, which represents 800 and 470 more deaths than 2020 and 2021, respectively. In 2022, melanoma is ranked as the fifth cause of new cases of cancer, with a total of 99,780 people. This illness is mainly diagnosed with a visual inspection of the skin, then, if doubts remain, a dermoscopic analysis is performed. The development of e_ective non-invasive diagnostic tools for the early stages of the illness should increase quality of life, and decrease the required economic resources. The early diagnosis of skin lesions remains a tough task even for expert dermatologists because of the complexity, variability, dubiousness of the symptoms, and similarities between the different categories among skin lesions. To achieve this goal, previous works have shown that early diagnosis from skin images can benefit greatly from using computational methods. Several studies have applied handcrafted-based methods on high quality dermoscopic and histological images, and on top of that, machine learning techniques, such as the k-nearest neighbors approach, support vector machines and random forest. However, one must bear in mind that although the previous extraction of handcrafted features incorporates an important knowledge base into the analysis, the quality of the extracted descriptors relies heavily on the contribution of experts. Lesion segmentation is also performed manually. The above procedures have a common issue: they are time-consuming manual processes prone to errors. Furthermore, an explicit definition of an intuitive and interpretable feature is hardly achievable, since it depends on pixel intensity space and, therefore, they are not invariant regarding the differences in the input images. On the other hand, the use of mobile devices has sharply increased, which offers an almost unlimited source of data. In the past few years, more and more attention has been paid to designing deep learning models for diagnosing melanoma, more specifically Convolutional Neural Networks. This type of model is able to extract and learn high-level features from raw images and/or other data without the intervention of experts. Several studies showed that deep learning models can overcome handcrafted-based methods, and even match the predictive performance of dermatologists. The International Skin Imaging Collaboration encourages the development of methods for digital skin imaging. Every year since 2016 to 2019, a challenge and a conference have been organized, in which more than 185 teams have participated. However, convolutional models present several issues for skin diagnosis. These models can fit on a wide diversity of non-linear data points, being prone to overfitting on datasets with small numbers of training examples per class and, therefore, attaining a poor generalization capacity. On the other hand, this type of model is sensitive to some characteristics in data, such as large inter-class similarities and intra-class variances, variations in viewpoints, changes in lighting conditions, occlusions, and background clutter, which can be mostly found in non-dermoscopic images. These issues represent challenges for the application of automatic diagnosis techniques in the early phases of the illness. As a consequence of the above, the aim of this Ph.D. thesis is to make significant contributions to the automatic diagnosis of melanoma. The proposals aim to avoid overfitting and improve the generalization capacity of deep models, as well as to achieve a more stable learning and better convergence. Bear in mind that research into deep learning commonly requires an overwhelming processing power in order to train complex architectures. For example, when developing NASNet architecture, researchers used 500 x NVidia P100s - each graphic unit cost from $5,899 to$7,374, which represents a total of $2,949,500.00 -$3,687,000.00. Unfortunately, the majority of research groups do not have access to such resources, including ours. In this Ph.D. thesis, the use of several techniques has been explored. First, an extensive experimental study was carried out, which included state-of-the-art models and methods to further increase the performance. Well-known techniques were applied, such as data augmentation and transfer learning. Data augmentation is performed in order to balance out the number of instances per category and act as a regularizer in preventing overfitting in neural networks. On the other hand, transfer learning uses weights of a pre-trained model from another task, as the initial condition for the learning of the target network. Results demonstrate that the automatic diagnosis of melanoma is a complex task. However, different techniques are able to mitigate such issues in some degree. Finally, suggestions are given about how to train convolutional models for melanoma diagnosis and future interesting research lines were presented. Next, the discovery of ensemble-based architectures is tackled by using genetic algorithms. The proposal is able to stabilize the training process. This is made possible by finding sub-optimal combinations of abstract features from the ensemble, which are used to train a convolutional block. Then, several predictive blocks are trained at the same time, and the final diagnosis is achieved by combining all individual predictions. We empirically investigate the benefits of the proposal, which shows better convergence, mitigates the overfitting of the model, and improves the generalization performance. On top of that, the proposed model is available online and can be consulted by experts. The next proposal is focused on designing an advanced architecture capable of fusing classical convolutional blocks and a novel model known as Dynamic Routing Between Capsules. This approach addresses the limitations of convolutional blocks by using a set of neurons instead of an individual neuron in order to represent objects. An implicit description of the objects is learned by each capsule, such as position, size, texture, deformation, and orientation. In addition, a hyper-tuning of the main parameters is carried out in order to ensure e_ective learning under limited training data. An extensive experimental study was conducted where the fusion of both methods outperformed six state-of-the-art models. On the other hand, a robust method for melanoma diagnosis, which is inspired on residual connections and Generative Adversarial Networks, is proposed. The architecture is able to produce plausible photorealistic synthetic 512 x 512 skin images, even with small dermoscopic and non-dermoscopic skin image datasets as problema domains. In this manner, the lack of data, the imbalance problems, and the overfitting issues are tackled. Finally, several convolutional modes are extensively trained and evaluated by using the synthetic images, illustrating its effectiveness in the diagnosis of melanoma. In addition, a framework, which is inspired on Active Learning, is proposed. The batch-based query strategy setting proposed in this work enables a more faster training process by learning about the complexity of the data. Such complexities allow us to adjust the training process after each epoch, which leads the model to achieve better performance in a lower number of iterations compared to random mini-batch sampling. Then, the training method is assessed by analyzing both the informativeness value of each image and the predictive performance of the models. An extensive experimental study is conducted, where models trained with the proposal attain significantly better results than the baseline models. The findings suggest that there is still space for improvement in the diagnosis of skin lesions. Structured laboratory data, unstructured narrative data, and in some cases, audio or observational data, are given by radiologists as key points during the interpretation of the prediction. This is particularly true in the diagnosis of melanoma, where substantial clinical context is often essential. For example, symptoms like itches and several shots of a skin lesion during a period of time proving that the lesion is growing, are very likely to suggest cancer. The use of different types of input data could help to improve the performance of medical predictive models. In this regard, a _rst evolutionary algorithm aimed at exploring multimodal multiclass data has been proposed, which surpassed a single-input model. Furthermore, the predictive features extracted by primary capsules could be used to train other models, such as Support Vector Machine

Deep learning for biomarker and outcome prediction in cancer

Machine learning in the form of deep learning (DL) has recently transformed how computer vision tasks are solved in numerous domains, including image-based medical diagnostics. DL-based methods have the potential to enable more precise quantitative characterisation of cancer tissue specimens routinely analysed in clinical pathology laboratories for diagnostic purposes. Computer-assisted tissue analysis within pathology is not restricted to the quantification and classification of specific tissue entities. DL allows to directly address clinically relevant questions related to the prediction of cancer outcome and efficacy of cancer treatment. This thesis focused on the following crucial research question: is it possible to predict cancer outcome, biomarker status, and treatment efficacy directly from the tissue morphology using DL without any special stains or molecular methods? To address this question, we utilised digitised hematoxylin-eosin-stained (H&E) tissue specimens from two common types of solid tumours – breast and colorectal cancer. Tissue specimens and corresponding clinical data were retrieved from retrospective patient series collected in Finland. First, a DL-based algorithm was developed to extract prognostic information for patients diagnosed with colorectal cancer, using digitised H&E images only. Computational analysis of tumour tissue samples with DL demonstrated a superhuman performance and surpassed a consensus of three expert pathologists in predicting five-year colorectal cancer-specific outcomes. Then, outcome prediction was studied in two independent breast cancer patient series. Particularly, generalisation of the trained algorithms to previously unseen patients from an independent series was examined on the large whole-slide tumour specimens. In breast cancer outcome prediction, we investigated a multitask learning approach by combining outcome and biomarker-supervised learning. Our experiments in breast and colorectal cancer show that tissue morphological features learned by the DL models supervised by patient outcome provided prognostic information independent of established prognostic factors such as histological grade, tumour size and lymph nodes status. Additionally, the accuracy of DL-based predictors was compared to other prognostic characteristics evaluated by pathologists in breast cancer, including mitotic count, nuclear pleomorphism, tubules formation, tumour necrosis and tumour-infiltrating lymphocytes. We further assessed if molecular biomarkers such as hormone receptor status and ERBB2 gene amplification can be predicted from H&E- stained tissue samples obtained at the time of diagnosis from patients with breast cancer and showed that molecular alterations are reflected in the basic tissue morphology and can be captured with DL. Finally, we studied how morphological features of breast cancer can be linked to molecularly targeted treatment response. The results showed that ERBB2-associated morphology extracted with DL correlated with the efficacy of adjuvant anti-ERBB2 treatment and can contribute to treatment-predictive information in breast cancer. Taken together, this thesis shows the potential utility of DL in tissue-based characterisation of cancer for prediction of cancer outcome, tumour molecular status and efficacy of molecularly targeted treatments. DL-based analysis of the basic tissue morphology can provide significant predictive information and be combined with clinicopathological and molecular data to improve the accuracy of cancer diagnostics.Koneoppiminen syväoppimisen (SO) muodossa on muuttanut, miten tietokonenäön tehtävät ratkaistaan monilla toimialueilla, kuten lääketieteellisessä kuvantamisdiagnostiikkassa. SO-perusteiset menetelmät mahdollistavat tarkemman kvantitatiivisen karakterisoinnin syöpäkas- vainnäytteistä, jotka rutiinisti analysoidaan kliinisen patologian laboratorioissa diagnosointia varten. Tietokoneavusteinen kudosanalyysi ei rajoitu ainoastaan tiettyjen kudosentiteettien määrittämiseen ja luokitteluun. SO:n avulla voidaan suoraan tutkia syövän ennustetta ja syöpähoitojen vastetta. Tämä väitöskirja keskittyi tärkeään tutkimuskysymykseen: onko syövän ennuste, biomarkke- rien status ja hoidon tehokkuus mahdollista ennustaa SO:lla suoraan kudosmorfologiasta ilman erillisiä värjäyksiä tai molekyylibiologisia testejä? Vastataksemme tähän kysymykseen käytimme digitaalisia hematoksyliini-eosiini (H&E)-värjättyjä kudosnäytteitä kahdesta taval- lisesta kiinteästä kasvaimesta, rinta- ja paksusuolensyövästä. Kudosnäytteet ja niihin liittyvät kliiniset tiedot saatiin Suomessa kerätystä retrospektiivisestä potilassarjasta. Ensimmäiseksi kehitimme SO-algoritmin, jolla poimimme prognostisen tiedon paksusuolensyöpäpotilaista käyttäen ainoastaan digitalisoituja H&E-värjäyksiä. Kudosnäytteistä SO:lla tehty laskennalli- nen analyysi osoitti ihmisasiantuntijaa parempaa suorituskykyä ja ylitti kolmen patologian asiantuntijan antaman yksimielisen viiden vuoden ennusteen syövän lopputulemasta. Seu- raavaksi lopputuleman ennustamista tutkittiin kahdessa erillisessä rintasyöpäpotilassarjassa. Erityisesti tutkimme koulutetun algoritmin kykyä yleistää syöpäkudosten kokoleikkeistä, jotka olivat peräisin erillisestä algoritmille aiemmin tuntemattomasta potilassarjasta. Rin- tasyövän ennusteen suhteen tutkimme ”multitask learning”-lähestymistapaa yhdistämällä eloonjäämis- ja biomarkkeri-valvotun oppimisen. Tutkimuksemme rinta- ja paksusuolen- syövän osalta osoittavat, että SO-mallien avulla, jotka ovat opetettu potilaan eloonjäämisen mukaan, voidaan kudosmorfologian perusteella saada ennuste, joka on rippumaton aiemmin saatavilla olevista ennustetekijöistä, kuten histologisesta luokittelusta, kasvaimen koosta ja imusolmukkeiden statuksesta. Lisäksi SO-perusteisten ennusteiden tarkkuutta rintasyövässä verrattiin patologien arvioimiin syovän, kuten mitoosien lukumäärä, tuman pleomorfismiin, tubulusten tiehyeiden erilaistumisasteeseen, kasvaimen nekroosiin ja kasvaimen infiltroiviin lymfosyytteihin. Tutkimme myös, voiko rintasyöpäpotilailta syöpädiagnosoinnin yhteydessä saaduista H&E-värjätyistä kudosnäytteistä ennustaa molekulaarisia biomarkkereita, kuten hormonireseptoristatusta ja ERBB2-geenin monistumista. Tutkimuksemme osoitti, että mo- lekulaariset muutokset löytyvät myös kudosmorfologiasta ja ne voi tunnistaa SO:n avulla. Lopuksi tutkimme, miten rintasyövän morfologiset piirteet voidaan yhdistää hoitovasteeseen. Tutkimuksemme osoitti, että SO:n tunnistama ERBB2-positiivisen kasvaimen morfologia kor- reloi anti-ERBB2-liitännäishoitojen tehokkuuden kanssa ja SO:ta voi käyttää ennustamaan rintasyövän lääkevastetta. Tämän väitöskirjatyön tulokset osoittavat, että SO:n syöpäkudoksen karakterisointi voi olla hyödyllinen syövän ennusteen arvioinnissa sekä, molekulaarisen statuksen ja lääkevas- teen ennustamisessa. SO-perusteinen kudosmorfologinen analyysi voi antaa merkittävää tietoa syövän ennusteesta ja se voidaan yhdistää kliiniseen patologiaan ja molekulaariseen informaatioon tarkemman syöpädiagnosoinnin mahdollistamiseksi

Task Sensitive Feature Exploration and Learning for Multitask Graph Classification

© 2016 IEEE. Multitask learning (MTL) is commonly used for jointly optimizing multiple learning tasks. To date, all existing MTL methods have been designed for tasks with feature-vector represented instances, but cannot be applied to structure data, such as graphs. More importantly, when carrying out MTL, existing methods mainly focus on exploring overall commonality or disparity between tasks for learning, but cannot explicitly capture task relationships in the feature space, so they are unable to answer important questions, such as what exactly is shared between tasks and what is the uniqueness of one task differing from others? In this paper, we formulate a new multitask graph learning problem, and propose a task sensitive feature exploration and learning algorithm for multitask graph classification. Because graphs do not have features available, we advocate a task sensitive feature exploration and learning paradigm to jointly discover discriminative subgraph features across different tasks. In addition, a feature learning process is carried out to categorize each subgraph feature into one of three categories: 1) common feature; 2) task auxiliary feature; and 3) task specific feature, indicating whether the feature is shared by all tasks, by a subset of tasks, or by only one specific task, respectively. The feature learning and the multiple task learning are iteratively optimized to form a multitask graph classification model with a global optimization goal. Experiments on real-world functional brain analysis and chemical compound categorization demonstrate the algorithm's performance. Results confirm that our method can be used to explicitly capture task correlations and uniqueness in the feature space, and explicitly answer what are shared between tasks and what is the uniqueness of a specific task

Opportunities and obstacles for deep learning in biology and medicine

Deep learning describes a class of machine learning algorithms that are capable of combining raw inputs into layers of intermediate features. These algorithms have recently shown impressive results across a variety of domains. Biology and medicine are data-rich disciplines, but the data are complex and often ill-understood. Hence, deep learning techniques may be particularly well suited to solve problems of these fields. We examine applications of deep learning to a variety of biomedical problems-patient classification, fundamental biological processes and treatment of patients-and discuss whether deep learning will be able to transform these tasks or if the biomedical sphere poses unique challenges. Following from an extensive literature review, we find that deep learning has yet to revolutionize biomedicine or definitively resolve any of the most pressing challenges in the field, but promising advances have been made on the prior state of the art. Even though improvements over previous baselines have been modest in general, the recent progress indicates that deep learning methods will provide valuable means for speeding up or aiding human investigation. Though progress has been made linking a specific neural network\u27s prediction to input features, understanding how users should interpret these models to make testable hypotheses about the system under study remains an open challenge. Furthermore, the limited amount of labelled data for training presents problems in some domains, as do legal and privacy constraints on work with sensitive health records. Nonetheless, we foresee deep learning enabling changes at both bench and bedside with the potential to transform several areas of biology and medicine