Bimodal network architectures for automatic generation of image annotation from text

Medical image analysis practitioners have embraced big data methodologies. This has created a need for large annotated datasets. The source of big data is typically large image collections and clinical reports recorded for these images. In many cases, however, building algorithms aimed at segmentation and detection of disease requires a training dataset with markings of the areas of interest on the image that match with the described anomalies. This process of annotation is expensive and needs the involvement of clinicians. In this work we propose two separate deep neural network architectures for automatic marking of a region of interest (ROI) on the image best representing a finding location, given a textual report or a set of keywords. One architecture consists of LSTM and CNN components and is trained end to end with images, matching text, and markings of ROIs for those images. The output layer estimates the coordinates of the vertices of a polygonal region. The second architecture uses a network pre-trained on a large dataset of the same image types for learning feature representations of the findings of interest. We show that for a variety of findings from chest X-ray images, both proposed architectures learn to estimate the ROI, as validated by clinical annotations. There is a clear advantage obtained from the architecture with pre-trained imaging network. The centroids of the ROIs marked by this network were on average at a distance equivalent to 5.1% of the image width from the centroids of the ground truth ROIs.Comment: Accepted to MICCAI 2018, LNCS 1107

Learning Cross-Modality Representations from Multi-Modal Images

Machine learning algorithms can have difficulties adapting to data from different sources, for example from different imaging modalities. We present and analyze three techniques for unsupervised cross-modality feature learning, using a shared autoencoder-like convolutional network that learns a common representation from multi-modal data. We investigate a form of feature normalization, a learning objective that minimizes crossmodality differences, and modality dropout, in which the network is trained with varying subsets of modalities. We measure the same-modality and cross-modality classification accuracies and explore whether the models learn modality-specific or shared features. This paper presents experiments on two public datasets, with knee images from two MRI modalities, provided by the Osteoarthritis Initiative, and brain tumor segmentation on four MRI modalities from the BRATS challenge. All three approaches improved the cross-modality classification accuracy, with modality dropout and per-feature normalization giving the largest improvement. We observed that the networks tend to learn a combination of cross-modality and modality-specific features. Overall, a combination of all three methods produced the most cross-modality features and the highest cross-modality classification accuracy, while maintaining most of the same-modality accuracy

Deep Learning in Cardiology

The medical field is creating large amount of data that physicians are unable to decipher and use efficiently. Moreover, rule-based expert systems are inefficient in solving complicated medical tasks or for creating insights using big data. Deep learning has emerged as a more accurate and effective technology in a wide range of medical problems such as diagnosis, prediction and intervention. Deep learning is a representation learning method that consists of layers that transform the data non-linearly, thus, revealing hierarchical relationships and structures. In this review we survey deep learning application papers that use structured data, signal and imaging modalities from cardiology. We discuss the advantages and limitations of applying deep learning in cardiology that also apply in medicine in general, while proposing certain directions as the most viable for clinical use.Comment: 27 pages, 2 figures, 10 table

A Survey on Deep Learning in Medical Image Analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.Comment: Revised survey includes expanded discussion section and reworked introductory section on common deep architectures. Added missed papers from before Feb 1st 201

Sparsely Activated Networks: A new method for decomposing and compressing data

Recent literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features, but without considering the description length of the representations. In this thesis, first we introduce the{\phi}metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined metric $\varphi$. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using $\varphi$ have small description representation length and consist of interpretable kernels.Comment: PhD Thesis in Greek, 158 pages for the main text, 23 supplementary pages for presentation, arXiv:1907.06592, arXiv:1904.13216, arXiv:1902.1112

Radiographic Assessment of Hip Disease in Children with Cerebral Palsy: Development of a Core Measurement Set and Analysis of an Artificial Intelligence System

Cerebral palsy is the most common physical disability during childhood. Cerebral palsy related hip disease is caused by an imbalance of muscle forces, resulting in progressive migration of the hip to complete dislocation. This can decrease function and quality of life. The prevention of hip dislocation is possible if detected early. Therefore, surveillance programmes have been set up to monitor children with cerebral palsy enabling clinicians to intervene early and improve outcomes. Currently, hip disease is assessed by analysing pelvic radiographs with various geometric measurements. This time-consuming task is undertaken frequently when monitoring a child with cerebral palsy. This thesis aimed to identify the key radiographic parameters used by clinicians (the core measurement set), and then build an artificial intelligence system to automate the calculation of this core measurement set. A systematic review was conducted identifying a comprehensive list of previously reported measurements from studies measuring radiographic outcomes in cerebral palsy children with hip pathologies. Fifteen measurements were identified from the systematic review, of which Reimers’ migration percentage was the most commonly reported. These measurements were used to perform a two-round Delphi study among orthopaedic surgeons and physiotherapists. Participants rated the importance of each measurement using a nine-point Likert scale (‘not important’ to critically important’). After the two rounds of the Delphi process, Reimers’ migration percentage was included in the core measurement set. Following the final consensus meeting, the femoral head-shaft angle was also included. The anteroposterior pelvic radiographs of 1650 children were then used to build an artificial intelligence system integrating the core measurement set, in collaboration with engineers from the University of Manchester. The newly developed artificial intelligence system was assessed by comparing its ability to calculate measurements and outline the pelvis and femur on a radiograph. The reliability of the dataset used to train the model was also analysed. The proposed artificial intelligence model achieved a ‘good to excellent’ inter-observer reliability across 450 radiographs when comparing its ability to calculate Reimers’ migration percentage to five clinicians. Its ability to outline the pelvis and proximal femur was ‘adequate’ with the better performance observed in the pelvis than the femur. The reliability of the training dataset used to teach the artificial intelligence model was ‘good’ to ‘very good’. Artificial intelligence systems are feasible solutions to optimise the efficiency of hip radiograph analysis in cerebral palsy. Studies are warranted to include the core measurement set as a minimum when reporting on hip disease in cerebral palsy. Future research should investigate the feasibility of implementing a risk score to predict the likelihood of hip displacement