Prototypes for Content-Based Image Retrieval in Clinical Practice

Content-based image retrieval (CBIR) has been proposed as key technology for computer-aided diagnostics (CAD). This paper reviews the state of the art and future challenges in CBIR for CAD applied to clinical practice

A graph-based approach for the retrieval of multi-modality medical images

Medical imaging has revolutionised modern medicine and is now an integral aspect of diagnosis and patient monitoring. The development of new imaging devices for a wide variety of clinical cases has spurred an increase in the data volume acquired in hospitals. These large data collections offer opportunities for search-based applications in evidence-based diagnosis, education, and biomedical research. However, conventional search methods that operate upon manual annotations are not feasible for this data volume. Content-based image retrieval (CBIR) is an image search technique that uses automatically derived visual features as search criteria and has demonstrable clinical benefits. However, very few studies have investigated the CBIR of multi-modality medical images, which are making a monumental impact in healthcare, e.g., combined positron emission tomography and computed tomography (PET-CT) for cancer diagnosis. In this thesis, we propose a new graph-based method for the CBIR of multi-modality medical images. We derive a graph representation that emphasises the spatial relationships between modalities by structurally constraining the graph based on image features, e.g., spatial proximity of tumours and organs. We also introduce a graph similarity calculation algorithm that prioritises the relationships between tumours and related organs. To enable effective human interpretation of retrieved multi-modality images, we also present a user interface that displays graph abstractions alongside complex multi-modality images. Our results demonstrated that our method achieved a high precision when retrieving images on the basis of tumour location within organs. The evaluation of our proposed UI design by user surveys revealed that it improved the ability of users to interpret and understand the similarity between retrieved PET-CT images. The work in this thesis advances the state-of-the-art by enabling a novel approach for the retrieval of multi-modality medical images

Semantic Assisted, Multiresolution Image Retrieval in 3D Brain MR Volumes

Content Based Image Retrieval (CBIR) is an important research area in the field of multimedia information retrieval. The application of CBIR in the medical domain has been attempted before, however the use of CBIR in medical diagnostics is a daunting task. The goal of diagnostic medical image retrieval is to provide diagnostic support by displaying relevant past cases, along with proven pathologies as ground truths. Moreover, medical image retrieval can be extremely useful as a training tool for medical students and residents, follow-up studies, and for research purposes. Despite the presence of an impressive amount of research in the area of CBIR, its acceptance for mainstream and practical applications is quite limited. The research in CBIR has mostly been conducted as an academic pursuit, rather than for providing the solution to a need. For example, many researchers proposed CBIR systems where the image database consists of images belonging to a heterogeneous mixture of man-made objects and natural scenes while ignoring the practical uses of such systems. Furthermore, the intended use of CBIR systems is important in addressing the problem of "Semantic Gap". Indeed, the requirements for the semantics in an image retrieval system for pathological applications are quite different from those intended for training and education. Moreover, many researchers have underestimated the level of accuracy required for a useful and practical image retrieval system. The human eye is extremely dexterous and efficient in visual information processing; consequently, CBIR systems should be highly precise in image retrieval so as to be useful to human users. Unsurprisingly, due to these and other reasons, most of the proposed systems have not found useful real world applications. In this dissertation, an attempt is made to address the challenging problem of developing a retrieval system for medical diagnostics applications. More specifically, a system for semantic retrieval of Magnetic Resonance (MR) images in 3D brain volumes is proposed. The proposed retrieval system has a potential to be useful for clinical experts where the human eye may fail. Previously proposed systems used imprecise segmentation and feature extraction techniques, which are not suitable for precise matching requirements of the image retrieval in this application domain. This dissertation uses multiscale representation for image retrieval, which is robust against noise and MR inhomogeneity. In order to achieve a higher degree of accuracy in the presence of misalignments, an image registration based retrieval framework is developed. Additionally, to speed-up the retrieval system, a fast discrete wavelet based feature space is proposed. Further improvement in speed is achieved by semantically classifying of the human brain into various "Semantic Regions", using an SVM based machine learning approach. A novel and fast identification system is proposed for identifying a 3D volume given a 2D image slice. To this end, we used SVM output probabilities for ranking and identification of patient volumes. The proposed retrieval systems are tested not only for noise conditions but also for healthy and abnormal cases, resulting in promising retrieval performance with respect to multi-modality, accuracy, speed and robustness. This dissertation furnishes medical practitioners with a valuable set of tools for semantic retrieval of 2D images, where the human eye may fail. Specifically, the proposed retrieval algorithms provide medical practitioners with the ability to retrieve 2D MR brain images accurately and monitor the disease progression in various lobes of the human brain, with the capability to monitor the disease progression in multiple patients simultaneously. Additionally, the proposed semantic classification scheme can be extremely useful for semantic based categorization, clustering and annotation of images in MR brain databases. This research framework may evolve in a natural progression towards developing more powerful and robust retrieval systems. It also provides a foundation to researchers in semantic based retrieval systems on how to expand existing toolsets for solving retrieval problems

WiFi-Based Human Activity Recognition Using Attention-Based BiLSTM

Recently, significant efforts have been made to explore human activity recognition (HAR) techniques that use information gathered by existing indoor wireless infrastructures through WiFi signals without demanding the monitored subject to carry a dedicated device. The key intuition is that different activities introduce different multi-paths in WiFi signals and generate different patterns in the time series of channel state information (CSI). In this paper, we propose and evaluate a full pipeline for a CSI-based human activity recognition framework for 12 activities in three different spatial environments using two deep learning models: ABiLSTM and CNN-ABiLSTM. Evaluation experiments have demonstrated that the proposed models outperform state-of-the-art models. Also, the experiments show that the proposed models can be applied to other environments with different configurations, albeit with some caveats. The proposed ABiLSTM model achieves an overall accuracy of 94.03%, 91.96%, and 92.59% across the 3 target environments. While the proposed CNN-ABiLSTM model reaches an accuracy of 98.54%, 94.25% and 95.09% across those same environments

Caracterización de Patrones Anormales en Mamografías

Abstract. Computer-guided image interpretation is an extensive research area whose main purpose is to provide tools to support decision-making, for which a large number of automatic techniques have been proposed, such as, feature extraction, pattern recognition, image processing, machine learning, among others. In breast cancer, the results obtained at this area, they have led to the development of diagnostic support systems, which have even been approved by the FDA (Federal Drug Administration). However, the use of those systems is not widely extended in clinic scenarios, mainly because their performance is unstable and poorly reproducible. This is due to the high variability of the abnormal patterns associated with this neoplasia. This thesis addresses the main problem associated with the characterization and interpretation of breast masses and architectural distortion, mammographic findings directly related to the presence of breast cancer with higher variability in their form, size and location. This document introduces the design, implementation and evaluation of strategies to characterize abnormal patterns and to improve the mammographic interpretation during the diagnosis process. The herein proposed strategies allow to characterize visual patterns of these lesions and the relationship between them to infer their clinical significance according to BI-RADS (Breast Imaging Reporting and Data System), a radiologic tool used for mammographic evaluation and reporting. The obtained results outperform some obtained by methods reported in the literature both tasks classification and interpretation of masses and architectural distortion, respectively, demonstrating the effectiveness and versatility of the proposed strategies.Resumen. La interpretación de imágenes guiada por computador es una área extensa de investigación cuyo objetivo principal es proporcionar herramientas para el soporte a la toma de decisiones, para lo cual se han usado un gran número de técnicas de extracción de características, reconocimiento de patrones, procesamiento de imágenes, aprendizaje de máquina, entre otras. En el cáncer de mama, los resultados obtenidos en esta área han dado lugar al desarrollo de sistemas de apoyo al diagnóstico que han sido incluso aprobados por la FDA (Federal Drug Administration). Sin embargo, el uso de estos sistemas no es ampliamente extendido, debido principalmente, a que su desempeño resulta inestable y poco reproducible frente a la alta variabilidad de los patrones anormales asociados a esta neoplasia. Esta tesis trata el principal problema asociado a la caracterización y análisis de masas y distorsión de la arquitectura debido a que son hallazgos directamente relacionados con la presencia de cáncer y que usualmente presentan mayor variabilidad en su forma, tamaño y localización, lo que altera los resultados diagnósticos. Este documento introduce el diseño, implementación y evaluación de un conjunto de estrategias para caracterizar patrones anormales relacionados con este tipo de hallazgos para mejorar la interpretación y soportar el diagnóstico mediante la imagen mamaria. Los modelos aquí propuestos permiten caracterizar patrones visuales y la relación entre estos para inferir su significado clínico según el estándar BI-RADS (Breast Imaging Reporting and Data System) usado para la evaluación y reporte mamográfico. Los resultados obtenidos han demostrado mejorar a los resultados obtenidos por los métodos reportados en la literatura en tareas como clasificación e interpretación de masas y distorsión arquitectural, demostrando la efectividad y versatilidad de las estrategia propuestas.Doctorad

Recuperação de informação multimodal em repositórios de imagem médica

The proliferation of digital medical imaging modalities in hospitals and other diagnostic facilities has created huge repositories of valuable data, often not fully explored. Moreover, the past few years show a growing trend of data production. As such, studying new ways to index, process and retrieve medical images becomes an important subject to be addressed by the wider community of radiologists, scientists and engineers. Content-based image retrieval, which encompasses various methods, can exploit the visual information of a medical imaging archive, and is known to be beneficial to practitioners and researchers. However, the integration of the latest systems for medical image retrieval into clinical workflows is still rare, and their effectiveness still show room for improvement. This thesis proposes solutions and methods for multimodal information retrieval, in the context of medical imaging repositories. The major contributions are a search engine for medical imaging studies supporting multimodal queries in an extensible archive; a framework for automated labeling of medical images for content discovery; and an assessment and proposal of feature learning techniques for concept detection from medical images, exhibiting greater potential than feature extraction algorithms that were pertinently used in similar tasks. These contributions, each in their own dimension, seek to narrow the scientific and technical gap towards the development and adoption of novel multimodal medical image retrieval systems, to ultimately become part of the workflows of medical practitioners, teachers, and researchers in healthcare.A proliferação de modalidades de imagem médica digital, em hospitais, clínicas e outros centros de diagnóstico, levou à criação de enormes repositórios de dados, frequentemente não explorados na sua totalidade. Além disso, os últimos anos revelam, claramente, uma tendência para o crescimento da produção de dados. Portanto, torna-se importante estudar novas maneiras de indexar, processar e recuperar imagens médicas, por parte da comunidade alargada de radiologistas, cientistas e engenheiros. A recuperação de imagens baseada em conteúdo, que envolve uma grande variedade de métodos, permite a exploração da informação visual num arquivo de imagem médica, o que traz benefícios para os médicos e investigadores. Contudo, a integração destas soluções nos fluxos de trabalho é ainda rara e a eficácia dos mais recentes sistemas de recuperação de imagem médica pode ser melhorada. A presente tese propõe soluções e métodos para recuperação de informação multimodal, no contexto de repositórios de imagem médica. As contribuições principais são as seguintes: um motor de pesquisa para estudos de imagem médica com suporte a pesquisas multimodais num arquivo extensível; uma estrutura para a anotação automática de imagens; e uma avaliação e proposta de técnicas de representation learning para deteção automática de conceitos em imagens médicas, exibindo maior potencial do que as técnicas de extração de features visuais outrora pertinentes em tarefas semelhantes. Estas contribuições procuram reduzir as dificuldades técnicas e científicas para o desenvolvimento e adoção de sistemas modernos de recuperação de imagem médica multimodal, de modo a que estes façam finalmente parte das ferramentas típicas dos profissionais, professores e investigadores da área da saúde.Programa Doutoral em Informátic

Computer-aided image quality assessment in automated 3D breast ultrasound images

Automated 3D breast ultrasound (ABUS) is a valuable, non-ionising adjunct to X-ray mammography for breast cancer screening and diagnosis for women with dense breasts. High image quality is an important prerequisite for diagnosis and has to be guaranteed at the time of acquisition. The high throughput of images in a screening scenario demands for automated solutions. In this work, an automated image quality assessment system rating ABUS scans at the time of acquisition was designed and implemented. Quality assessment of present diagnostic ultrasound images has rarely been performed demanding thorough analysis of potential image quality aspects in ABUS. Therefore, a reader study was initiated, making two clinicians rate the quality of clinical ABUS images. The frequency of specific quality aspects was evaluated revealing that incorrect positioning and insufficiently applied contact fluid caused the most relevant image quality issues. The relative position of the nipple in the image, the acoustic shadow caused by the nipple as well as the shape of the breast contour reflect patient positioning and ultrasound transducer handling. Morphological and histogram-based features utilized for machine learning to reproduce the manual classification as provided by the clinicians. At 97 % specificity, the automatic classification achieved sensitivities of 59 %, 45 %, and 46 % for the three aforementioned aspects, respectively. The nipple is an important landmark in breast imaging, which is generally---but not always correctly---pinpointed by the technicians. An existing nipple detection algorithm was extended by probabilistic atlases and exploited for automatic detection of incorrectly annotated nipple marks. The nipple detection rate was increased from 82 % to 85 % and the classification achieved 90 % sensitivity at 89 % specificity. A lack of contact fluid between transducer and skin can induce reverberation patterns and acoustic shadows, which can possibly obscure lesions. Parameter maps were computed in order to localize these artefact regions and yielded a detection rate of 83 % at 2.6 false positives per image. Parts of the presented work were integrated to clinical workflow making up a novel image quality assessment system that supported technicians in their daily routine by detecting images of insufficient quality and indicating potential improvements for a repeated scan while the patient was still in the examination room. First evaluations showed that the proposed method sensitises technicians for the radiologists' demands on diagnostically valuable images

Cloud-Based Benchmarking of Medical Image Analysis

Medical imagin