Generative Adversarial Network (GAN) for Medical Image Synthesis and Augmentation

Medical image processing aided by artificial intelligence (AI) and machine learning (ML) significantly improves medical diagnosis and decision making. However, the difficulty to access well-annotated medical images becomes one of the main constraints on further improving this technology. Generative adversarial network (GAN) is a DNN framework for data synthetization, which provides a practical solution for medical image augmentation and translation. In this study, we first perform a quantitative survey on the published studies on GAN for medical image processing since 2017. Then a novel adaptive cycle-consistent adversarial network (Ad CycleGAN) is proposed. We respectively use a malaria blood cell dataset (19,578 images) and a COVID-19 chest X-ray dataset (2,347 images) to test the new Ad CycleGAN. The quantitative metrics include mean squared error (MSE), root mean squared error (RMSE), peak signal-to-noise ratio (PSNR), universal image quality index (UIQI), spatial correlation coefficient (SCC), spectral angle mapper (SAM), visual information fidelity (VIF), Frechet inception distance (FID), and the classification accuracy of the synthetic images. The CycleGAN and variant autoencoder (VAE) are also implemented and evaluated as comparison. The experiment results on malaria blood cell images indicate that the Ad CycleGAN generates more valid images compared to CycleGAN or VAE. The synthetic images by Ad CycleGAN or CycleGAN have better quality than those by VAE. The synthetic images by Ad CycleGAN have the highest accuracy of 99.61%. In the experiment on COVID-19 chest X-ray, the synthetic images by Ad CycleGAN or CycleGAN have higher quality than those generated by variant autoencoder (VAE). However, the synthetic images generated through the homogenous image augmentation process have better quality than those synthesized through the image translation process. The synthetic images by Ad CycleGAN have higher accuracy of 95.31% compared to the accuracy of the images by CycleGAN of 93.75%. In conclusion, the proposed Ad CycleGAN provides a new path to synthesize medical images with desired diagnostic or pathological patterns. It is considered a new approach of conditional GAN with effective control power upon the synthetic image domain. The findings offer a new path to improve the deep neural network performance in medical image processing

Fire Pattern Analysis, Junk Science, Old Wives Tales, and Ipse Dixit: Emerging Forensic 3D Imaging Technologies to the Rescue?

Forensic science is undergoing a period of transformation as legal and scientific forces converge and force older forensic sciences toward a new scientific paradigm. Fire investigative undertakings are not an exception to this trend. Skeptical defense attorneys who routinely formulate astute Daubert challenges to contest the scientific validity and reliability of every major forensic science discipline are one catalyst to this revolution. Furthermore, a steady influx of novel scientific advances makes possible the formulation of consistent and scientifically-based quantitative forensic evidence analyses to overcome the “undervalidated and oversold” problems affecting many areas of forensic science

A deep learning algorithm for contour detection in synthetic 2D biplanar X-ray images of the scapula: towards improved 3D reconstruction of the scapula

Three-dimensional (3D) reconstruction from X-ray images using statistical shape models (SSM) provides a cost-effective way of increasing the diagnostic utility of two-dimensional (2D) X-ray images, especially in low-resource settings. The landmark-constrained model fitting approach is one way to obtain patient-specific models from a statistical model. This approach requires an accurate selection of corresponding features, usually landmarks, from the bi-planar X-ray images. However, X-ray images are 2D representations of 3D anatomy with super-positioned structures, which confounds this approach. The literature shows that detection and use of contours to locate corresponding landmarks within biplanar X-ray images can address this limitation. The aim of this research project was to train and validate a deep learning algorithm for detection the contour of a scapula in synthetic 2D bi-planar Xray images. Synthetic bi-planar X-ray images were obtained from scapula mesh samples with annotated landmarks generated from a validated SSM obtained from the Division of Biomedical Engineering, University of Cape Town. This was followed by the training of two convolutional neural network models as the first objective of the project; the first model was trained to predict the lateral (LAT) scapula image given the anterior-posterior (AP) image. The second model was trained to predict the AP image given the LAT image. The trained models had an average Dice coefficient value of 0.926 and 0.964 for the predicted LAT and AP images, respectively. However, the trained models did not generalise to the segmented real X-ray images of the scapula. The second objective was to perform landmark-constrained model fitting using the corresponding landmarks embedded in the predicted images. To achieve this objective, the 2D landmark locations were transformed into 3D coordinates using the direct linear transformation. The 3D point localization yielded average errors of (0.35, 0.64, 0.72) mm in the X, Y and Z directions, respectively, and a combined coordinate error of 1.16 mm. The reconstructed landmarks were used to reconstruct meshes that had average surface-to-surface distances of 3.22 mm and 1.72 mm for 3 and 6 landmarks, respectively. The third objective was to reconstruct the scapula mesh using matching points on the scapula contour in the bi-planar images. The average surface-to-surface distances of the reconstructed meshes with 8 matching contour points and 6 corresponding landmarks of the same meshes were 1.40 and 1.91 mm, respectively. In summary, the deep learning models were able to learn the mapping between the bi-planar images of the scapula. Increasing the number of corresponding landmarks from the bi-planar images resulted into better 3D reconstructions. However, obtaining these corresponding landmarks was non-trivial, necessitating the use of matching points selected from the scapulae contours. The results from the latter approach signal a need to explore contour matching methods to obtain more corresponding points in order to improve the scapula 3D reconstruction using landmark-constrained model fitting

Segmentation and quantification of spinal cord gray matter–white matter structures in magnetic resonance images

This thesis focuses on finding ways to differentiate the gray matter (GM) and white matter (WM) in magnetic resonance (MR) images of the human spinal cord (SC). The aim of this project is to quantify tissue loss in these compartments to study their implications on the progression of multiple sclerosis (MS). To this end, we propose segmentation algorithms that we evaluated on MR images of healthy volunteers. Segmentation of GM and WM in MR images can be done manually by human experts, but manual segmentation is tedious and prone to intra- and inter-rater variability. Therefore, a deterministic automation of this task is necessary. On axial 2D images acquired with a recently proposed MR sequence, called AMIRA, we experiment with various automatic segmentation algorithms. We first use variational model-based segmentation approaches combined with appearance models and later directly apply supervised deep learning to train segmentation networks. Evaluation of the proposed methods shows accurate and precise results, which are on par with manual segmentations. We test the developed deep learning approach on images of conventional MR sequences in the context of a GM segmentation challenge, resulting in superior performance compared to the other competing methods. To further assess the quality of the AMIRA sequence, we apply an already published GM segmentation algorithm to our data, yielding higher accuracy than the same algorithm achieves on images of conventional MR sequences. On a different topic, but related to segmentation, we develop a high-order slice interpolation method to address the large slice distances of images acquired with the AMIRA protocol at different vertebral levels, enabling us to resample our data to intermediate slice positions. From the methodical point of view, this work provides an introduction to computer vision, a mathematically focused perspective on variational segmentation approaches and supervised deep learning, as well as a brief overview of the underlying project's anatomical and medical background

High-quality dense stereo vision for whole body imaging and obesity assessment

textThe prevalence of obesity has necessitated developing safe and convenient tools for timely assessing and monitoring this condition for a broad range of population. Three-dimensional (3D) body imaging has become a new mean for obesity assessment. Moreover, it generates body shape information that is meaningful for fitness, ergonomics, and personalized clothing. In the previous work of our lab, we developed a prototype active stereo vision system that demonstrated a potential to fulfill this goal. But the prototype required four computer projectors to cast artificial textures on the body which facilitate the stereo-matching on texture-deficient images (e.g., skin). This decreases the mobility of the system when used to collect a large population data. In addition, the resolution of the generated 3D~images is limited by both cameras and projectors available during the project. The study reported in this dissertation highlights our continued effort in improving the capability of 3Dbody imaging through simplified hardware for passive stereo and advanced computation techniques. The system utilizes high-resolution single-lens reflex (SLR) cameras, which became widely available lately, and is configured in a two-stance design to image the front and back surfaces of a person. A total of eight cameras are used to form four pairs of stereo units. Each unit covers a quarter of the body surface. The stereo units are individually calibrated with a specific pattern to determine cameras' intrinsic and extrinsic parameters for stereo matching. The global orientation and position of each stereo unit within a common world coordinate system is calculated through a 3Dregistration step. The stereo calibration and 3Dregistration procedures do not need to be repeated for a deployed system if the cameras' relative positions have not changed. This property contributes to the portability of the system, and tremendously alleviates the maintenance task. The image acquisition time is around two seconds for a whole-body capture. The system works in an indoor environment with a moderate ambient light. Advanced stereo computation algorithms are developed by taking advantage of high-resolution images and by tackling the ambiguity problem in stereo matching. A multi-scale, coarse-to-fine matching framework is proposed to match large-scale textures at a low resolution and refine the matched results over higher resolutions. This matching strategy reduces the complexity of the computation and avoids ambiguous matching at the native resolution. The pixel-to-pixel stereo matching algorithm follows a classic, four-step strategy which consists of matching cost computation, cost aggregation, disparity computation and disparity refinement. The system performance has been evaluated on mannequins and human subjects in comparison with other measurement methods. It was found that the geometrical measurements from reconstructed 3Dbody models, including body circumferences and whole volume, are highly repeatable and consistent with manual and other instrumental measurements (CV 0.99). The agreement of percent body fat (%BF) estimation on human subjects between stereo and dual-energy X-ray absorptiometry (DEXA) was found to be improved over the previous active stereo system, and the limits of agreement with 95% confidence were reduced by half. Our achieved %BF estimation agreement is among the lowest ones of other comparative studies with commercialized air displacement plethysmography (ADP) and DEXA. In practice, %BF estimation through a two-component model is sensitive to body volume measurement, and the estimation of lung volume could be a source of variation. Protocols for this type of measurement should still be created with an awareness of this factor.Biomedical Engineerin

Visual attention models and arse representations for morphometrical image analysis

Abstract. Medical diagnosis, treatment, follow-up and research activities are nowadays strongly supported on different types of diagnostic images, whose main goal is to provide an useful exchange of medical knowledge. This multi-modal information needs to be processed in order to extract information exploitable within the context of a particular medical task. In despite of the relevance of these complementary sources of medical knowledge, medical images are rarely further processed in actual clinical practice, so the specialists take decisions only based in the raw data. A new trend in the development of medical image processing and analysis tools follows the idea of biologically-inspired methods, which resemble the performance of the human vision system. Visual attention models and sparse representations are examples of this tendency. Based on this, the aim of this thesis was the development of a set of computational methods for automatic morph metrical analysis, combining the relevant region extraction power of visual attention models with the incorporation of a priori information capabilities of sparse representations. The combination of these biologically inspired tools with common machine learning techniques allowed the identification of visual patterns relevant for pathology discrimination, improving the accuracy and interpretability of morph metric measures and comparisons. After extensive validations with different image data sets, the computational methods proposed in this thesis seems to be promising tools for the definition of anatomical biomarkers, based on visual pattern analysis, and suitable for patient's diagnosis, prognosis and follow-up.Las actividades de diagnóstico, tratamiento, seguimiento e investigación en medicina están actualmente soportadas en diferentes clases de imágenes diagnósticas, cuyo objetivo principal es el de proveer un intercambio efectivo de conocimiento médico. Esta información multimodal necesita ser procesada con el objetivo de extraer información aprovechable en el contexto de una tarea médica particular. A pesar de la relevancia de estas fuentes complementarias de información clínica, las imágenes médicas son raramente procesadas en la práctica clínica actual, de forma que los especialistas sólo toman decisiones basados en los datos crudos. Una nueva tendencia en el desarrollo de herramientas de análisis y procesamiento de imágenes médicas persigue la idea de métodos biológicamente inspirados, que se asemejan al sistema de visión humana. Son ejemplos de esta tendencia los modelos de atención visual y las representaciones escasas (sparse representations). Con base en esto, el objetivo de esta tesis fue el desarrollo de un conjunto de métodos computacionales para soportar automáticamente los análisis morfo métricos, combinando el poder de extracción de regiones relevantes de los modelos de atención visual junto con la capacidad de incorporación de información a priori de las representaciones escasas. La combinación de estos métodos biológicamente inspirados con técnicas de aprendizaje de maquina facilito la identificación de patrones visuales relevantes para discriminar patologías cerebrales, mejorando la precisión e interpretabilidad de las medidas y comparaciones morfo métricas. Después de extensivas validaciones con diferentes conjuntos de imágenes, los métodos computacionales propuestos en esta tesis se perfilan como herramientas prometedoras para la definición de biomarcadores anatómicos, basados en el análisis visual de patrones, y convenientes para el diagnóstico, pronóstico y seguimiento del paciente.Doctorad

Fusion of magnetic resonance and ultrasound images for endometriosis detection

Endometriosis is a gynecologic disorder that typically affects women in their reproductive age and is associated with chronic pelvic pain and infertility. In the context of pre-operative diagnosis and guided surgery, endometriosis is a typical example of pathology that requires the use of both magnetic resonance (MR) and ultrasound (US) modalities. These modalities are used side by sidebecause they contain complementary information. However, MRI and US images have different spatial resolutions, fields of view and contrasts and are corrupted by different kinds of noise, which results in important challenges related to their analysis by radiologists. The fusion of MR and US images is a way of facilitating the task of medical experts and improve the pre-operative diagnosis and the surgery mapping. The object of this PhD thesis is to propose a new automatic fusion method for MRI and US images. First, we assume that the MR and US images to be fused are aligned, i.e., there is no geometric distortion between these images. We propose a fusion method for MR and US images, which aims at combining the advantages of each modality, i.e., good contrast and signal to noise ratio for the MR image and good spatial resolution for the US image. The proposed algorithm is based on an inverse problem, performing a super-resolution of the MR image and a denoising of the US image. A polynomial function is introduced to modelthe relationships between the gray levels of the MR and US images. However, the proposed fusion method is very sensitive to registration errors. Thus, in a second step, we introduce a joint fusion and registration method for MR and US images. Registration is a complicated task in practical applications. The proposed MR/US image fusion performs jointly super-resolution of the MR image and despeckling of the US image, and is able to automatically account for registration errors. A polynomial function is used to link ultrasound and MR images in the fusion process while an appropriate similarity measure is introduced to handle the registration problem. The proposed registration is based on a non-rigid transformation containing a local elastic B-spline model and a global affine transformation. The fusion and registration operations are performed alternatively simplifying the underlying optimization problem. The interest of the joint fusion and registration is analyzed using synthetic and experimental phantom images