Multi-modal brain tumor segmentation via conditional synthesis with Fourier domain adaptation

Accurate brain tumor segmentation is critical for diagnosis and treatment planning, whereby multi-modal magnetic resonance imaging (MRI) is typically used for analysis. However, obtaining all required sequences and expertly labeled data for training is challenging and can result in decreased quality of segmentation models developed through automated algorithms. In this work, we examine the possibility of employing a conditional generative adversarial network (GAN) approach for synthesizing multi-modal images to train deep learning-based neural networks aimed at high-grade glioma (HGG) segmentation. The proposed GAN is conditioned on auxiliary brain tissue and tumor segmentation masks, allowing us to attain better accuracy and control of tissue appearance during synthesis. To reduce the domain shift between synthetic and real MR images, we additionally adapt the low-frequency Fourier space components of synthetic data, reflecting the style of the image, to those of real data. We demonstrate the impact of Fourier domain adaptation (FDA) on the training of 3D segmentation networks and attain significant improvements in both the segmentation performance and prediction confidence. Similar outcomes are seen when such data is used as a training augmentation alongside the available real images. In fact, experiments on the BraTS2020 dataset reveal that models trained solely with synthetic data exhibit an improvement of up to 4% in Dice score when using FDA, while training with both real and FDA-processed synthetic data through augmentation results in an improvement of up to 5% in Dice compared to using real data alone. This study highlights the importance of considering image frequency in generative approaches for medical image synthesis and offers a promising approach to address data scarcity in medical imaging segmentation.</p

An evolution of near surface geophysical imaging : directionality, physical properties and challenging conventional wisdom

The fundamental changes in applied geophysics in the last few decades have to a great extent been in the development of near-surface geophysics (NSG) – what used to be called environmental and engineering geophysics. In some locations and for some purposes, it still is. The developments in my publications have, to some extent, paralleled and sometimes foreshadowed some significant developments. My earliest papers were on marine electromagnetic (EM) sounding, and some of those papers are still cited. The work from my PhD and my post-doctoral fellowship laid the groundwork for what was to become controlled source EM (CSEM), a technique of growing importance in marine oil and gas exploration. Because the depths involved were less than 1 km, it can still perhaps be called “near surface”, but that was not the original intention. However, a theme central to that early work has carried on, explicitly or implicitly, through much of my research – anisotropy and the directionality of the geophysical response. One of my early theoretical papers was on the inclusion of anisotropy in Maxwell’s equations, and recent papers have used the directionality of the EM response as a tool in archaeological imaging. Another pair of linked themes that have recurred almost from the beginning are the influence of physical properties on the geophysical response, and the inter-relationships of physical properties. It allowed me to determine the physical property variations at depth in Middle Valley, on the northern Juan de Fuca Ridge. Those predictions were confirmed by the results from Ocean Drilling Program (ODP) Leg 139, which drilled those Middle Valley sites. While the marine research was interesting and rewarding, I was also moving more and more onshore, and began doing archaeological imaging in the late 1980’s. Much of that work was focussed around student projects, but then expanded into forensic geoscience, and ultimately to the non-invasive imaging of burial sites. That work continues today. The onshore research also allowed me to move from EM induction methods into ground penetrating radar (GPR), which involved the propagation of high-frequency EM waves. There are many hypotheses and approaches to GPR that were based on incorrect assumptions. For example, it was often assumed that rocky debris in debris-covered and debris-laden glaciers would not prevent the propagation of significant GPR energy at depth, an assumption that we proved wrong. The publication from 1994 on GPR imaging of the debris-covered lower Tasman Glacier was not followed by a paper by other researchers on GPR imaging of debris-laden glaciers until 1997, and GPR is now a common technique for imaging of all types of glaciers. Thus glacier imaging has been an ongoing application, and has expanded to include imaging of permafrost, including 4-dimensional (4D) imaging, i.e. time lapse 3-dimensional (3D) imaging, of permafrost polygonal patterned ground (PPG) in the Dry Valleys of Antarctica. The utility of near-surface geophysics in Antarctica has expanded greatly over the years. Similarly, surface water was assumed to degrade GPR signal penetration. Again, this was based on an incorrect assumption – that water was inherently conductive. While the presence of water does increase the electrical conductivity, if the water is fresh then the conductivity still remains quite low, and the attenuation of the GPR signal is minimal. 6 Thus the applications for EM and GPR have expanded, and the principles and applications are better understood now, ranging from archaeological and forensic geoscience, through non-destructive testing (NDT) and other geotechnical projects, to neotectonics and the imaging of active faults. Recently, I and my students have combined GPR more and more with electrical imaging. The two complement each other nicely. Finally, I have included two review papers, each in the section of greatest relevance. I recognise that this is not standard practice, but one from 1996 was used as a benchmark and a starting point for the later reviews of the environmental applications of EM, and the other from 2011 provides what I hope will be a paper used to help glacier imaging surveys to be better designed and completed. Both also include recent research results that had yet to be published, and thus represented the state of the art. I would note that I have included a number of papers from conference proceedings. In applied geophysics, the conference papers are normally peer reviewed, just as in engineering. Sometimes those papers are then expanded and augmented and subsequently published in peer-reviewed journals. If the peer-reviewed conference papers were later published as peerreviewed journal articles, then the journal article is included here. There are papers I decided not to include because they did not fit into the overall theme of this collection of papers – the evolution of my work in near-surface geophysics, which I took very broadly to embrace my work in marine geophysics as well. The papers not included here were two papers on paleoclimatology, for which I did the crucial spectral analysis, and three papers on social science and philosophy of science. I also excluded a few papers that were superseded by later work. There appears to be no set configuration to the form of a DSc, beyond collecting the papers together into some sort of coherent form that reflects the themes the work represents. In principle, a collection of papers submitted for the DSc represents the best of a lifetime of work. However, I hope that my best work is still to come. Only time will tell if that is true. For the papers submitted here, I have done a significant amount of the work, if not the majority of the work. In the case of papers based on student projects that I supervised, if the student wrote the first draft, then I made them first author, regardless of how much additional work was required to get the paper to its published form

Cardiac displacement tracking with data assimilation combining a biomechanical model and an automatic contour detection

International audienceData assimilation in computational models represents an essential step in building patient-specific simulations. This work aims at circumventing one major bottleneck in the practical use of data assimilation strategies in cardiac applications, namely, the difficulty of formulating and effectively computing adequate data-fitting term for cardiac imaging such as cine MRI. We here provide a proof-of-concept study of data assimilation based on automatic contour detection. The tissue motion simulated by the data assimilation framework is then assessed with displacements extracted from tagged MRI in six subjects, and the results illustrate the performance of the proposed method, including for circumferential displacements, which are not well extracted from cine MRI alone

A Framework for Simulating Cardiac MR Images with Varying Anatomy and Contrast

One of the limiting factors for the development and adoption of novel deep-learning (DL) based medical image analysis methods is the scarcity of labeled medical images. Medical image simulation and synthesis can provide solutions by generating ample training data with corresponding ground truth labels. Despite recent advances, generated images demonstrate limited realism and diversity. In this work, we develop a flexible framework for simulating cardiac magnetic resonance (MR) images with variable anatomical and imaging characteristics for the purpose of creating a diversified virtual population. We advance previous works on both cardiac MR image simulation and anatomical modeling to increase the realism in terms of both image appearance and underlying anatomy. To diversify the generated images, we define parameters: 1) to alter the anatomy, 2) to assign MR tissue properties to various tissue types, and 3) to manipulate the image contrast via acquisition parameters. The proposed framework is optimized to generate a substantial number of cardiac MR images with ground truth labels suitable for downstream supervised tasks. A database of virtual subjects is simulated and its usefulness for aiding a DL segmentation method is evaluated. Our experiments show that training completely with simulated images can perform comparable with a model trained with real images for heart cavity segmentation in mid-ventricular slices. Moreover, such data can be used in addition to classical augmentation for boosting the performance when training data is limited, particularly by increasing the contrast and anatomical variation, leading to better regularization and generalization. The database is publicly available at https://osf.io/ bkzhm/ and the simulation code will be available at https: //github.com/sinaamirrajab/CMRI_Simulation

Reducing segmentation failures in cardiac MRI via late feature fusion and GAN-based augmentation

Cardiac magnetic resonance (CMR) image segmentation is an integral step in the analysis of cardiac function and diagnosis of heart related diseases. While recent deep learning-based approaches in automatic segmentation have shown great promise to alleviate the need for manual segmentation, most of these are not applicable to realistic clinical scenarios. This is largely due to training on mainly homogeneous datasets, without variation in acquisition, which typically occurs in multi-vendor and multi-site settings, as well as pathological data. Such approaches frequently exhibit a degradation in prediction performance, particularly on outlier cases commonly associated with difficult pathologies, artifacts and extensive changes in tissue shape and appearance. In this work, we present a model aimed at segmenting all three cardiac structures in a multi-center, multi-disease and multi-view scenario. We propose a pipeline, addressing different challenges with segmentation of such heterogeneous data, consisting of heart region detection, augmentation through image synthesis and a late-fusion segmentation approach. Extensive experiments and analysis demonstrate the ability of the proposed approach to tackle the presence of outlier cases during both training and testing, allowing for better adaptation to unseen and difficult examples. Overall, we show that the effective reduction of segmentation failures on outlier cases has a positive impact on not only the average segmentation performance, but also on the estimation of clinical parameters, leading to a better consistency in derived metrics

A Stratified Cascaded Approach for Brain Tumor Segmentation with the Aid of Multi-modal Synthetic Data

Gliomas are one of the most widespread and aggressive forms of brain tumors. Accurate brain tumor segmentation is crucial for evaluation, monitoring and treatment of gliomas. Recent advances in deep learning methods have made a significant step towards a robust and automated brain tumor segmentation. However, due to the variation in shape and location of gliomas, as well as their appearance across different tumor grades, obtaining an accurate and generalizable segmentation model is still a challenge. To alleviate this, we propose a cascaded segmentation pipeline, aimed at introducing more robustness to segmentation performance through data stratification. In other words, we train separate models per tumor grade, aided with synthetic brain tumor images generated through conditional generative adversarial networks. To handle the variety in size, shape and location of tumors, we utilize a localization module, focusing the training and inference in the vicinity of the tumor. Finally, to identify which tumor grade segmentation model to utilize at inference time, we train a dense, attention-based 3D classification model. The obtained results suggest that both stratification and the addition of synthetic data to training significantly improve the segmentation performance, whereby up to 55% of test cases exhibit a performance improvement by more than 5% and up to 40% of test cases exhibit an improvement by more than 10% in Dice score.</p

On the usability of synthetic data for improving the robustness of deep learning-based segmentation of cardiac magnetic resonance images

Deep learning-based segmentation methods provide an effective and automated way for assessing the structure and function of the heart in cardiac magnetic resonance (CMR) images. However, despite their state-of-the-art performance on images acquired from the same source (same scanner or scanner vendor) as images used during training, their performance degrades significantly on images coming from different domains. A straightforward approach to tackle this issue consists of acquiring large quantities of multi-site and multi-vendor data, which is practically infeasible. Generative adversarial networks (GANs) for image synthesis present a promising solution for tackling data limitations in medical imaging and addressing the generalization capability of segmentation models. In this work, we explore the usability of synthesized short-axis CMR images generated using a segmentation-informed conditional GAN, to improve the robustness of heart cavity segmentation models in a variety of different settings. The GAN is trained on paired real images and corresponding segmentation maps belonging to both the heart and the surrounding tissue, reinforcing the synthesis of semantically-consistent and realistic images. First, we evaluate the segmentation performance of a model trained solely with synthetic data and show that it only slightly underperforms compared to the baseline trained with real data. By further combining real with synthetic data during training, we observe a substantial improvement in segmentation performance (up to 4% and 40% in terms of Dice score and Hausdorff distance) across multiple data-sets collected from various sites and scanner. This is additionally demonstrated across state-of-the-art 2D and 3D segmentation networks, whereby the obtained results demonstrate the potential of the proposed method in tackling the presence of the domain shift in medical data. Finally, we thoroughly analyze the quality of synthetic data and its ability to replace real MR images during training, as well as provide an insight into important aspects of utilizing synthetic images for segmentation