뇌의 미·거시적 특성을 이용한 자기공명영상의 딥러닝 이미지 간 변환 기법

학위논문(석사) -- 서울대학교대학원 : 자연과학대학 뇌인지과학과, 2023. 8. 차지욱.뇌는 여러 뇌 영역의 고도로 국소화된 기능과 신경 연결을 통한 영역의 통합으로 구성된다. 이러한 뇌의 신경 연결은 끊임없이 변화하는 환경에 효과적으로 대응하기 위해 시스템 및 시냅스 수준에서 지속적으로 변화한다. 이러한 동적 상호작용을 가능하게 하는 중요한 요소 중 하나는 인간 두뇌의 거시적 및 미시적 규모에서의 구조적 가소성이다. 뇌의 거시적 및 미세적 구조는 서로 다르지만 보완적인 정보를 전달하기 때문에 두 구조를 모두 고려하는 것은 뇌의 구조적 가소성과 인지 작업 중 연결성을 이해하는데 매우 중요하다. 하지만 기존의 연구는 이를 효과적으로 고려하지 못했다. 본 연구에선 인간 뇌의 거시적 및 미세적 구조를 통합하여 기존에 알지 못했던 인간 뇌의 의미와 새로운 표현형을 얻기 위해 구조 MRI (sMRI)에서 고품질 확산 텐서 이미징 (DTI) 및 트랙토그래피를 생성하도록 설계된 새로운 딥러닝 프레임워크인 Macro2Micro를 제시한다. 본 연구는 거시적 구조로부터 미시적 구조 정보를 유추할 수 있다는 가설을 전제로 하여, 초기에 한 가지 영상 기법만 획득하더라도 질병 진단 및 연구에 유익한 추가적인 영상 기법을 생성한다. 신경 영상 영역에서 전례가 없는 이 접근 방식은 멀티모달 이미지 번역의 이점을 활용하여 상당한 시간과 비용을 절감한다. Macro2Micro는 3D T1을 입력으로 사용하여 2D T1 슬라이스를 생성한 다음 적대적 생성망(GAN)을 통해 처리되어 2D DTI (FA) 슬라이스와 2D 트랙토그래피 이미지를 생성한다. 이 프로세스의 핵심 요소는 이미지 특성을 주파수 대역에 따라 분리하는 옥타브 합성곱을 사용하는 것이다. 이 프레임워크는 청소년 뇌 인지 발달(ABCD) 데이터를 사용하여 훈련되었으며, 이미지 픽셀 손실, 지각 손실, GAN 손실 및 뇌 중심 패치 GAN 손실을 통해 훈련 손실이 정의되었다. 본 연구의 결과는 정량적 및 정성적으로 뛰어난 성능을 보였으며 단순히 이미지 분포만을 학습한 것이 아닌 뇌의 구조적 및 미시적 구조의 생물학적인 특성까지 학습했다는 점에서 의의가 있다. 이미지 변환 모델을 데이터 증대 방법으로 잠재적으로 적용하면 데이터 불균형 및 희소성 문제를 해결할 수 있다. 이 연구는 발전하는 질병 모델링에서 다중 모드 이미징, 특히 T1, DTI 및 트랙토그래피의 조합 사용의 잠재력을 제시한다.The brain consists of the highly localized functions of several brain regions and the integration of these regions through neural connections. These brain neural connections are constantly changing at the systemic and synaptic levels to effectively respond to the ever-changing environment. One of the key factors enabling these dynamic interactions is the structural plasticity of the human brain at the macro and micro scale. Because the brain's macro- and micro-structures convey different but complementary information, considering both structures is critical to understanding the brain's structural plasticity and connectivity during cognitive tasks. However, previous studies have not effectively considered this issue. In this study, a novel deep learning framework, Macro2Micro, is proposed to generate high-quality Diffusion Tensor Imaging (DTI) and tractography from structural MRI (sMRI). The study is premised on the hypothesis that micro-scale structural information can be inferred from macro-scale structures, enabling the generation of different imaging modalities beneficial for disease diagnosis and research, even when only one modality is initially obtained. This approach, unprecedented in the realm of neuroimaging, leverages the benefits of cross-modality image translation, offering significant time and cost savings. The Macro2Micro framework utilizes 3D T1 to generate 2D T1 slices as input, which are then processed through a Generative Adversarial Network (GAN) to produce 2D DTI (FA) slices and subsequently 2D tractography. The key element of this process is the use of Octave Convolutions, which facilitate the analysis of connections between various scale MR modalities. The framework was trained using the Adolescent Brain Cognitive Development (ABCD) dataset, with training losses evaluated through Image Pixel loss, Perceptual loss, GAN loss, and brain-focused patch GAN loss. The results not only showed superior performance compared to other algorithms quantitatively and qualitatively but also have significant meaning in neuroscience in that they learned not only the image distribution but also the biological characteristics of the structural and microscopic structures of the brain. The potential application of this image translation model as a data augmentation method could address issues of data imbalance and scarcity. This research underscores the potential of multimodal imaging, specifically the combined use of T1, DTI, and tractography, in advancing disease modeling.Chapter 1. INTRODUCTION 1 Chapter 2. RELATED WORK 7 Chapter 3. METHOD 15 3.1. Architecture Overview 3.2. Octave Convolution 3.3. Networks 3.4. Training Losses 3.5. Image Quality Metrics 3.6. Comparison of generated and real FA images in low-dimensional representation 3.7. Prediction of biological and cognitive variables using predicted FA images 3.8. Prediction of Tractography from FA images 3.9. Experimental Settings Chapter 4. RESULTS 26 4.1. Qualitative Evaluation 4.2. Quantitative Evaluation 4.3. Generated FA images by Macro2Micro can efficiently predict sex, ADHD, and intelligence 4.4 Ablation Studies 4.5. Effectiveness of Macro2Micro along the distance from the center of the brain 4.6. FA Image Translation to Tractography Chapter 5. DISCUSSION AND CONCLUSIONS 40 Bibliography 45 국문초록 52석

A Deep Network for Explainable Prediction of Non-Imaging Phenotypes using Anatomical Multi-View Data

Large datasets often contain multiple distinct feature sets, or views, that offer complementary information that can be exploited by multi-view learning methods to improve results. We investigate anatomical multi-view data, where each brain anatomical structure is described with multiple feature sets. In particular, we focus on sets of white matter microstructure and connectivity features from diffusion MRI, as well as sets of gray matter area and thickness features from structural MRI. We investigate machine learning methodology that applies multi-view approaches to improve the prediction of non-imaging phenotypes, including demographics (age), motor (strength), and cognition (picture vocabulary). We present an explainable multi-view network (EMV-Net) that can use different anatomical views to improve prediction performance. In this network, each individual anatomical view is processed by a view-specific feature extractor and the extracted information from each view is fused using a learnable weight. This is followed by a wavelet transform-based module to obtain complementary information across views which is then applied to calibrate the view-specific information. Additionally, the calibrator produces an attention-based calibration score to indicate anatomical structures' importance for interpretation.Comment: 2023 The Medical Image Computing and Computer Assisted Intervention Society worksho

HA-HI: Synergising fMRI and DTI through Hierarchical Alignments and Hierarchical Interactions for Mild Cognitive Impairment Diagnosis

Early diagnosis of mild cognitive impairment (MCI) and subjective cognitive decline (SCD) utilizing multi-modal magnetic resonance imaging (MRI) is a pivotal area of research. While various regional and connectivity features from functional MRI (fMRI) and diffusion tensor imaging (DTI) have been employed to develop diagnosis models, most studies integrate these features without adequately addressing their alignment and interactions. This limits the potential to fully exploit the synergistic contributions of combined features and modalities. To solve this gap, our study introduces a novel Hierarchical Alignments and Hierarchical Interactions (HA-HI) method for MCI and SCD classification, leveraging the combined strengths of fMRI and DTI. HA-HI efficiently learns significant MCI- or SCD- related regional and connectivity features by aligning various feature types and hierarchically maximizing their interactions. Furthermore, to enhance the interpretability of our approach, we have developed the Synergistic Activation Map (SAM) technique, revealing the critical brain regions and connections that are indicative of MCI/SCD. Comprehensive evaluations on the ADNI dataset and our self-collected data demonstrate that HA-HI outperforms other existing methods in diagnosing MCI and SCD, making it a potentially vital and interpretable tool for early detection. The implementation of this method is publicly accessible at https://github.com/ICI-BCI/Dual-MRI-HA-HI.git

3D Deep Learning on Medical Images: A Review

The rapid advancements in machine learning, graphics processing technologies and availability of medical imaging data has led to a rapid increase in use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) CNNs have been employed for analysis of medical images. In this paper, we trace the history of how the 3D CNN was developed from its machine learning roots, give a brief mathematical description of 3D CNN and the preprocessing steps required for medical images before feeding them to 3D CNNs. We review the significant research in the field of 3D medical imaging analysis using 3D CNNs (and its variants) in different medical areas such as classification, segmentation, detection, and localization. We conclude by discussing the challenges associated with the use of 3D CNNs in the medical imaging domain (and the use of deep learning models, in general) and possible future trends in the field.Comment: 13 pages, 4 figures, 2 table

PSACNN: Pulse Sequence Adaptive Fast Whole Brain Segmentation

With the advent of convolutional neural networks~(CNN), supervised learning methods are increasingly being used for whole brain segmentation. However, a large, manually annotated training dataset of labeled brain images required to train such supervised methods is frequently difficult to obtain or create. In addition, existing training datasets are generally acquired with a homogeneous magnetic resonance imaging~(MRI) acquisition protocol. CNNs trained on such datasets are unable to generalize on test data with different acquisition protocols. Modern neuroimaging studies and clinical trials are necessarily multi-center initiatives with a wide variety of acquisition protocols. Despite stringent protocol harmonization practices, it is very difficult to standardize the gamut of MRI imaging parameters across scanners, field strengths, receive coils etc., that affect image contrast. In this paper we propose a CNN-based segmentation algorithm that, in addition to being highly accurate and fast, is also resilient to variation in the input acquisition. Our approach relies on building approximate forward models of pulse sequences that produce a typical test image. For a given pulse sequence, we use its forward model to generate plausible, synthetic training examples that appear as if they were acquired in a scanner with that pulse sequence. Sampling over a wide variety of pulse sequences results in a wide variety of augmented training examples that help build an image contrast invariant model. Our method trains a single CNN that can segment input MRI images with acquisition parameters as disparate as $T_1$-weighted and $T_2$-weighted contrasts with only $T_1$-weighted training data. The segmentations generated are highly accurate with state-of-the-art results~(overall Dice overlap$=0.94$), with a fast run time~($\approx$ 45 seconds), and consistent across a wide range of acquisition protocols.Comment: Typo in author name corrected. Greves -> Grev

Brain MRI-to-PET Synthesis using 3D Convolutional Attention Networks

Accurate quantification of cerebral blood flow (CBF) is essential for the diagnosis and assessment of a wide range of neurological diseases. Positron emission tomography (PET) with radiolabeled water (15O-water) is considered the gold-standard for the measurement of CBF in humans. PET imaging, however, is not widely available because of its prohibitive costs and use of short-lived radiopharmaceutical tracers that typically require onsite cyclotron production. Magnetic resonance imaging (MRI), in contrast, is more readily accessible and does not involve ionizing radiation. This study presents a convolutional encoder-decoder network with attention mechanisms to predict gold-standard 15O-water PET CBF from multi-sequence MRI scans, thereby eliminating the need for radioactive tracers. Inputs to the prediction model include several commonly used MRI sequences (T1-weighted, T2-FLAIR, and arterial spin labeling). The model was trained and validated using 5-fold cross-validation in a group of 126 subjects consisting of healthy controls and cerebrovascular disease patients, all of whom underwent simultaneous $15O-water PET/MRI. The results show that such a model can successfully synthesize high-quality PET CBF measurements (with an average SSIM of 0.924 and PSNR of 38.8 dB) and is more accurate compared to concurrent and previous PET synthesis methods. We also demonstrate the clinical significance of the proposed algorithm by evaluating the agreement for identifying the vascular territories with abnormally low CBF. Such methods may enable more widespread and accurate CBF evaluation in larger cohorts who cannot undergo PET imaging due to radiation concerns, lack of access, or logistic challenges.Comment: 19 pages, 14 figure