지식 증류의 구조적 가지치기에의 적용가능성에 대한 포괄적인 연구

학위논문(박사) -- 서울대학교대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2024. 2. 곽노준.Model compression is an actively pursued area of research over the last few years, with the goal of deploying deep networks in low-power and re- source limited devices without significant drop in accuracy. Since the overwhelming performance of deep neural networks entails large resource consumption, rather than simply using the deep learning models that performs better, many researchers sought ways to make better trade-offs between the performance and capacity. Currently, there are four types of deep learning model compression techniques: Network pruning, Quantization, Low-rank approximation, and Knowledge distillation. Each method is separately studied in the early years of the prevalence of the deep learning researches, and the current flow of researches head toward combining the separate model compression techniques appropriately. In this dissertation, we make a comprehensive study on knowledge distillation and find ways to appropriately combine knowledge distillation to the structured pruning for better model compression. First, we speculate conventional knowledge distillation methods based on the difference of the feature-map-level knowledge distillation and label-based knowledge distillation. We revisit the conventional knowledge distillation methods based on the view of model ensemble, and introduce techniques for distilling knowledge from multiple network into a single student network, and raise a problem that the an ensemble teacher is very efficient model compression method, but it cannot be conducted in conventional methods at feature-map-level. For a solution, we propose parallel FEED that able distilling ensemble knowledge at the feature-map-level. Second, we revisit the conventional network pruning methods, and also review the studies that tried to use the knowledge distillation and network pruning simultaneously, by using knowledge distillation at the fine-tuning stage of the network pruning. We name this as Post-Distill. We point out that there still re- main rooms for better accuracy and computation trade-offs, and point out that the ideal goal of combining the two methods is their orthogonal performance improvement. For a solution, we propose to use knowledge distillation beforehand of the network pruning, which sets a higher upper-bound for future performance restoration after network pruning. We name this as Pre-Distill. We show that when knowledge distillation is used used at both stages, called pre-post-distill, the model can achieve almost orthogonal efficiency improvement. Also, using the Pre-Distilled student as the teacher for the pruned network at the fine-tuning stage, named self-distill, brings slightly better performance improvement. Finally, we propose to even prune the teacher network before using it for distillation. This teacher pruning makes the distillation process pruning-aware distillation. The so-called dark knowledge of the teacher network delivers the sparse information of the teacher. Thus, the student network learned from the sparse teacher network are more robust from the degradation of the network pruning, and shows better performance at all pruning ratio after the fine-tuning.모델 압축은 지난 몇년간 활발히 연구되어왔던 분야로, 저전력 등의 자원이 한정된 기기들에서 지나친 성능 저하 없이 심층신경망을 사용할 수 있게 하는것을 목표로 한다. 압도적인 성능을 내는 심층 신경망은 큰 자원 소모를 수반하기 때문에 많은 연구자들은 단순히 더 성능이 좋은 딥 러닝 모델을 사용하기보다는, 성능과 용량 사이에서 더 좋은 트레이드오프를 찾는 방법을 연구했다. 현재, 딥러닝 기반 모델 압축 방법에는 네 가지 방법이 존재한다: 네트워크 가지치기, 파라미터 양자화, 낮은 계수 근사법, 지식 증류. 각각의 방법은 초기 딥러닝의 발전 시기부터 각각 연구되어 왔고, 근래 연구의 흐름은 각기 다른 모델 압축 방법을 적절히 합치는 방향으로 흘러가고 있다. 본 논문에서는 지식 증류 방법에 특화된 모델 압축에서의 어려운 문제들을 해결하는 방법을 소개한다. 첫 번째로, 우리는 모델 앙상블 기반의 지식증류 기법을 재방문한다. 우리는 다수의 교사 신경망의 지식을 하나의 학생 신경망에게 증류하는 방법을 소개하고, 해당 방법의 모델 압축 관점에서의 효율성을 보여주면서 동시에 이 지식 증류를 특징맵 레벨에서 수행하는것은 기존 방법으로는 가능하지 않다는 문제를 제기한다. 해당 문제에 대한 해결방법으로, 우리는 앙상블 지식을 특징맵 레벨에서 증류를 할 수 있게 해주는 parallel FEED를 제안한다. 두 번째로, 우리는 기존의 네트워크 가지치기 방법을 재방문한다. 우리는 기존의 네트워크 가지치기와 지식 증류 방법을 네트워크의 파인튜닝 단계에서 지식 증류를 사용함으로써 동시에 적용하는 것을 시도했던 지난 연구들을 답습한다. 우리는 이것을 추후-증류 이라고 이름짓고, 아직 더 나은 트레이드오프를 확보할 수 있는 성능 차이가 존재하고, 이상적으로는 두 방법의 모델 압축 효율이 서로 독립적으로 작용하는 것이라는 문제를 제기한다. 이에 대한 해결 방법으로, 우리는 네트워크 가지치기 단계의 이전에 지식 증류를 사용하는 것을 제안한다. 이 방법은 추후 파인튜닝 단계에서의 성능 복구에 대한 상한범위를 먼저 높인다, 우리는 이 방법을 사전-증류 라고 명명하고, 초기 학습 단계와 파인튜닝 단계에서 두 증류 방법을 각각 적용하면서 네트워크 프루닝과 합쳤을 때 네트워크 가지치기와 지식 증류의 압축 효율이 독립적으로 성능 향상에 기여하는 것을 보인다. 또한, 초기 학습단계에서 획득한 학생 네트워크를 파인튜닝 단계에서 교사 네트워크로 사용하는 자기-증류 방법으로 더 높은 성능을 기록할수 있음을 보인다. 마지막으로, 초기 학습에서의 지식 증류 이전에 우리는 교사 네트워크를 미리 프루닝을 하는것을 제안한다. 이 교사 네트워크 가지치기는 네트워크의 초기 학습단계에서의 지식 증류를 추후의 네트워크 가지치기를 의식하는 지식 증류로 만들어준다. 지식 증류에서 소위 말하는 암흑 지식이 교사 네 트워크의 성긴 정도에 대한 지식도 전달함으로써, 성긴 교사 네트워크로부터 학습한 학생 네트워크는 네트워크 가지치기로 인한 성능 저하로부터 모든 가지치기 비율에서 더 강건한 모습을 보인다.Abstract i Contents iii List of Tables vi List of Figures viii 1 Introduction 1 1.1 Base Tasks 2 1.1.1 Knowledge Distillation 3 1.1.2 Network Pruning 5 1.2 Contributions and Outline 9 2 Related works 12 2.1 Knowledge Distillation . 12 2.1.1 Label distillation 13 2.1.2 Feature distillation 14 2.2 Network Pruning . 15 2.2.1 Structured pruning 16 2.2.2 Unstructured pruning . 16 iii 3 Feature-level Ensemble Knowledge Distillation for Aggregating Knowl- edge from Multiple Networks 18 3.1 Introduction . 18 3.2 Proposed Training Algorithm . 21 3.2.1 Parallel FEED . 22 3.2.2 Sequential FEED 24 3.3 Experiments . 25 3.3.1 Effectiveness of standalone feature-map-level distilla- tion losses . 27 3.3.2 Parallel FEED . 27 3.3.3 Sequential FEED 29 3.3.4 Qualitative Analysis . 32 3.3.5 Implementation Details 35 3.4 Discussion 36 3.5 Conclusion . 37 4 Combining Knowledge Distillation with Network Pruning 39 4.1 Introduction . 39 4.2 Proposed Training Processes . 41 4.2.1 Vanilla Network Pruning 41 4.2.2 Post-Distill 42 4.2.3 Pre-Distill . 43 4.2.4 Pre-Post and Self-Distill 44 4.3 Experiments . 46 4.3.1 Results on the CIFAR-100 Dataset 46 iv 4.3.2 Results on the Large Scale Dataset 53 4.3.3 Analysis . 56 4.4 Conclusion . 60 5 Distilling from a Pruned Teacher for a Pruning-friendly Student 62 5.1 Introduction . 62 5.2 Proposed Method . 64 5.2.1 Motivation 65 5.2.2 Method 66 5.3 Experiments . 68 5.3.1 Settings 68 5.3.2 Quantitative Results . 69 5.3.3 Qualitative Results 75 5.4 Conclusion . 76 6 Conclusion 78 6.1 Summary 78 6.2 Limitation and Future Works . 80 Abstract (In Korean) 88 감사의글 90박

당뇨망막병증 황반부종과 노인성 황반변성에서 망막혈액장벽 손상에 관한 중개연구

학위논문 (박사)-- 서울대학교 대학원 의과대학 의과학과, 2017. 8. 김정훈.Introduction: Diabetic retinopathy (DR) and age-related macular degeneration (AMD) are leading causes of blindness. In DR, macular edema (vascular leakage) and neovascularization (angiogenesis) cause severe vision loss. While neovascularization cause severe vision loss only in the later proliferative phase of DR, macular edema caused by vascular leakage can occur at any stage of DR and impair visual acuity. The two types of AMD are: dry and wet AMD. In wet AMD, new blood vessels (known as choroidal neovascularization) grow into the macula and damage the retina. Dry AMD is characterized by the presence of drusen and atrophy of the retinal pigment epithelium (RPE) cells. For the treatment of DR, I focused on three cellular components of inner blood-retinal barrier (BRB)endothelial cells, pericytes, and astrocytes. Especially, I aimed to investigate the role of Ang2 in pericyte loss and astrocyte loss in DR. While laser-induced choroidal neovascularization has been extensively used in the studies of wet AMD, there is no single mouse model that fully recapitulates the cardinal features of human dry AMD. Here, I focused on the Aβ-related pathogenesis in dry AMD using 5XFAD mice and Aβ-injected mice. I investigated the mechanism of Aβ uptake via receptor for advanced glycation end product (RAGE) and the role of intracellular Aβ in autophagy dysfunction as a dry AMD pathogenesis. Wet AMD is associated with retinal over-expression of, rather than mutations in, the VEGFA gene. RNA-guided genome surgery using CRISPR-Cas9 nucleases has shown promise for the treatment of diverse genetic diseases. Yet, the potential of such nucleases for therapeutic applications in non-genetic diseases is largely unexplored. Here, I used two genome editing toolsthe preassembled, Vegfa gene-specific Cas9 ribonucleoproteins (RNPs) and the smallest Cas9 orthologue characterized to date, derived from Campylobacter jejuni (CjCas9) targeted to the Vegfa or Hif1a gene in RPE cells for the treatment of wet AMD. Methods: In in vivo experiments, I used streptozotocin induced diabetic mice for DR study, 5XFAD mice and Aβ-injected mice for dry AMD study, and laser-induced choroidal neovascularization (CNV) mice for wet AMD study. In in vitro experiments, I performed cell viability assay, western blot, RT-PCR, flow cytometry, immunocytochemistry, ELISA, targeted deep sequencing, and microarray, etc. Results: Ang2 induced pericyte apoptosis under high glucose via α3β1 integrin/p53 pathway. Ang2 also induced astrocyte apoptosis under high glucose via αvβ5 integrin/GSK3β/β-catenin pathway. In addition, microglia derived-IL-6/STAT3 signaling in endothelial cell increased vascular leakage by attenuating tight junction proteins. Intracellular Aβ contributed to dry AMD-like pathology in 5XFAD mice. RAGE-mediated p38 MAPK signaling contributes to endocytosis of Aβ in RPE. Intracellular Aβ induced breakdown of tight junction and autophagy dysfunction by lysosomal impairment. Cas9 RNPs and CjCas9 effectively achieved in vivo genome editing in RPE cells. Both Cas9 RNPs and AAV/CjCas9 targeting Vegfa reduced the area of laser-induced CNV in a mouse model of wet AMD. Genome-wide profiling of Cas9 off-target effects via Digenome-seq showed that off-target mutations were rarely induced in the human genome. Conclusions: Ang2/integrin signaling could be a potential therapeutic target to prevent pericyte loss and vascular leakage by astrocyte loss in DR. IL-6/STAT3 signaling is another therapeutic target to prevent vascular leakage in DR. Intracellular Aβ contributes to dry AMD pathogenesis. 5XFAD mice could be used a dry AMD mouse model. Targeting Aβ and modulating autophagy could be novel therapeutic approaches for the treatment of dry AMD. In vivo genome editing with Cas9 RNPs or CjCas9 has the potential for the local treatment for wet AMD, non-genetic degenerative diseases, expanding the scope of RNA-guided genome surgery to a new dimension. * This work is based on published articles in Diabetes (1), Cell Death and Disease (2), Journal of Cellular Physiology (3), Neurobiology of Aging (4), Oncotarget (5, 6), Genome Research (7), and Nature Communications (8).General Introduction 1 Chapter 1 Targeting Neurovascular Units to Treat Diabetic Retinopathy 5 Introduction 6 Material and Methods 11 Results 23 Discussion 75 Chapter 2 Targeting Amyloid Beta to Treat Dry Age-related Macular Degeneration 87 Introduction 88 Material and Methods 93 Results 104 Discussion 143 Chapter 3 In vivo Genome Surgery with CRISPR-Cas9 to Treat Wet Age-related Macular Degeneration 153 Introduction 154 Material and Methods 158 Results 178 Discussion 215 References 218 Abstract in Korean 246Docto

Radar system for supporting detection type of multiple radars

본 발명은 탐지용 레이더 시스템에 관한 것으로서, 본 발명의 레이더 시스템은 다수의 레이더 탐지 방식에서 사용되는 각각의 송수신 파형에 대한 기저대역 신호를 처리하고, 중간 주파수(IF:Immediate Frequency) 신호를 생성하는 베이스밴드 모듈 및 상기 베이스밴드 모듈에서 생성된 중간 주파수 신호를 이용하여 고주파 신호로 변환하여 송신하거나, 외부로부터 고주파 신호를 수신하여 상기 베이스밴드 모듈에서 처리가능한 중간 주파수 신호로 변환하여 상기 베이스밴드 모듈에 전달하는 RF 모듈을 포함한다. 본 발명에 의하면 하나의 레이더 시스템에서 다수의 레이더탐지 방식을 구현할 수 있으므로, 각 레이더 탐지 방식의 장점을 유지하면서 단점을 보완할 수 있는 효과가 있다

Intracellular expression of amyloid beta 42 in retinal pigment epithelium of an Alzheimers disease mouse model

Purpose: The expression of amyloid beta 42 in the retina of transgenic mice with 5 familial autosomal dominant mutations (5xFAD) mice (Alzheimers disease mouse model) and the role of amyloid beta 42 on tight junction of retinal pigmented epithelium will be investigated. Methods: The retina of 5xFAD +/- mice (Tg6799) at post-natal 6, 8 months were investigated. Hematoxylin-eosin stained sections were examined for retinal pathologic changes in 5xFAD mice. With immunofluorescence (IF), the retinal pigment epithelium (RPE) expression of amyloid beta 42 and tight junction markers including occludin, zonular occludens (ZO)-1 were studied in the retina of 5xFAD mice and ARPE-19 cells. Cell viability was determined by the MTT assay in ARPE-19 cells with exogenous treatment of oligomeric amyloid beta 42 (OAβ-42). Western blot analysis for occludin, ZO-1 was also performed with treatment of OAβ-42. Results: Amyloid beta 42 was mainly expressed in the RPE layer of 5xFAD mice, especially intracellular expression. ZO-1 expression was markedly decreased, but occludin expression was not changed significantly in RPE of 5xFAD mice. Exogenous OAβ-42 could be uptaken in RPE cells in culture without cell death, subsequently induced breakdown of ZO-1, not occludin. Conclusions: Our data suggest that amyloid beta 42 is expressed in the RPE layer of 5xFAD mice of an Alzheimer disease mouse model. Intracellular amyloid beta 42 could play a role in the breakdown of tight junction of RPE in 5xFAD mice. 5xFAD mice could be used as an animal model for the study of relationship between Alzheimer disease and age-related macular degeneration.1. Introduction 1 2. Methods 3 2.1 Transgenic mice 3 2.2 Cell cultures 4 2.3 Histology 4 2.4 Immunofluorescence staining 5 2.5 Preparation of oligomeric amyloid beta 1-42 solution 8 2.6 Cell Viability Assay 9 2.7 Western blots 9 2.8 Immunocytochemistry 11 2.9 Statistical Analysis 13 3. Results 14 3.1 Histological change and immunostaining in 5XFAD mice retina 14 3.2 The effect of exogenous oligomeric Aβ42 on tight junction protein of RPE cells 17 3.3 RPE uptakes extracellular amyloid beta which attenuates tight junction of RPE 19 4. Discussion 24 5. Reference 29 6. 국문초록 31Maste