Deep Neural Network Transformation and Learning based on Knowledge Distillation

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2019. 2. 최기영.심층 신경망은 오늘날 이미지 분류나 음성 인식 등 여러 분야에서 뛰어난 성능으로 널리 사용되고 있다. 하지만 더 높은 인식률을 얻기 위하여 보다 깊고 보다 넓은 네트워크가 등장하였고, 이러한 네트워크들은 더 높은 연산 능력과 메모리 저장 공간을 요구하기 때문에, 상대적으로 작은 IoT 디바이스나 임베디드 시스템에 큰 네트워크를 적용하는 것이 쉽지 않았다. 따라서 크기는 작음에도 불구하고 더 높은 인식률을 가지는 네트워크를 얻을 수 있다면 이를 해결하는데 매우 유용할 것이고, 이에 네트워크 압축 방법 중 하나인 지식 증류법 (knowledge distillation)에 대하여 연구를 진행하였다. 첫 번째, 낮은 층의 압축률을 높이고 높은 층의 압축률을 낮추는 tapered-ratio compression을 제안하였다. 수직선이나 수평선과 같은 단순한 특징들은 낮은 층에서 그 특징들이 추출되고 단순한 특징들은 중복될 가능성이 높기 때문에 낮은 층의 압축률을 높일 수 있을 것이라 예상하였고, 이는 레지듀얼 경로 (residual path)를 가지는 레지듀얼 네트워크 (residual networkResNet)에서 효과적으로 동작함을 보였다. 레지듀얼 네트워크에 대하여 tapered-ratio compression을 적용하여, 모든 층에 대하여 일정하게 압축률을 적용하는 uniform-ratio compression에 비해 더 적은 곱셈 연산 수와 더 작은 양의 메모리 접근만을 가지고도 높은 인식률에 도달할 수 있었다. 두 번째, 학습 될 student 네트워크와 비교하여 다양한 크기와 종류의 teacher 네트워크를 이용하여 지식 증류법을 적용하여, 그 효과를 보이고 결과를 분석하였다. Student 네트워크보다 더 크고 더 높은 인식률을 가지는 teacher 네트워크를 사용하여 지식 증류법을 적용하는 것이 일반적이지만, 이전 연구 born again neural network에서 보여준 같은 크기의 teacher 네트워크를 사용하는 경우에도 지식 증류법이 효과적으로 적용될 수 있음을 보였고, 이에 더 나아가 student 네트워크보다 작은 teacher 네트워크를 적용하였을 때도 효과적으로 동작함을 보여주었다. 이 때, 로스 (loss) 분석을 통하여 지식 증류법이 regularizer로 동작함을 보였고, 이에 더하여 같은 종류의 네트워크간의 지식 증류법뿐만 아니라 다른 종류의 네트워크 간의 지식 증류법 또한 효과적으로 동작함을 보였다.Deep Neural Networks (DNNs) are widely used for many applications such as image classification and voice recognition due to their outstanding performance. However, the deeper and wider networks require the higher computing power and larger memory space and bandwidth. It is not easy for IoT devices or embedded system to adopt such networks due to their small memory space and low computing power. Thus, if one can obtain a smaller network even with higher accuracy, it will be very useful for integrating DNNs into such a system. In this thesis, we study network compression method base on knowledge distillation. First, we propose tapered-ratio compression which applies lower compression ratio toward top layer (output side). Generally, simple features like horizontal line or vertical edge are extracted in lower layers (close to the bottom layer) and more complex features are extracted in upper layers (close to the top layer). Simple features are more likely to have redundancies than complex features. Thus we expect that more filters in lower layers can be removed than those in upper layers. Experimental results show that tapered-ratio compression can be applied effectively for residual network. By applying tapered-ratio compression into residual network, student networks achieve higher accuracy in spite of fewer multiplications and smaller amount of memory access than uniform-ratio compression cases. Second, we analyze the results of knowledge distillation with teacher networks of several sizes and different types. In general, a knowledge distillation with a bigger teacher network which has higher accuracy is commonly used. However, as shown in previous work called born again neural networks, a knowledge distillation with a teacher network which has the same size as the student network or even with a teacher network smaller than the student network can be applied effectively too. Base on analysis of loss, we show that knowledge distillation works as a regularizer. We also show that knowledge distillation can be applied successfully to different types of network.초 록 i 목 차 iii 그림 목차 iv 표 목차 v 제 1 장 서 론 1 제 2 장 배경 지식 및 관련 연구 2 제 1 절 네트워크 아키텍처 2 제 2 절 네트워크 압축 5 제 3 절 CIFAR 데이터세트 7 제 3 장 본 론 8 제 1 절 Tapered-ratio compression 8 제 2 절 실험 결과 - Tapered-ratio compression 12 제 3 절 지식 증류법의 다양한 적용 방법 19 제 4 절 실험 결과 - 지식 증류법의 다양한 적용 방법 21 제 4 장 결 론 29 참고 문헌 31 부 록 34 Abstract 43Maste

Hypermethylation of RUNX3 gene promoter in human colorectal cancer cell lines

학위논문(박사)--서울대학교 대학원 :의학과 외과학 전공,2004.Docto

가족성 용종증에서 대장암 동반과 관련된 인자 : 십이지장 용종

Thesis (master`s)--서울대학교 대학원 :의학과 외과학전공,2001.Maste

Fast defect detection and classification of patterned thin film transistor-liquid crystal displays

Thesis(master`s)--서울대학교 대학원 :기계항공공학부,2006.Maste

Experiences on Obturator Hernia and Review of Korean Cases

Purpose: We performed this study to share experiences in the management of obturator hernia, which is a very rare disease among elderly women, because rarity of this disease will not approve an expert institute or surgeon and because the aging society, Korea, can increase the incidence. Methods: Patient characteristics, clinical manifestations and treatment results were retrospectively collected from the 12 obturator hernia patients since 2000 in the three hospitals of the authors. Twenty-one obturator hernia cases reported in the Korean literatures were reviewed. In addition, we analyzed clinical features and treatment results of the total 33 patients, collectively. Results: Most of the patients were elderly women except one Young, poorly nourished, male patient with pulmonary tuberculosis. Their mean age was 79.6 years. Sixty-four percent (21/33) of the patients had preoperative morbidity. Symptoms from the compression of the obturator nerve, which is an important Clue to the diagnosis, were observed in 67% (22/33). Interestingly, spontaneous or suspicious-spontaneous reductions were reported in 7 (21%) patients. Abdominal CT scan was the major tool for diagnosis. Abdominal approach alone could successfully manage most cases, though 84% (27/32) needed anastomosis of the bowel. Operative mortality was 2 of the 32 cases but morbidity was 44% of the 32 patients and the mean hospital period after operation was 21 days. Conclusion: Management of patients with obturator hernia connotes high morbidity and mortality. Sharing the experience of management, prompt diagnosis and Surgical management and proper peri-operative care as well as appropriate managing the families is essential for improved results. (J Korean Surg Soc 2010;78:41-50)Hwang KT, 2009, J KOREAN SURG SOC, V77, P211, DOI 10.4174/jkss.2009.77.3.211Park CY, 2009, J KOREAN SURG SOC, V76, P192Rodriguez-Hermosa JI, 2008, HERNIA, V12, P289, DOI 10.1007/s10029-007-0328-yCHOI SI, 2008, J KOREAN SURG SOC, V75, P415Haraguchi M, 2007, ANN ACAD MED SINGAP, V36, P413Kim MJ, 2006, DIABETES RES CLIN PR, V73, P8, DOI 10.1016/j.diabres.2005.11.013KIM HA, 2005, J KOREAN SURG SOC, V68, P168Kammori M, 2004, AM J SURG, V187, P549, DOI 10.1016/j.amjsurg.2003.12.041BAE JD, 2004, J KOREAN SURG SOC, V66, P438CHOI HJ, 2002, J KOREAN SURG SOC, V63, P509Schmidt PH, 2001, AM SURGEON, V67, P191CHANG SJ, 2001, J KOREAN SURG SOC, V61, P216Skandalakis LJ, 2000, SURG CLIN N AM, V80, P71YOON YD, 1998, J KOREAN SURG SOC, V54, P1038ZINNER MJ, 1997, MAINGOTS ABDOMINAL O, P540BERGSTEIN JM, 1996, SURGERY, V119, P133KIM SY, 1995, J KOREAN SURG SOC, V49, P146LEE MS, 1994, J KOREAN SURG SOC, V46, P887KIM MY, 1994, J KOREAN RADIOL SOC, V30, P875PARK YS, 1991, J KOREAN SURG SOC, V40, P132KANG YS, 1987, J KOREAN SURG SOC, V33, P116KWON TW, 1987, J KOREAN SURG SOC, V33, P756CHOI KH, 1987, J KOREAN SURG SOC, V32, P723SUH SH, 1968, J KOREAN SURG SOC, V10, P35

항문직장의 악성흑색종

P urpose: Anorectal melanoma is an uncommon tumor that comprises less than 1% of all malignancies of the anorectum.¹Its epidemiologic features, and clinical characteristics have not been well studied. We reviewed our experience in management of malignant melanoma of anorectal region. M ethods: Eight patients with malignant melanoma of anorectal region were treated at Seoul National University Hospital in the period of 1980∼2001. They represented 0.36% of the 2,246 patients with colorectal cancer seen at Seoul National University Hospital during the same period. R esults: Five patients were female. The median age was 60 years. Common presenting symptoms were decreased stool caliber (63%), and anal bleeding (50%). All lesions developed at the area between 2cm and 7cm from anal verge and could be palpated on digital rectal examination. Two patients had multiple hepatic metastases at initial presentation. Of the remaining 6, abdominoperineal resection was performed for 4 patients, and local excision for 2. During the follow-up period with median length of 11 months, local recurrence occurred in 1 patient and distant metastases occurred in 3. Metastatic disease involved lung and/or brain. Five patients died during the follow-up period and the length of mean survival was 12 months. C onclusion: Malignant melanoma of anorectum seems to have a poor prognosis with an appreciable incidence of regional node metastases. Common symptoms were similar to those of common anorectal disease. But all lesions were palpable on digital rectal examination. Increased awareness of this rare condition may lead to early detection and therefore to improved results

Open versus laparoscopic surgery for mid or low rectal cancer after neoadjuvant chemoradiotherapy (COREAN trial): 10-year follow-up of an open-label, non-inferiority, randomised controlled trial

Background Laparoscopic surgery has been widely used for rectal cancer; however, its long-term outcomes remain controversial. This study aimed to assess the long-term oncological safety of laparoscopic surgery for rectal cancer using 10-year follow-up data of the Comparison of Open versus laparoscopic surgery for mid or low REctal cancer After Neoadjuvant chemoradiotherapy (COREAN) trial. Methods The COREAN trial is an open-label, non-inferiority, randomised controlled trial. Eligible participants were aged 18-80 years, had cT3N0-2M0 middle or low rectal cancer with lesions located within 9 cm of the anal verge, and had been treated with preoperative chemoradiotherapy. Patients were randomly assigned (1:1) to open or laparoscopic surgery with a computer-generated random allocation sequence with a random permuted block design. Neither patients nor clinicians were masked to treatment assignment. Open or laparoscopic total mesorectal excision was done 6-8 weeks after the administration of preoperative concurrent chemoradiotherapy (fluoropyrimidines alone, doublet therapy, or triplet therapy) at a dose of 50.5 Gy over 5.5 weeks. Postoperative adjuvant chemotherapy was administered for 4 months. The primary endpoint of 3-year disease-free survival was published previously. Here, we report 10-year overall survival, disease-free survival, and local recurrence. Analyses were done in the modified intention-to-treat population of all participants who were randomly assigned and provided follow-up data. This study is registered with ClinicalTrials.gov, NCT00470951. Findings Of the 340 patients enrolled in the COREAN trial between April 4, 2006, and Aug 26, 2009 (170 patients in each group), two patients in the laparoscopic surgery group moved abroad and were lost to follow-up, so were not included in this 10-year analysis. The median duration of follow-up was 143 months (IQR 122-156). No differences were observed in 10-year overall survival (74.1% [95% CI 66.8-80.0] in the open surgery group vs 76.8% [69.6-82.5] in the laparoscopic surgery group; p=0.44), 10-year disease-free survival (59.3% [51.1-66.5] vs 64.3% [56.0-71.5]; p=0.20), or 10-year local recurrence (8.9% [5.2-15.0] vs 3.4% [1.4-7.9]; p=0.050) between the open surgery and laparoscopic surgery groups at 10 years after surgery. The stratified hazard ratios, adjusted for ypT and ypN classification and tumour regression grade, for open surgery versus laparoscopic surgery were 0.94 (95% CI 0.63-1.43) for overall survival, 1.05 (0.74-1.49) for disease-free survival, and 2.22 (0.78-6.34) for local recurrence. Interpretation The 10-year follow-up of the COREAN trial confirms the long-term oncological safety of laparoscopic surgery in patients with rectal cancer treated with preoperative chemoradiotherapy. Similar to open surgery, laparoscopic surgery does not compromise long-term survival outcomes in rectal cancer when performed by well trained surgeons. Copyright (C) 2021 Elsevier Ltd. All rights reserved