6 research outputs found
What does fault tolerant Deep Learning need from MPI?
Deep Learning (DL) algorithms have become the de facto Machine Learning (ML)
algorithm for large scale data analysis. DL algorithms are computationally
expensive - even distributed DL implementations which use MPI require days of
training (model learning) time on commonly studied datasets. Long running DL
applications become susceptible to faults - requiring development of a fault
tolerant system infrastructure, in addition to fault tolerant DL algorithms.
This raises an important question: What is needed from MPI for de- signing
fault tolerant DL implementations? In this paper, we address this problem for
permanent faults. We motivate the need for a fault tolerant MPI specification
by an in-depth consideration of recent innovations in DL algorithms and their
properties, which drive the need for specific fault tolerance features. We
present an in-depth discussion on the suitability of different parallelism
types (model, data and hybrid); a need (or lack thereof) for check-pointing of
any critical data structures; and most importantly, consideration for several
fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI
and their applicability to fault tolerant DL implementations. We leverage a
distributed memory implementation of Caffe, currently available under the
Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches
by ex- tending MaTEx-Caffe for using ULFM-based implementation. Our evaluation
using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies
demonstrates the effectiveness of the proposed fault tolerant DL implementation
using OpenMPI based ULFM
Artificial Neural Network Pruning to Extract Knowledge
Artificial Neural Networks (NN) are widely used for solving complex problems
from medical diagnostics to face recognition. Despite notable successes, the
main disadvantages of NN are also well known: the risk of overfitting, lack of
explainability (inability to extract algorithms from trained NN), and high
consumption of computing resources. Determining the appropriate specific NN
structure for each problem can help overcome these difficulties: Too poor NN
cannot be successfully trained, but too rich NN gives unexplainable results and
may have a high chance of overfitting. Reducing precision of NN parameters
simplifies the implementation of these NN, saves computing resources, and makes
the NN skills more transparent. This paper lists the basic NN simplification
problems and controlled pruning procedures to solve these problems. All the
described pruning procedures can be implemented in one framework. The developed
procedures, in particular, find the optimal structure of NN for each task,
measure the influence of each input signal and NN parameter, and provide a
detailed verbal description of the algorithms and skills of NN. The described
methods are illustrated by a simple example: the generation of explicit
algorithms for predicting the results of the US presidential election.Comment: IJCNN 202
๋ฅ๋ฌ๋ ๋ฐฉ๋ฒ๋ก ์ ์ด์ฉํ ๋์ ์ ์ฉ์ฑ์ ๊ฐ์ง ์๊ฒฝ์ฌ๋ฐฐ ํํ๋ฆฌ์นด ๋์ ์ ์ฐจ ๊ธฐ๋ฐ ๋ชจ๋ธ ๊ฐ๋ฐ
ํ์๋
ผ๋ฌธ(๋ฐ์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ๋์
์๋ช
๊ณผํ๋ํ ๋๋ฆผ์๋ฌผ์์ํ๋ถ, 2022. 8. ์์ ์ต.Many agricultural challenges are entangled in a complex interaction between crops and the environment. As a simplifying tool, crop modeling is a process of abstracting and interpreting agricultural phenomena. Understanding based on this interpretation can play a role in supporting academic and social decisions in agriculture. Process-based crop models have solved the challenges for decades to enhance the productivity and quality of crop production; the remaining objectives have led to demand for crop models handling multidirectional analyses with multidimensional information. As a possible milestone to satisfy this goal, deep learning algorithms have been introduced to the complicated tasks in agriculture. However, the algorithms could not replace existing crop models because of the research fragmentation and low accessibility of the crop models. This study established a developmental protocol for a process-based crop model with deep learning methodology. Literature Review introduced deep learning and crop modeling, and it explained the reasons for the necessity of this protocol despite numerous deep learning applications for agriculture. Base studies were conducted with several greenhouse data in Chapters 1 and 2: transfer learning and U-Net structure were utilized to construct an infrastructure for the deep learning application; HyperOpt, a Bayesian optimization method, was tested to calibrate crop models to compare the existing crop models with the developed model. Finally, the process-based crop model with full deep neural networks, DeepCrop, was developed with an attention mechanism and multitask decoders for hydroponic sweet peppers (Capsicum annuum var. annuum) in Chapter 3. The methodology for data integrity showed adequate accuracy, so it was applied to the data in all chapters. HyperOpt was able to calibrate food and feed crop models for sweet peppers. Therefore, the compared models in the final chapter were optimized using HyperOpt. DeepCrop was trained to simulate several growth factors with environment data. The trained DeepCrop was evaluated with unseen data, and it showed the highest modeling efficiency (=0.76) and the lowest normalized root mean squared error (=0.18) than the compared models. With the high adaptability of DeepCrop, it can be used for studies on various scales and purposes. Since all methods adequately solved the given tasks and underlay the DeepCrop development, the established protocol can be a high throughput for enhancing accessibility of crop models, resulting in unifying crop modeling studies.๋์
์์คํ
์์ ๋ฐ์ํ๋ ๋ฌธ์ ๋ค์ ์๋ฌผ๊ณผ ํ๊ฒฝ์ ์ํธ์์ฉ ํ์ ๋ณต์กํ๊ฒ ์ฝํ ์๋ค. ์๋ฌผ ๋ชจ๋ธ๋ง์ ๋์์ ๋จ์ํํ๋ ๋ฐฉ๋ฒ์ผ๋ก์จ, ๋์
์์ ์ผ์ด๋๋ ํ์์ ์ถ์ํํ๊ณ ํด์ํ๋ ๊ณผ์ ์ด๋ค. ๋ชจ๋ธ๋ง์ ํตํด ๋์์ ์ดํดํ๋ ๊ฒ์ ๋์
๋ถ์ผ์ ํ์ ์ ๋ฐ ์ฌํ์ ๊ฒฐ์ ์ ์ง์ํ ์ ์๋ค. ์ง๋ ์๋
๊ฐ ์ ์ฐจ ๊ธฐ๋ฐ ์๋ฌผ ๋ชจ๋ธ์ ๋์
์ ๋ฌธ์ ๋ค์ ํด๊ฒฐํ์ฌ ์๋ฌผ ์์ฐ์ฑ ๋ฐ ํ์ง์ ์ฆ์ง์์ผฐ์ผ๋ฉฐ, ํ์ฌ ์๋ฌผ ๋ชจ๋ธ๋ง์ ๋จ์์๋ ๊ณผ์ ๋ค์ ๋ค์ฐจ์ ์ ๋ณด๋ฅผ ๋ค๋ฐฉํฅ์์ ๋ถ์ํ ์ ์๋ ์๋ฌผ ๋ชจ๋ธ์ ํ์๋ก ํ๊ฒ ๋์๋ค. ์ด๋ฅผ ๋ง์กฑ์ํฌ ์ ์๋ ์ง์นจ์ผ๋ก์จ, ๋ณต์กํ ๋์
์ ๊ณผ์ ๋ค์ ๋ชฉํ๋ก ๋ฅ๋ฌ๋ ์๊ณ ๋ฆฌ์ฆ์ด ๋์
๋์๋ค. ๊ทธ๋ฌ๋, ์ด ์๊ณ ๋ฆฌ์ฆ๋ค์ ๋ฎ์ ๋ฐ์ดํฐ ์๊ฒฐ์ฑ ๋ฐ ๋์ ์ฐ๊ตฌ ๋ค์์ฑ ๋๋ฌธ์ ๊ธฐ์กด์ ์๋ฌผ ๋ชจ๋ธ๋ค์ ๋์ฒดํ์ง๋ ๋ชปํ๋ค. ๋ณธ ์ฐ๊ตฌ์์๋ ๋ฅ๋ฌ๋ ๋ฐฉ๋ฒ๋ก ์ ์ด์ฉํ์ฌ ์ ์ฐจ ๊ธฐ๋ฐ ์๋ฌผ ๋ชจ๋ธ์ ๊ตฌ์ถํ๋ ๊ฐ๋ฐ ํ๋กํ ์ฝ์ ํ๋ฆฝํ์๋ค. Literature Review์์๋ ๋ฅ๋ฌ๋๊ณผ ์๋ฌผ ๋ชจ๋ธ์ ๋ํด ์๊ฐํ๊ณ , ๋์
์ผ๋ก์ ๋ฅ๋ฌ๋ ์ ์ฉ ์ฐ๊ตฌ๊ฐ ๋ง์์๋ ์ด ํ๋กํ ์ฝ์ด ํ์ํ ์ด์ ๋ฅผ ์ค๋ช
ํ์๋ค. ์ 1์ฅ๊ณผ 2์ฅ์์๋ ๊ตญ๋ด ์ฌ๋ฌ ์ง์ญ์ ๋ฐ์ดํฐ๋ฅผ ์ด์ฉํ์ฌ ์ ์ด ํ์ต ๋ฐ U-Net ๊ตฌ์กฐ๋ฅผ ํ์ฉํ์ฌ ๋ฅ๋ฌ๋ ๋ชจ๋ธ ์ ์ฉ์ ์ํ ๊ธฐ๋ฐ์ ๋ง๋ จํ๊ณ , ๋ฒ ์ด์ง์ ์ต์ ํ ๋ฐฉ๋ฒ์ธ HyperOpt๋ฅผ ์ฌ์ฉํ์ฌ ๊ธฐ์กด ๋ชจ๋ธ๊ณผ ๋ฅ๋ฌ๋ ๊ธฐ๋ฐ ๋ชจ๋ธ์ ๋น๊ตํ๊ธฐ ์ํด ์ํ์ ์ผ๋ก WOFOST ์๋ฌผ ๋ชจ๋ธ์ ๋ณด์ ํ๋ ๋ฑ ๋ชจ๋ธ ๊ฐ๋ฐ์ ์ํ ๊ธฐ๋ฐ ์ฐ๊ตฌ๋ฅผ ์ํํ์๋ค. ๋ง์ง๋ง์ผ๋ก, ์ 3์ฅ์์๋ ์ฃผ์ ๋ฉ์ปค๋์ฆ ๋ฐ ๋ค์ค ์์
๋์ฝ๋๋ฅผ ๊ฐ์ง ์์ ์ฌ์ธต ์ ๊ฒฝ๋ง ์ ์ฐจ ๊ธฐ๋ฐ ์๋ฌผ ๋ชจ๋ธ์ธ DeepCrop์ ์๊ฒฝ์ฌ๋ฐฐ ํํ๋ฆฌ์นด(Capsicum annuum var. annuum) ๋์์ผ๋ก ๊ฐ๋ฐํ์๋ค. ๋ฐ์ดํฐ ์๊ฒฐ์ฑ์ ์ํ ๊ธฐ์ ๋ค์ ์ ํฉํ ์ ํ๋๋ฅผ ๋ณด์ฌ์ฃผ์์ผ๋ฉฐ, ์ ์ฒด ์ฑํฐ ๋ฐ์ดํฐ์ ์ ์ฉํ์๋ค. HyperOpt๋ ์๋ ๋ฐ ์ฌ๋ฃ ์๋ฌผ ๋ชจ๋ธ๋ค์ ํํ๋ฆฌ์นด ๋์์ผ๋ก ๋ณด์ ํ ์ ์์๋ค. ๋ฐ๋ผ์, ์ 3์ฅ์ ๋น๊ต ๋์ ๋ชจ๋ธ๋ค์ ๋ํด HyperOpt๋ฅผ ์ฌ์ฉํ์๋ค. DeepCrop์ ํ๊ฒฝ ๋ฐ์ดํฐ๋ฅผ ์ด์ฉํ๊ณ ์ฌ๋ฌ ์์ก ์งํ๋ฅผ ์์ธกํ๋๋ก ํ์ต๋์๋ค. ํ์ต์ ์ฌ์ฉํ์ง ์์ ๋ฐ์ดํฐ๋ฅผ ์ด์ฉํ์ฌ ํ์ต๋ DeepCrop๋ฅผ ํ๊ฐํ์์ผ๋ฉฐ, ์ด ๋ ๋น๊ต ๋ชจ๋ธ๋ค ์ค ๊ฐ์ฅ ๋์ ๋ชจํ ํจ์จ(EF=0.76)๊ณผ ๊ฐ์ฅ ๋ฎ์ ํ์คํ ํ๊ท ์ ๊ณฑ๊ทผ ์ค์ฐจ(NRMSE=0.18)๋ฅผ ๋ณด์ฌ์ฃผ์๋ค. DeepCrop์ ๋์ ์ ์ฉ์ฑ์ ๊ธฐ๋ฐ์ผ๋ก ๋ค์ํ ๋ฒ์์ ๋ชฉ์ ์ ๊ฐ์ง ์ฐ๊ตฌ์ ์ฌ์ฉ๋ ์ ์์ ๊ฒ์ด๋ค. ๋ชจ๋ ๋ฐฉ๋ฒ๋ค์ด ์ฃผ์ด์ง ์์
์ ์ ์ ํ ํ์ด๋๊ณ DeepCrop ๊ฐ๋ฐ์ ๊ทผ๊ฑฐ๊ฐ ๋์์ผ๋ฏ๋ก, ๋ณธ ๋
ผ๋ฌธ์์ ํ๋ฆฝํ ํ๋กํ ์ฝ์ ์๋ฌผ ๋ชจ๋ธ์ ์ ๊ทผ์ฑ์ ํฅ์์ํฌ ์ ์๋ ํ๊ธฐ์ ์ธ ๋ฐฉํฅ์ ์ ์ํ์๊ณ , ์๋ฌผ ๋ชจ๋ธ ์ฐ๊ตฌ์ ํตํฉ์ ๊ธฐ์ฌํ ์ ์์ ๊ฒ์ผ๋ก ๊ธฐ๋ํ๋ค.LITERATURE REVIEW 1
ABSTRACT 1
BACKGROUND 3
REMARKABLE APPLICABILITY AND ACCESSIBILITY OF DEEP LEARNING 12
DEEP LEARNING APPLICATIONS FOR CROP PRODUCTION 17
THRESHOLDS TO APPLY DEEP LEARNING TO CROP MODELS 18
NECESSITY TO PRIORITIZE DEEP-LEARNING-BASED CROP MODELS 20
REQUIREMENTS OF THE DEEP-LEARNING-BASED CROP MODELS 21
OPENING REMARKS AND THESIS OBJECTIVES 22
LITERATURE CITED 23
Chapter 1 34
Chapter 1-1 35
ABSTRACT 35
INTRODUCTION 37
MATERIALS AND METHODS 40
RESULTS 50
DISCUSSION 59
CONCLUSION 63
LITERATURE CITED 64
Chapter 1-2 71
ABSTRACT 71
INTRODUCTION 73
MATERIALS AND METHODS 75
RESULTS 84
DISCUSSION 92
CONCLUSION 101
LITERATURE CITED 102
Chapter 2 108
ABSTRACT 108
NOMENCLATURE 110
INTRODUCTION 112
MATERIALS AND METHODS 115
RESULTS 124
DISCUSSION 133
CONCLUSION 137
LITERATURE CITED 138
Chapter 3 144
ABSTRACT 144
INTRODUCTION 146
MATERIALS AND METHODS 149
RESULTS 169
DISCUSSION 182
CONCLUSION 187
LITERATURE CITED 188
GENERAL DISCUSSION 196
GENERAL CONCLUSION 201
ABSTRACT IN KOREAN 203
APPENDIX 204๋ฐ