기계 학습 알고리즘의 점근 성질 연구

학위논문(박사)--서울대학교 대학원 :자연과학대학 통계학과,2020. 2. 김용대.In this thesis, we study the asymptotic properties of three machine learning algorithms including two supervised learning algorithms with deep neural networks and a Bayesian learning method for high-dimensional factor models. The first research problem involves learning deep neural network (DNN) classifiers. We derive the fast convergence rates of a DNN classifier learned using the hinge loss. We consider various cases for a true probability model and show that the DNN classifier achieves fast convergence rates for all cases, provided its architecture is carefully selected. The second research topic is learning sparse DNNs. We propose a sparse learning algorithm, which minimizes penalized empirical risk using a novel sparsity-inducing penalty. We establish an oracle inequality for the excess risk of the proposed sparse DNN estimator and derive convergence rates for several learning tasks. In particular, the proposed sparse DNN estimator can adaptively attain minimax optimal convergence rates for nonparametric regression problems. The third part of the thesis is devoted to Bayesian non-parametric learning for high-dimensional factor models. We propose a prior distribution based on the two-parameter Indian buffet process, which is computationally tractable. We proved that the resulting posterior distribution concentrates on the true factor dimensionality as well as contracts to the true covariance matrix at a near-optimal rate.본 논문은 세 가지 기계 학습 알고리즘의 점근 성질을 연구한다. 처음 두 장은 지도학습에 사용되는 깊은 신경망 학습을 다루며 마지막 장은 고차원 요인 모형의 베이지안 추정 방법에 대하여 연구한다. 첫 번째 장에서는 깊은 신경망 분류기에 대하여 연구한다. 우리는 힌지 손실 함수로 학습한 깊은 신경망 분류기가 몇 가지 확률 모형에 대해 빠른 수렴 속도를 달성함을 보였다. 두 번째 장에서는 깊은 신경망의 희소 학습에 대하여 연구한다. 우리는 경험 위험과 희소성을 부여하는 벌점 함수를 더한 목적 함수를 최소화하는 학습 방법을 제안하였다. 우리는 제안하는 깊은 희소 신경망 추정량에 대한 신의 부등식을 얻었으며, 이를 통해 몇 가지 통계 문제에서의 수렴 속도를 구하였다. 특히 제안하는 깊은 희소 신경망 추정량은 비모수 회귀 문제에서 적응적으로 최소최대 최적성을 달성함을 보였다. 마지막 장은 고차원 요인 모형에서 베이지안 학습의 점근 성질을 연구한다. 우리는 모수가 두개인 인도부페과정을 기반으로 한 사전분포를 제안하였다. 제안한 사전분포로부터 유도된 사후분포가 공분산 행렬을 거의 최적의 수렴 속도로 추정함과 동시에 요인 차원을 일관되게 추정할 수 있음을 증명하였다.Introduction 1 0.1 Motivation 1 0.2 Outline and contributions2 1 Fast convergence rates of deep neural networks for classification 5 1.1 Introduction 5 1.1.1 Notation 7 1.2 Estimation of the classifier with DNNs 8 1.2.1 About the hinge loss 8 1.2.2 Learning DNN with the hinge loss 10 1.3 Fast convergence rates of DNN classifiers with the hinge loss 12 1.3.1 Case 1: Smooth conditional class probability 12 1.3.2 Case 2: Smooth boundary 13 1.3.3 Case 3: Margin condition 17 1.4 Adaptive estimation 18 1.5 Use of the logistic loss 21 1.6 Concluding remarks 24 1.7 Proofs 25 1.7.1 Complexity of a class of DNNs 25 1.7.2 Convergence rate of the excess surrogate risk for general surrogate losses 26 1.7.3 Generic convergence rate for the hinge loss 31 1.7.4 Proof of Theorem 1.3.1 33 1.7.5 Proof of Theorem 1.3.2 35 1.7.6 Proof of Theorem 1.3.3 37 1.7.7 Proof of Theorem 1.3.4 40 1.7.8 Proof of Theorem 1.4.1 42 1.7.9 Proof of Theorem 1.5.1 47 1.7.10 Proof of Proposition 1.7.9 50 2 Rate-optimal sparse learning for deep neural networks 53 2.1 Introduction 53 2.1.1 Notation 54 2.1.2 Deep neural networks 55 2.1.3 Empirical risk minimization algorithm with a sparsity constraint and its nonadaptiveness 56 2.1.4 Outline 57 2.2 Learning sparse deep neural networks with the clipped L1 penalty 57 2.3 Main results 59 2.3.1 Nonparametric regression 59 2.3.2 Classification with strictly convex losses 65 2.4 Implementation 67 2.5 Numerical studies 69 2.5.1 Regression with simulated data 69 2.5.2 Classification with real data 71 2.6 Conclusion 73 2.7 Proofs 74 2.7.1 Covering numbers of classes of DNNs 74 2.7.2 Proofs of Theorem 2.3.1 and Theorem 2.3.3 77 2.7.3 Proofs of Theorem 2.3.2 and Theorem 2.3.4 84 3 Posterior consistency of the factor dimensionality in high-dimensional sparse factor models 87 3.1 Introduction 87 3.1.1 Notation 89 3.2 Assumptions and prior distribution 90 3.2.1 Assumptions 90 3.2.2 Prior distribution and its properties 92 3.2.2.1 Induced distribution of the factor dimensionality 93 3.2.2.2 Induced distribution of the sparsity 94 3.2.2.3 Prior concentration near the true loading matrix 94 3.3 Asymptotic properties of the posterior distribution 96 3.3.1 Posterior contraction rate for covariance matrix 96 3.3.2 Posterior consistency of the factor dimensionality 97 3.4 Numerical results 98 3.4.1 MCMC algorithm 99 3.4.2 Simulation study 101 3.5 Discussions about adaptive priors 103 3.6 Concluding remarks 105 3.7 Proofs 106 3.7.1 Proofs of lemmas and corollary in Section 3.2 106 3.7.2 Proofs of theorems in Section 3.3 112 3.7.3 Proof of Theorem 3.5.1. 118 3.7.4 Auxiliary lemmas 121 Appendix A Smooth function approximation by deep neural networks with general activation functions 129 A.1 Introduction 129 A.1.1 Notation 130 A.2 Deep neural networks 131 A.3 Classes of activation functions 132 A.3.1 Piecewise linear activation functions 132 A.3.2 Locally quadratic activation functions 133 A.4 Approximation of Hlder smooth functions by deep neural networks 135 A.5 Application to statistical learning theory 139 A.5.1 Application to regression 141 A.5.2 Application to binary classification 142 A.6 Proofs 144 A.6.1 Proof of Theorem A.4.1 for piecewise linear activation functions 144 A.6.2 Proof of Theorem A.4.1 for locally quadratic activation functions 146 A.6.3 Proof of Proposition A.5.1 154 A.6.4 Proof of Theorem A.5.2. 155 A.6.5 Proof of Theorem A.5.3. 157 Appendix B Poisson mixture of finite feature models 159 B.1 Overview 159 B.1.1 Equivalence classes 160 B.1.2 Notation 161 B.2 Equivalent representations 161 B.2.1 Urn schemes 161 B.2.2 Hierarchical representation 163 B.3 Application to sparse Bayesian factor models 165 B.3.1 Model and prior 165 B.3.2 Assumptions on the true distribution 166 B.3.3 Preliminary results 167 B.3.4 Asymptotic properties 169 B.4 Proofs 170 B.4.1 Proofs of results in Section B.2 170 B.4.2 Proofs of results in Section B.3.3 174 B.4.3 Proof of Theorem B.3.5 177 Bibliography 181Docto