이상치 탐지를 위한 적대적 사전 학습 알고리즘

학위논문 (석사) -- 서울대학교 대학원 : 공과대학 기계공학부, 2020. 8. 박종우.In this thesis, we propose a semi-supervised dictionary learning algorithm that learns representations of only non-outlier data. The presence of outliers in a dataset is a major drawback for dictionary learning, resulting in less than desirable performance in real-world applications. Our adversarial dictionary learning (ADL) algorithm exploits a supervision dataset composed of known outliers. The algorithm penalizes the dictionary expressing the known outliers well. Penalizing the known outliers makes dictionary learning robust to the outliers present in the dataset. The proposed method can handle highly corrupted dataset which cannot be effectively dealt with using conventional robust dictionary learning algorithms. We empirically show the usefulness of our algorithm with extensive experiments on anomaly detection, using both synthetic univariate time-series data and multivariate point data.본 논문에서는 이상치가 아닌 데이터의 희소 표현만을 학습하는 준지도 사전 학습 알고리즘을 제안한다. 데이터셋에 섞여 있는 이상치는 사전 학습의 주요한 문제로, 실제 문제에 적용 시 바람직하지 않은 성능을 초래한다. 본 연구에서 제안하는 적대적 사전 학습(ADL) 알고리즘은 이상치 데이터로 구성된 감독 데이터셋을 학습에 이용한다. 우리의 알고리즘은 주어진 이상치 데이터를 잘 표현하는 사전에 페널티를 주고, 이것은 사전이 학습 데이터셋에 섞여 있는 이상치에 강건하게 학습되도록 한다. 제안된 방법은 기존의 사전 학습 방법들과 비교해 이상치의 비중이 높은 데이터셋에서도 효과적으로 사전을 학습해 낸다. 이 연구에서는 인공적인 단변량 시계열 데이터와 다변량 점 데이터에 대한 이상치 탐지 실험을 통해 알고리즘의 유용성을 경험적으로 검증한다.1 Introduction 1 1.1 Related Works 4 1.2 Contributions of This Thesis 5 1.3 Organization 6 2 Sparse Representation and Dictionary Learning 7 2.1 Sparse Representation 7 2.1.1 Problem De nition of Sparse Representation 7 2.1.2 Sparse representation with l0-norm regularization 10 2.1.3 Sparse representation with l1-norm regularization 11 2.1.4 Sparse representation with lp-norm regularization (0 < p < 1) 12 2.2 Dictionary Learning 12 2.2.1 Problem De nition of Dictionary Learning 12 2.2.2 Dictionary Learning Methods 14 3 Adversarial Dictionary Learning 18 3.1 Problem Formulation 18 3.2 Adversarial Loss 19 3.3 Optimization Algorithm 20 4 Experiments 25 4.1 Data Description 26 4.1.1 Univariate Time-series Data 26 4.1.2 Multivariate Point Data 29 4.2 Evaluation Process 30 4.2.1 A Baseline of Anomaly Detection 30 4.2.2 ROC Curve and AUC 34 4.3 Experiment Setting 35 4.4 Results 36 5 Conclusion 43 Bibliography 45 국문초록 50Maste

DC Proximal Newton for Non-Convex Optimization Problems

We introduce a novel algorithm for solving learning problems where both the loss function and the regularizer are non-convex but belong to the class of difference of convex (DC) functions. Our contribution is a new general purpose proximal Newton algorithm that is able to deal with such a situation. The algorithm consists in obtaining a descent direction from an approximation of the loss function and then in performing a line search to ensure sufficient descent. A theoretical analysis is provided showing that the iterates of the proposed algorithm {admit} as limit points stationary points of the DC objective function. Numerical experiments show that our approach is more efficient than current state of the art for a problem with a convex loss functions and non-convex regularizer. We have also illustrated the benefit of our algorithm in high-dimensional transductive learning problem where both loss function and regularizers are non-convex

Shakeout: A New Approach to Regularized Deep Neural Network Training

Recent years have witnessed the success of deep neural networks in dealing with a plenty of practical problems. Dropout has played an essential role in many successful deep neural networks, by inducing regularization in the model training. In this paper, we present a new regularized training approach: Shakeout. Instead of randomly discarding units as Dropout does at the training stage, Shakeout randomly chooses to enhance or reverse each unit's contribution to the next layer. This minor modification of Dropout has the statistical trait: the regularizer induced by Shakeout adaptively combines $L_0$, $L_1$ and $L_2$ regularization terms. Our classification experiments with representative deep architectures on image datasets MNIST, CIFAR-10 and ImageNet show that Shakeout deals with over-fitting effectively and outperforms Dropout. We empirically demonstrate that Shakeout leads to sparser weights under both unsupervised and supervised settings. Shakeout also leads to the grouping effect of the input units in a layer. Considering the weights in reflecting the importance of connections, Shakeout is superior to Dropout, which is valuable for the deep model compression. Moreover, we demonstrate that Shakeout can effectively reduce the instability of the training process of the deep architecture.Comment: Appears at T-PAMI 201