Training Group Orthogonal Neural Networks with Privileged Information

Learning rich and diverse representations is critical for the performance of deep convolutional neural networks (CNNs). In this paper, we consider how to use privileged information to promote inherent diversity of a single CNN model such that the model can learn better representations and offer stronger generalization ability. To this end, we propose a novel group orthogonal convolutional neural network (GoCNN) that learns untangled representations within each layer by exploiting provided privileged information and enhances representation diversity effectively. We take image classification as an example where image segmentation annotations are used as privileged information during the training process. Experiments on two benchmark datasets -- ImageNet and PASCAL VOC -- clearly demonstrate the strong generalization ability of our proposed GoCNN model. On the ImageNet dataset, GoCNN improves the performance of state-of-the-art ResNet-152 model by absolute value of 1.2% while only uses privileged information of 10% of the training images, confirming effectiveness of GoCNN on utilizing available privileged knowledge to train better CNNs.Comment: Proceedings of the IJCAI-1

Efficient CNN with uncorrelated Bag of Features pooling

Despite the superior performance of CNN, deploying them on low computational power devices is still limited as they are typically computationally expensive. One key cause of the high complexity is the connection between the convolution layers and the fully connected layers, which typically requires a high number of parameters. To alleviate this issue, Bag of Features (BoF) pooling has been recently proposed. BoF learns a dictionary, that is used to compile a histogram representation of the input. In this paper, we propose an approach that builds on top of BoF pooling to boost its efficiency by ensuring that the items of the learned dictionary are non-redundant. We propose an additional loss term, based on the pair-wise correlation of the items of the dictionary, which complements the standard loss to explicitly regularize the model to learn a more diverse and rich dictionary. The proposed strategy yields an efficient variant of BoF and further boosts its performance, without any additional parameters.Comment: 6 pages, 2 Figure

Single-channel EEG classification of sleep stages based on REM microstructure

Rapid-eye movement (REM) sleep, or paradoxical sleep, accounts for 20–25% of total night-time sleep in healthy adults and may be related, in pathological cases, to parasomnias. A large percentage of Parkinson's disease patients suffer from sleep disorders, including REM sleep behaviour disorder and hypokinesia; monitoring their sleep cycle and related activities would help to improve their quality of life. There is a need to accurately classify REM and the other stages of sleep in order to properly identify and monitor parasomnias. This study proposes a method for the identification of REM sleep from raw single-channel electroencephalogram data, employing novel features based on REM microstructures. Sleep stage classification was performed by means of random forest (RF) classifier, K-nearest neighbour (K-NN) classifier and random Under sampling boosted trees (RUSBoost); the classifiers were trained using a set of published and novel features. REM detection accuracy ranges from 89% to 92.7%, and the classifiers achieved a F-1 score (REM class) of about 0.83 (RF), 0.80 (K-NN), and 0.70 (RUSBoost). These methods provide encouraging outcomes in automatic sleep scoring and REM detection based on raw single-channel electroencephalogram, assessing the feasibility of a home sleep monitoring device with fewer channels

합성곱 커널 정규화를 위한 고른 각도분산방법

학위논문(박사) -- 서울대학교대학원 : 자연과학대학 수리과학부, 2022. 8. 강명주.In this thesis, we propose new convolutional kernel regularization methods. Along with the development of deep learning, there have been attempts to effectively regularize a convolutional layer, which is an important basic module of deep neural networks. Convolutional neural networks (CNN) are excellent at abstracting input data, but deepening causes gradient vanishing or explosion issues and produces redundant features. An approach to solve these issues is to directly regularize convolutional kernel weights of CNN. Its basic idea is to convert a convolutional kernel weight into a matrix and make the row or column vectors of the matrix orthogonal. However, this approach has some shortcomings. Firstly, it requires appropriate manipulation because overcomlete issue occurs when the number of vectors is larger than the dimension of vectors. As a method to deal with this issue, we define the concept of evenly dispersed state and propose PH0 and MST regularizations using this. Secondly, prior regularizations which enforce the Gram matrix of a matrix to be an identity matrix might not be an optimal approach for orthogonality of the matrix. We point out that these rather reduces the update of angles between some two vectors when two vectors are adjacent. Therefore, to complement for this issue, we propose EADK and EADC regularizations which update directly the angle. Through various experiments, we demonstrate that EADK and EADC regularizations outperform prior methods in some neural network architectures and, in particular, EADK has fast learning time.이 논문에서는 합성곱커널에 대한 새로운 정규화 방법들을 제안한다. 딥러닝의 발달과 더불어 신경망의 가장 기본적인 모듈인 합성곱 레이어를 효과적으로 정규화 하려는 시도들이 있어 왔다. 합성곱신경망는 인풋데이터를 추상화하는데 탁월하지만 네트워크의 깊이가 깊어지면 그레디언트 소멸이나 폭발 문제를 일으키고 중복된 피쳐들을 만든다. 이러한 문제들을 해결하기 위한 접근법 중 하나는 직접 합성곱 신경망의 합성곱커널을 직접 정규화 하는 것이다. 이 방법은 합성곱커널을 어떤 행렬로 변환하고 행렬의 행 또는 열들의 벡터들을 직교시키는 것이다. 그러나 이러한 접근법은 몇가지 단점이 있다. 첫째로, 벡터의 수가 벡터의 차원보다 많을 때는 모든 벡터를 직교화 시킬 수 없게 되므로 적절한 기법들을 필요로 한다. 이 문제를 다루기 위한 한 가지 방법으로 우리는 분산 상태라는 개념을 정의하고 이 개념을 활용한 PH0와 MST 정규화법을 제안한다. 둘째로, 그람행렬을 항등행렬로 근사시키는 방법을 사용하는 기존 정규화법이 벡터들을 직교화시키는 최적의 방법이 아닐 수 있다는 점이다. 즉, 기존의 정규화법이 두 벡터가 가까울 때는 오히려 각도의 업데이트를 줄이게 된다.따라서 이를 보완하기 위하여 우리는 각도를 직접 업데이트하는 EADK와 EADC 정규화법을 제안한다. 그리고 다양한 실험을 통해 EADK와 EADC 정규화법이 다수의 신경망구조에서 기존의 방법들보다 우수한 성능을 보이고 특히 EADK는 빠른 학습시간을 가진다는 것을 확인한다.Abstract i 1 Introduction 1 2 Preliminaries 4 2.1 Two Ways of Understanding CNN Layers as Matrix Operations 5 2.1.1 Kernel Matrix 6 2.1.2 Convolution Matrix 7 2.2 Soft Orthogonality 11 2.2.1 SO Regularization 11 2.2.2 DSO Regularization 12 2.3 Mutual Coherence 13 2.3.1 MC Regularization 13 2.4 Spectral Restricted Isometry Property 13 2.4.1 Restricted Isometry Property 13 2.4.2 SRIP Regularization 15 2.5 Orthogonal Convolutional Neural Networks 18 2.5.1 OCNN Regularizaiton 18 3 Topological Dispersing Regularizations 22 3.1 Evenly Dispersed State 23 3.1.1 Dispersing Vectors on Sphere 23 3.1.2 Evenly Dispersed State in the Real Projective Spaces 25 3.2 Persistent Homology Regularization 33 3.2.1 Cech and Vietoris-Rips Complexes 35 3.2.2 Persistent Homology 36 3.2.3 PH0 Regularization 38 3.3 Minimum Spanning Tree Regularization 39 3.3.1 Minimum Spanning Tree 39 3.3.2 MST Regularization 41 4 Evenly Angle Dispersing Regularizations 42 4.1 Analysis of Soft Orthogonality 43 4.1.1 Analysis of Soft Orthogonality 43 4.2 Evenly Angle Dispersing Regularizations 47 4.2.1 Evenly Angle Dispersing Regularization with Kernel Matrix 47 4.2.2 Evenly Angle Dispersing Regularization with Convolution Matrix 52 5 Algorithms & Experiments 54 5.1 Algorithms 55 5.1.1 PH0 and MST 55 5.1.2 EADK 57 5.1.3 EADC 58 5.2 Experiments 59 5.2.1 Analysis for Angle Dispersing 59 5.2.2 Experimental Setups 62 5.2.3 Classification Accuracy 68 5.2.4 Additional Experiments 76 6 Conclusion 78 The bibliography 80 Abstract (in Korean) 85박

Pruning Ternary Quantization

Inference time, model size, and accuracy are three key factors in deep model compression. Most of the existing work addresses these three key factors separately as it is difficult to optimize them all at the same time. For example, low-bit quantization aims at obtaining a faster model; weight sharing quantization aims at improving compression ratio and accuracy; and mixed-precision quantization aims at balancing accuracy and inference time. To simultaneously optimize bit-width, model size, and accuracy, we propose pruning ternary quantization (PTQ): a simple, effective, symmetric ternary quantization method. We integrate L2 normalization, pruning, and the weight decay term to reduce the weight discrepancy in the gradient estimator during quantization, thus producing highly compressed ternary weights. Our method brings the highest test accuracy and the highest compression ratio. For example, it produces a 939kb (49$\times$) 2bit ternary ResNet-18 model with only 4\% accuracy drop on the ImageNet dataset. It compresses 170MB Mask R-CNN to 5MB (34$\times$) with only 2.8\% average precision drop. Our method is verified on image classification, object detection/segmentation tasks with different network structures such as ResNet-18, ResNet-50, and MobileNetV2