Utilizing Class Information for Deep Network Representation Shaping

Statistical characteristics of deep network representations, such as sparsity and correlation, are known to be relevant to the performance and interpretability of deep learning. When a statistical characteristic is desired, often an adequate regularizer can be designed and applied during the training phase. Typically, such a regularizer aims to manipulate a statistical characteristic over all classes together. For classification tasks, however, it might be advantageous to enforce the desired characteristic per class such that different classes can be better distinguished. Motivated by the idea, we design two class-wise regularizers that explicitly utilize class information: class-wise Covariance Regularizer (cw-CR) and class-wise Variance Regularizer (cw-VR). cw-CR targets to reduce the covariance of representations calculated from the same class samples for encouraging feature independence. cw-VR is similar, but variance instead of covariance is targeted to improve feature compactness. For the sake of completeness, their counterparts without using class information, Covariance Regularizer (CR) and Variance Regularizer (VR), are considered together. The four regularizers are conceptually simple and computationally very efficient, and the visualization shows that the regularizers indeed perform distinct representation shaping. In terms of classification performance, significant improvements over the baseline and L1/L2 weight regularization methods were found for 21 out of 22 tasks over popular benchmark datasets. In particular, cw-VR achieved the best performance for 13 tasks including ResNet-32/110.Comment: Published in AAAI 201

Basic Enhancement Strategies When Using Bayesian Optimization for Hyperparameter Tuning of Deep Neural Networks

Compared to the traditional machine learning models, deep neural networks (DNN) are known to be highly sensitive to the choice of hyperparameters. While the required time and effort for manual tuning has been rapidly decreasing for the well developed and commonly used DNN architectures, undoubtedly DNN hyperparameter optimization will continue to be a major burden whenever a new DNN architecture needs to be designed, a new task needs to be solved, a new dataset needs to be addressed, or an existing DNN needs to be improved further. For hyperparameter optimization of general machine learning problems, numerous automated solutions have been developed where some of the most popular solutions are based on Bayesian Optimization (BO). In this work, we analyze four fundamental strategies for enhancing BO when it is used for DNN hyperparameter optimization. Specifically, diversification, early termination, parallelization, and cost function transformation are investigated. Based on the analysis, we provide a simple yet robust algorithm for DNN hyperparameter optimization - DEEP-BO (Diversified, Early-termination-Enabled, and Parallel Bayesian Optimization). When evaluated over six DNN benchmarks, DEEP-BO mostly outperformed well-known solutions including GP-Hedge, BOHB, and the speed-up variants that use Median Stopping Rule or Learning Curve Extrapolation. In fact, DEEP-BO consistently provided the top, or at least close to the top, performance over all the benchmark types that we have tested. This indicates that DEEP-BO is a robust solution compared to the existing solutions. The DEEP-BO code is publicly available at <uri>https://github.com/snu-adsl/DEEP-BO</uri>

The validation of Implicit Association Test measures for smartphone and Internet addiction in at-risk children and adolescents

Background: Potential concerns are increasing that smartphone and Internet addictions may have deleterious effects on the mental health. Despite the recognition of the important role that implicit associations may have over explicit processes in addiction, such implicit associations have not been comprehensively investigated with respect to Internet addiction. Therefore, we modified the Implicit Association Test (IAT) for smartphone and Internet addictions and investigated its validity in children and adolescents. Methods: In this experimental study, 78 at-risk children and adolescents ranging in age from 7 to 17 years completed an IAT modified with pictures captured from the most popular Internet games among youth. Furthermore, measures of Internet and smartphone addictions, mental health and problem behaviors, impulsive tendencies, self-esteem, daily stress, and quality of life were assessed simultaneously. Results: Significant correlations were found between IAT D2SD scores and standardized scales for Internet (r = .28, p < .05) and smartphone (r = .33, p < .01) addictions. There were no significant correlations between IAT parameters and other scales measuring the constructs that are less relevant to the features of addiction, such as daily stress levels, impulsivity, and quality of life. Multiple regression analysis revealed that the IAT D2SD was independently and positively associated with smartphone addiction (p = .03) after controlling for other clinical correlates. Conclusions: This study demonstrated good convergent and discriminant validity of this IAT as a novel measurement relating to Internet and smartphone addictions. Further longitudinal and prospective studies are needed to evaluate its potential utility in clinical and community settings

Regulation of synaptic Rac1 activity, long-term potentiation maintenance, and learning and memory by BCR and ABR Rac GTPase-activating proteins

Rho family small GTPases are important regulators of neuronal development. Defective Rho regulation causes nervous system dysfunctions including mental retardation and Alzheimer's disease. Rac1, a member of the Rho family, regulates dendritic spines and excitatory synapses, but relatively little is known about how synaptic Rac1 is negatively regulated. Breakpoint cluster region (BCR) is a Rac GTPase-activating protein known to form a fusion protein with the c-Abl tyrosine kinase in Philadelphia chromosome-positive chronic myelogenous leukemia. Despite the fact that BCR mRNAs are abundantly expressed in the brain, the neural functions of BCR protein have remained obscure. We report here that BCR and its close relative active BCR-related (ABR) localize at excitatory synapses and directly interact with PSD-95, an abundant postsynaptic scaffolding protein. Mice deficient for BCR or ABR show enhanced basal Rac1 activity but only a small increase in spine density. Importantly, mice lacking BCR or ABR exhibit a marked decrease in the maintenance, but not induction, of long-term potentiation, and show impaired spatial and object recognition memory. These results suggest that BCR and ABR have novel roles in the regulation of synaptic Rac1 signaling, synaptic plasticity, and learning and memory, and that excessive Rac1 activity negatively affects synaptic and cognitive functions.This work was supported by the National Creative Research Initiative Program of the Korean Ministry of Education, Science and Technology (E.K.), Neuroscience Program Grant 2009-0081468 (S.-Y.C.), 21st Century Frontier R&D Program in Neuroscience Grant 2009K001284 (H.K.), Basic Science Research Program Grant R13-2008-009-01001-0 (Y.C.B.), and United States Public Health Service Grants HL071945 (J.G.) and HL060231 (J.G., N.H.)