Efficient Neural Architecture Search using Genetic Algorithm

NASNet and AmoebaNet are state-of-the-art neural architecture search systems that were able to achieve better accuracy than state-of-the-art human-made convolutional neural networks. Despite the innovation of the NASNet search space, it lacks the ability to express flexibility in terms of optimizing non-convolutional operation layers, such as batch normalization, activation, and dropout. These layers are hand designed by the architect prior to optimization, limiting the exploration possible for model architectures by narrowing down the search space. In addition, the NASNet search space can not allow for many non-classical optimization techniques to be applied as it lacks the ability to be expressed in a fixed-length, floating-point, multidimensional array. Lastly, both NASNet and AmoebaNet use an extensive amount of computation, both evaluating 20,000 models during optimization, consuming 2,000 GPU hours worth of computation. This work addresses these limitations by, first, changing the NASNet search space to include optimization of non-convolutional operation layers through the addition of a building block that allows for the optimization for the order and inclusion of these layers; second, proposing a fixed-length, floating-point, multidimensional array representation to allow other non-classical optimization techniques, such as particle swarm optimization, to be applied; and third, proposing an efficient genetic algorithm, while using state of-the-art techniques to reduce training comiv plexity. After only 1,300 models evaluated, consuming 190 GPU hours, evolving on the CIFAR-10 benchmark dataset, the best model configuration yielded a test accuracy of 94.6% with only 1.3 million parameters, and a test accuracy of 95.09% with only 5.17 million parameters, outperforming both ResNet110 and WideResNet. When transferring to the CIFAR-100 benchmark dataset, the best model configuration yielded a test accuracy of 71.1% with only 1.3 million parameters, and a test accuracy of 76.53% with only 5.17 million parameters

Automated Architecture Design for Deep Neural Networks

Machine learning has made tremendous progress in recent years and received large amounts of public attention. Though we are still far from designing a full artificially intelligent agent, machine learning has brought us many applications in which computers solve human learning tasks remarkably well. Much of this progress comes from a recent trend within machine learning, called deep learning. Deep learning models are responsible for many state-of-the-art applications of machine learning. Despite their success, deep learning models are hard to train, very difficult to understand, and often times so complex that training is only possible on very large GPU clusters. Lots of work has been done on enabling neural networks to learn efficiently. However, the design and architecture of such neural networks is often done manually through trial and error and expert knowledge. This thesis inspects different approaches, existing and novel, to automate the design of deep feedforward neural networks in an attempt to create less complex models with good performance that take away the burden of deciding on an architecture and make it more efficient to design and train such deep networks.Comment: Undergraduate Thesi

Master-slave Deep Architecture for Top-K Multi-armed Bandits with Non-linear Bandit Feedback and Diversity Constraints

We propose a novel master-slave architecture to solve the top-$K$ combinatorial multi-armed bandits problem with non-linear bandit feedback and diversity constraints, which, to the best of our knowledge, is the first combinatorial bandits setting considering diversity constraints under bandit feedback. Specifically, to efficiently explore the combinatorial and constrained action space, we introduce six slave models with distinguished merits to generate diversified samples well balancing rewards and constraints as well as efficiency. Moreover, we propose teacher learning based optimization and the policy co-training technique to boost the performance of the multiple slave models. The master model then collects the elite samples provided by the slave models and selects the best sample estimated by a neural contextual UCB-based network to make a decision with a trade-off between exploration and exploitation. Thanks to the elaborate design of slave models, the co-training mechanism among slave models, and the novel interactions between the master and slave models, our approach significantly surpasses existing state-of-the-art algorithms in both synthetic and real datasets for recommendation tasks. The code is available at: \url{https://github.com/huanghanchi/Master-slave-Algorithm-for-Top-K-Bandits}.Comment: IEEE Transactions on Neural Networks and Learning System

The State-of-the-Art Survey on Optimization Methods for Cyber-physical Networks

Cyber-Physical Systems (CPS) are increasingly complex and frequently integrated into modern societies via critical infrastructure systems, products, and services. Consequently, there is a need for reliable functionality of these complex systems under various scenarios, from physical failures due to aging, through to cyber attacks. Indeed, the development of effective strategies to restore disrupted infrastructure systems continues to be a major challenge. Hitherto, there have been an increasing number of papers evaluating cyber-physical infrastructures, yet a comprehensive review focusing on mathematical modeling and different optimization methods is still lacking. Thus, this review paper appraises the literature on optimization techniques for CPS facing disruption, to synthesize key findings on the current methods in this domain. A total of 108 relevant research papers are reviewed following an extensive assessment of all major scientific databases. The main mathematical modeling practices and optimization methods are identified for both deterministic and stochastic formulations, categorizing them based on the solution approach (exact, heuristic, meta-heuristic), objective function, and network size. We also perform keyword clustering and bibliographic coupling analyses to summarize the current research trends. Future research needs in terms of the scalability of optimization algorithms are discussed. Overall, there is a need to shift towards more scalable optimization solution algorithms, empowered by data-driven methods and machine learning, to provide reliable decision-support systems for decision-makers and practitioners