SMASH: One-Shot Model Architecture Search through HyperNetworks

Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMAS

Molding the Knowledge in Modular Neural Networks

Problem description. The learning of monolithic neural networks becomes harder with growing network size. Likewise the knowledge obtained while learning becomes harder to extract. Such disadvantages are caused by a lack of internal structure, that by its presence would reduce the degrees of freedom in evolving to a training target. A suitable internal structure with respect to modular network construction as well as to nodal discrimination is required. Details on the grouping and selection of nodes can sometimes be concluded from the characteristics of the application area; otherwise a comprehensive search within the solution space is necessary

A scalable saliency-based Feature selection method with instance level information

Classic feature selection techniques remove those features that are either irrelevant or redundant, achieving a subset of relevant features that help to provide a better knowledge extraction. This allows the creation of compact models that are easier to interpret. Most of these techniques work over the whole dataset, but they are unable to provide the user with successful information when only instance information is needed. In short, given any example, classic feature selection algorithms do not give any information about which the most relevant information is, regarding this sample. This work aims to overcome this handicap by developing a novel feature selection method, called Saliency-based Feature Selection (SFS), based in deep-learning saliency techniques. Our experimental results will prove that this algorithm can be successfully used not only in Neural Networks, but also under any given architecture trained by using Gradient Descent techniques

KNOWLEDGE-BASED NEURAL NETWORK FOR LINE FLOW CONTINGENCY SELECTION AND RANKING

The Line flow Contingency Selection and Ranking (CS & R) is performed to rank the critical contingencies in order of their severity. An Artificial Neural Network based method for MW security assessment corresponding to line outage events have been reported by various authors in the literature. One way to provide an understanding of the behaviour of Neural Networks is to extract rules that can be provided to the user. The domain knowledge (fuzzy rules extracted from Multi-layer Perceptron model trained by Back Propagation algorithm) is integrated into a Neural Network for fast and accurate CS & R in an IEEE 14-bus system, for unknown load patterns and are found to be suitable for on-line applications at Energy Management Centers. The system user is provided with the capability to determine the set of conditions under which a line-outage is critical, and if critical, then how severe it is, thereby providing some degree of transparency of the ANN solution