A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks

Neuron pruning is an efficient method to compress the network into a slimmer one for reducing the computational cost and storage overhead. Most of state-of-the-art results are obtained in a layer-by-layer optimization mode. It discards the unimportant input neurons and uses the survived ones to reconstruct the output neurons approaching to the original ones in a layer-by-layer manner. However, an unnoticed problem arises that the information loss is accumulated as layer increases since the survived neurons still do not encode the entire information as before. A better alternative is to propagate the entire useful information to reconstruct the pruned layer instead of directly discarding the less important neurons. To this end, we propose a novel Layer Decomposition-Recomposition Framework (LDRF) for neuron pruning, by which each layer's output information is recovered in an embedding space and then propagated to reconstruct the following pruned layers with useful information preserved. We mainly conduct our experiments on ILSVRC-12 benchmark with VGG-16 and ResNet-50. What should be emphasized is that our results before end-to-end fine-tuning are significantly superior owing to the information-preserving property of our proposed framework.With end-to-end fine-tuning, we achieve state-of-the-art results of 5.13x and 3x speed-up with only 0.5% and 0.65% top-5 accuracy drop respectively, which outperform the existing neuron pruning methods.Comment: accepted by AAAI19 as ora

Towards Optimal Randomized Strategies in Adversarial Example Game

The vulnerability of deep neural network models to adversarial example attacks is a practical challenge in many artificial intelligence applications. A recent line of work shows that the use of randomization in adversarial training is the key to find optimal strategies against adversarial example attacks. However, in a fully randomized setting where both the defender and the attacker can use randomized strategies, there are no efficient algorithm for finding such an optimal strategy. To fill the gap, we propose the first algorithm of its kind, called FRAT, which models the problem with a new infinite-dimensional continuous-time flow on probability distribution spaces. FRAT maintains a lightweight mixture of models for the defender, with flexibility to efficiently update mixing weights and model parameters at each iteration. Furthermore, FRAT utilizes lightweight sampling subroutines to construct a random strategy for the attacker. We prove that the continuous-time limit of FRAT converges to a mixed Nash equilibria in a zero-sum game formed by a defender and an attacker. Experimental results also demonstrate the efficiency of FRAT on CIFAR-10 and CIFAR-100 datasets.Comment: Extended version of paper https://doi.org/10.1609/aaai.v37i9.26247 which appeared in AAAI 202

MSAT: Matrix stability analysis tool for shock-capturing schemes

The simulation of supersonic or hypersonic flows often suffers from numerical shock instabilities if the flow field contains strong shocks, limiting the further application of shock-capturing schemes. In this paper, we develop the unified matrix stability analysis method for schemes with three-point stencils and present MSAT, an open-source tool to quantitatively analyze the shock instability problem. Based on the finite-volume approach on the structured grid, MSAT can be employed to investigate the mechanism of the shock instability problem, evaluate the robustness of numerical schemes, and then help to develop robust schemes. Also, MSAT has the ability to analyze the practical simulation of supersonic or hypersonic flows, evaluate whether it will suffer from shock instabilities, and then assist in selecting appropriate numerical schemes accordingly. As a result, MSAT is a helpful tool that can investigate the shock instability problem and help to cure it.Comment: 18 pages, 6 figure

Neural Inheritance Relation Guided One-Shot Layer Assignment Search

Layer assignment is seldom picked out as an independent research topic in neural architecture search. In this paper, for the first time, we systematically investigate the impact of different layer assignments to the network performance by building an architecture dataset of layer assignment on CIFAR-100. Through analyzing this dataset, we discover a neural inheritance relation among the networks with different layer assignments, that is, the optimal layer assignments for deeper networks always inherit from those for shallow networks. Inspired by this neural inheritance relation, we propose an efficient one-shot layer assignment search approach via inherited sampling. Specifically, the optimal layer assignment searched in the shallow network can be provided as a strong sampling priori to train and search the deeper ones in supernet, which extremely reduces the network search space. Comprehensive experiments carried out on CIFAR-100 illustrate the efficiency of our proposed method. Our search results are strongly consistent with the optimal ones directly selected from the architecture dataset. To further confirm the generalization of our proposed method, we also conduct experiments on Tiny-ImageNet and ImageNet. Our searched results are remarkably superior to the handcrafted ones under the unchanged computational budgets. The neural inheritance relation discovered in this paper can provide insights to the universal neural architecture search.Comment: AAAI202

Efficient Cross-Device Federated Learning Algorithms for Minimax Problems

In many machine learning applications where massive and privacy-sensitive data are generated on numerous mobile or IoT devices, collecting data in a centralized location may be prohibitive. Thus, it is increasingly attractive to estimate parameters over mobile or IoT devices while keeping data localized. Such learning setting is known as cross-device federated learning. In this paper, we propose the first theoretically guaranteed algorithms for general minimax problems in the cross-device federated learning setting. Our algorithms require only a fraction of devices in each round of training, which overcomes the difficulty introduced by the low availability of devices. The communication overhead is further reduced by performing multiple local update steps on clients before communication with the server, and global gradient estimates are leveraged to correct the bias in local update directions introduced by data heterogeneity. By developing analyses based on novel potential functions, we establish theoretical convergence guarantees for our algorithms. Experimental results on AUC maximization, robust adversarial network training, and GAN training tasks demonstrate the efficiency of our algorithms

Nanoindentation characterization on local plastic response of Ti-6Al-4V under high-load spherical indentation

After high-load spherical indentation treatment, the variations of hardness on the plastic zone of Ti-6Al-4V were investigated via nanoindentation method. The hardness within the center of plastic zone was measured by nanoindenter, and the magnitude decreased gradually along the depth, which were caused by the different extent of plastic deformation under the residual imprint. The microstructure of indentation were observed using scanning electron microscope (SEM) before and after surface etching, and the results showed that the microhardness revealed the average hardness of α and β phases of Ti-6Al-4V. The maximum hardness reached 6.438 GPa in the depth of 132 μm. In addition, the two and three dimensional contour profiles of residual imprint introduced by high-load spherical indentation were measured by the white-light interferometer and the shape of residual imprint was obtained. All results were discussed in detail