1,105 research outputs found
A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
Neuron pruning is an efficient method to compress the network into a slimmer
one for reducing the computational cost and storage overhead. Most of
state-of-the-art results are obtained in a layer-by-layer optimization mode. It
discards the unimportant input neurons and uses the survived ones to
reconstruct the output neurons approaching to the original ones in a
layer-by-layer manner. However, an unnoticed problem arises that the
information loss is accumulated as layer increases since the survived neurons
still do not encode the entire information as before. A better alternative is
to propagate the entire useful information to reconstruct the pruned layer
instead of directly discarding the less important neurons. To this end, we
propose a novel Layer Decomposition-Recomposition Framework (LDRF) for neuron
pruning, by which each layer's output information is recovered in an embedding
space and then propagated to reconstruct the following pruned layers with
useful information preserved. We mainly conduct our experiments on ILSVRC-12
benchmark with VGG-16 and ResNet-50. What should be emphasized is that our
results before end-to-end fine-tuning are significantly superior owing to the
information-preserving property of our proposed framework.With end-to-end
fine-tuning, we achieve state-of-the-art results of 5.13x and 3x speed-up with
only 0.5% and 0.65% top-5 accuracy drop respectively, which outperform the
existing neuron pruning methods.Comment: accepted by AAAI19 as ora
Towards Optimal Randomized Strategies in Adversarial Example Game
The vulnerability of deep neural network models to adversarial example
attacks is a practical challenge in many artificial intelligence applications.
A recent line of work shows that the use of randomization in adversarial
training is the key to find optimal strategies against adversarial example
attacks. However, in a fully randomized setting where both the defender and the
attacker can use randomized strategies, there are no efficient algorithm for
finding such an optimal strategy. To fill the gap, we propose the first
algorithm of its kind, called FRAT, which models the problem with a new
infinite-dimensional continuous-time flow on probability distribution spaces.
FRAT maintains a lightweight mixture of models for the defender, with
flexibility to efficiently update mixing weights and model parameters at each
iteration. Furthermore, FRAT utilizes lightweight sampling subroutines to
construct a random strategy for the attacker. We prove that the continuous-time
limit of FRAT converges to a mixed Nash equilibria in a zero-sum game formed by
a defender and an attacker. Experimental results also demonstrate the
efficiency of FRAT on CIFAR-10 and CIFAR-100 datasets.Comment: Extended version of paper https://doi.org/10.1609/aaai.v37i9.26247
which appeared in AAAI 202
MSAT: Matrix stability analysis tool for shock-capturing schemes
The simulation of supersonic or hypersonic flows often suffers from numerical
shock instabilities if the flow field contains strong shocks, limiting the
further application of shock-capturing schemes. In this paper, we develop the
unified matrix stability analysis method for schemes with three-point stencils
and present MSAT, an open-source tool to quantitatively analyze the shock
instability problem. Based on the finite-volume approach on the structured
grid, MSAT can be employed to investigate the mechanism of the shock
instability problem, evaluate the robustness of numerical schemes, and then
help to develop robust schemes. Also, MSAT has the ability to analyze the
practical simulation of supersonic or hypersonic flows, evaluate whether it
will suffer from shock instabilities, and then assist in selecting appropriate
numerical schemes accordingly. As a result, MSAT is a helpful tool that can
investigate the shock instability problem and help to cure it.Comment: 18 pages, 6 figure
Neural Inheritance Relation Guided One-Shot Layer Assignment Search
Layer assignment is seldom picked out as an independent research topic in
neural architecture search. In this paper, for the first time, we
systematically investigate the impact of different layer assignments to the
network performance by building an architecture dataset of layer assignment on
CIFAR-100. Through analyzing this dataset, we discover a neural inheritance
relation among the networks with different layer assignments, that is, the
optimal layer assignments for deeper networks always inherit from those for
shallow networks. Inspired by this neural inheritance relation, we propose an
efficient one-shot layer assignment search approach via inherited sampling.
Specifically, the optimal layer assignment searched in the shallow network can
be provided as a strong sampling priori to train and search the deeper ones in
supernet, which extremely reduces the network search space. Comprehensive
experiments carried out on CIFAR-100 illustrate the efficiency of our proposed
method. Our search results are strongly consistent with the optimal ones
directly selected from the architecture dataset. To further confirm the
generalization of our proposed method, we also conduct experiments on
Tiny-ImageNet and ImageNet. Our searched results are remarkably superior to the
handcrafted ones under the unchanged computational budgets. The neural
inheritance relation discovered in this paper can provide insights to the
universal neural architecture search.Comment: AAAI202
Efficient Cross-Device Federated Learning Algorithms for Minimax Problems
In many machine learning applications where massive and privacy-sensitive
data are generated on numerous mobile or IoT devices, collecting data in a
centralized location may be prohibitive. Thus, it is increasingly attractive to
estimate parameters over mobile or IoT devices while keeping data localized.
Such learning setting is known as cross-device federated learning. In this
paper, we propose the first theoretically guaranteed algorithms for general
minimax problems in the cross-device federated learning setting. Our algorithms
require only a fraction of devices in each round of training, which overcomes
the difficulty introduced by the low availability of devices. The communication
overhead is further reduced by performing multiple local update steps on
clients before communication with the server, and global gradient estimates are
leveraged to correct the bias in local update directions introduced by data
heterogeneity. By developing analyses based on novel potential functions, we
establish theoretical convergence guarantees for our algorithms. Experimental
results on AUC maximization, robust adversarial network training, and GAN
training tasks demonstrate the efficiency of our algorithms
Nanoindentation characterization on local plastic response of Ti-6Al-4V under high-load spherical indentation
After high-load spherical indentation treatment, the variations of hardness on the plastic zone of Ti-6Al-4V were investigated via nanoindentation method. The hardness within the center of plastic zone was measured by nanoindenter, and the magnitude decreased gradually along the depth, which were caused by the different extent of plastic deformation under the residual imprint. The microstructure of indentation were observed using scanning electron microscope (SEM) before and after surface etching, and the results showed that the microhardness revealed the average hardness of α and β phases of Ti-6Al-4V. The maximum hardness reached 6.438 GPa in the depth of 132 μm. In addition, the two and three dimensional contour profiles of residual imprint introduced by high-load spherical indentation were measured by the white-light interferometer and the shape of residual imprint was obtained. All results were discussed in detail
- …