7 research outputs found
Pruning Convolutional Neural Networks with Self-Supervision
Convolutional neural networks trained without supervision come close to
matching performance with supervised pre-training, but sometimes at the cost of
an even higher number of parameters. Extracting subnetworks from these large
unsupervised convnets with preserved performance is of particular interest to
make them less computationally intensive. Typical pruning methods operate
during training on a task while trying to maintain the performance of the
pruned network on the same task. However, in self-supervised feature learning,
the training objective is agnostic on the representation transferability to
downstream tasks. Thus, preserving performance for this objective does not
ensure that the pruned subnetwork remains effective for solving downstream
tasks. In this work, we investigate the use of standard pruning methods,
developed primarily for supervised learning, for networks trained without
labels (i.e. on self-supervised tasks). We show that pruned masks obtained with
or without labels reach comparable performance when re-trained on labels,
suggesting that pruning operates similarly for self-supervised and supervised
learning. Interestingly, we also find that pruning preserves the transfer
performance of self-supervised subnetwork representations
Pruning Convolutional Neural Networks with Self-Supervision
Convolutional neural networks trained without supervision come close to matching performance with supervised pre-training, but sometimes at the cost of an even higher number of parameters. Extracting subnetworks from these large unsupervised convnets with preserved performance is of particular interest to make them less computationally intensive. Typical pruning methods operate during training on a task while trying to maintain the performance of the pruned network on the same task. However, in self-supervised feature learning, the training objective is agnostic on the representation transferability to downstream tasks. Thus, preserving performance for this objective does not ensure that the pruned subnetwork remains effective for solving downstream tasks. In this work, we investigate the use of standard pruning methods, developed primarily for supervised learning, for networks trained without labels (i.e. on self-supervised tasks). We show that pruned masks obtained with or without labels reach comparable performance when retrained on labels, suggesting that pruning operates similarly for self-supervised and supervised learning. Interestingly, we also find that pruning preserves the transfer performance of self-supervised subnetwork representations
Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices
Seismic full waveform inversion (FWI) is a widely used technique in
geophysics for inferring subsurface structures from seismic data. And
InversionNet is one of the most successful data-driven machine learning models
that is applied to seismic FWI. However, the high computing costs to run
InversionNet have made it challenging to be efficiently deployed on edge
devices that are usually resource-constrained. Therefore, we propose to employ
the structured pruning algorithm to get a lightweight version of InversionNet,
which can make an efficient inference on edge devices. And we also made a
prototype with Raspberry Pi to run the lightweight InversionNet. Experimental
results show that the pruned InversionNet can achieve up to 98.2 % reduction in
computing resources with moderate model performance degradation
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Unsupervised image representations have significantly reduced the gap with
supervised pretraining, notably with the recent achievements of contrastive
learning methods. These contrastive methods typically work online and rely on a
large number of explicit pairwise feature comparisons, which is computationally
challenging. In this paper, we propose an online algorithm, SwAV, that takes
advantage of contrastive methods without requiring to compute pairwise
comparisons. Specifically, our method simultaneously clusters the data while
enforcing consistency between cluster assignments produced for different
augmentations (or views) of the same image, instead of comparing features
directly as in contrastive learning. Simply put, we use a swapped prediction
mechanism where we predict the cluster assignment of a view from the
representation of another view. Our method can be trained with large and small
batches and can scale to unlimited amounts of data. Compared to previous
contrastive methods, our method is more memory efficient since it does not
require a large memory bank or a special momentum network. In addition, we also
propose a new data augmentation strategy, multi-crop, that uses a mix of views
with different resolutions in place of two full-resolution views, without
increasing the memory or compute requirements much. We validate our findings by
achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as
surpassing supervised pretraining on all the considered transfer tasks.Comment: NeurIPS 202
A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
Modern deep neural networks, particularly recent large language models, come
with massive model sizes that require significant computational and storage
resources. To enable the deployment of modern models on resource-constrained
environments and accelerate inference time, researchers have increasingly
explored pruning techniques as a popular research direction in neural network
compression. However, there is a dearth of up-to-date comprehensive review
papers on pruning. To address this issue, in this survey, we provide a
comprehensive review of existing research works on deep neural network pruning
in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to
prune, and 4) fusion of pruning and other compression techniques. We then
provide a thorough comparative analysis of seven pairs of contrast settings for
pruning (e.g., unstructured/structured) and explore emerging topics, including
post-training pruning, different levels of supervision for pruning, and broader
applications (e.g., adversarial robustness) to shed light on the commonalities
and differences of existing methods and lay the foundation for further method
development. To facilitate future research, we build a curated collection of
datasets, networks, and evaluations on different applications. Finally, we
provide some valuable recommendations on selecting pruning methods and prospect
promising research directions. We build a repository at
https://github.com/hrcheng1066/awesome-pruning
Pruning Convolutional Neural Networks with Self-Supervision
Convolutional neural networks trained without supervision come close to matching performance with supervised pre-training, but sometimes at the cost of an even higher number of parameters. Extracting subnetworks from these large unsupervised convnets with preserved performance is of particular interest to make them less computationally intensive. Typical pruning methods operate during training on a task while trying to maintain the performance of the pruned network on the same task. However, in self-supervised feature learning, the training objective is agnostic on the representation transferability to downstream tasks. Thus, preserving performance for this objective does not ensure that the pruned subnetwork remains effective for solving downstream tasks. In this work, we investigate the use of standard pruning methods, developed primarily for supervised learning, for networks trained without labels (i.e. on self-supervised tasks). We show that pruned masks obtained with or without labels reach comparable performance when retrained on labels, suggesting that pruning operates similarly for self-supervised and supervised learning. Interestingly, we also find that pruning preserves the transfer performance of self-supervised subnetwork representations