2,084 research outputs found
Surrogate Lagrangian Relaxation: A Path To Retrain-free Deep Neural Network Pruning
Network pruning is a widely used technique to reduce computation cost and
model size for deep neural networks. However, the typical three-stage pipeline
significantly increases the overall training time. In this paper, we develop a
systematic weight-pruning optimization approach based on Surrogate Lagrangian
relaxation, which is tailored to overcome difficulties caused by the discrete
nature of the weight-pruning problem. We prove that our method ensures fast
convergence of the model compression problem, and the convergence of the SLR is
accelerated by using quadratic penalties. Model parameters obtained by SLR
during the training phase are much closer to their optimal values as compared
to those obtained by other state-of-the-art methods. We evaluate our method on
image classification tasks using CIFAR-10 and ImageNet with state-of-the-art
MLP-Mixer, Swin Transformer, and VGG-16, ResNet-18, ResNet-50 and ResNet-110,
MobileNetV2. We also evaluate object detection and segmentation tasks on COCO,
KITTI benchmark, and TuSimple lane detection dataset using a variety of models.
Experimental results demonstrate that our SLR-based weight-pruning optimization
approach achieves a higher compression rate than state-of-the-art methods under
the same accuracy requirement and also can achieve higher accuracy under the
same compression rate requirement. Under classification tasks, our SLR approach
converges to the desired accuracy faster on both of the datasets.
Under object detection and segmentation tasks, SLR also converges
faster to the desired accuracy. Further, our SLR achieves high model accuracy
even at the hard-pruning stage without retraining, which reduces the
traditional three-stage pruning into a two-stage process. Given a limited
budget of retraining epochs, our approach quickly recovers the model's
accuracy.Comment: arXiv admin note: text overlap with arXiv:2012.1007
Applications of Simple Markov Models to Computer Vision
In this report we advocate the use of computationally simple algorithms for computer vision, operating in parallel. The design of these algorithms is based on physical constraints present in the image and object spaces. In particular, we discuss the design, implementation, and performance of a Markov Random Field based algorithm for low level segmentation. In addition to having a simple and fast implementation, the algorithm is flexible enough to allow intensity information to be fused with motion and edge information from other sources
Integration of traditional imaging, expert systems, and neural network techniques for enhanced recognition of handwritten information
Includes bibliographical references (p. 33-37).Research supported by the I.F.S.R.C. at M.I.T.Amar Gupta, John Riordan, Evelyn Roman
Bottleneck Potentials in {Markov Random Fields}
We consider general discrete Markov Random Fields(MRFs) with additional bottleneck potentials which penalize the maximum (instead of the sum) over local potential value taken by the MRF-assignment. Bottleneck potentials or analogous constructions have been considered in (i) combinatorial optimization (e.g. bottleneck shortest path problem, the minimum bottleneck spanning tree problem, bottleneck function minimization in greedoids), (ii) inverse problems with -norm regularization, and (iii) valued constraint satisfaction on the -pre-semirings. Bottleneck potentials for general discrete MRFs are a natural generalization of the above direction of modeling work to Maximum-A-Posteriori (MAP) inference in MRFs. To this end, we propose MRFs whose objective consists of two parts: terms that factorize according to (i) , i.e. potentials as in plain MRFs, and (ii) , i.e. bottleneck potentials. To solve the ensuing inference problem, we propose high-quality relaxations and efficient algorithms for solving them. We empirically show efficacy of our approach on large scale seismic horizon tracking problems
Integrating Learning and Reasoning with Deep Logic Models
Deep learning is very effective at jointly learning feature representations
and classification models, especially when dealing with high dimensional input
patterns. Probabilistic logic reasoning, on the other hand, is capable to take
consistent and robust decisions in complex environments. The integration of
deep learning and logic reasoning is still an open-research problem and it is
considered to be the key for the development of real intelligent agents. This
paper presents Deep Logic Models, which are deep graphical models integrating
deep learning and logic reasoning both for learning and inference. Deep Logic
Models create an end-to-end differentiable architecture, where deep learners
are embedded into a network implementing a continuous relaxation of the logic
knowledge. The learning process allows to jointly learn the weights of the deep
learners and the meta-parameters controlling the high-level reasoning. The
experimental results show that the proposed methodology overtakes the
limitations of the other approaches that have been proposed to bridge deep
learning and reasoning
Relational Neural Machines
Deep learning has been shown to achieve impressive results in several tasks
where a large amount of training data is available. However, deep learning
solely focuses on the accuracy of the predictions, neglecting the reasoning
process leading to a decision, which is a major issue in life-critical
applications. Probabilistic logic reasoning allows to exploit both statistical
regularities and specific domain expertise to perform reasoning under
uncertainty, but its scalability and brittle integration with the layers
processing the sensory data have greatly limited its applications. For these
reasons, combining deep architectures and probabilistic logic reasoning is a
fundamental goal towards the development of intelligent agents operating in
complex environments. This paper presents Relational Neural Machines, a novel
framework allowing to jointly train the parameters of the learners and of a
First--Order Logic based reasoner. A Relational Neural Machine is able to
recover both classical learning from supervised data in case of pure
sub-symbolic learning, and Markov Logic Networks in case of pure symbolic
reasoning, while allowing to jointly train and perform inference in hybrid
learning tasks. Proper algorithmic solutions are devised to make learning and
inference tractable in large-scale problems. The experiments show promising
results in different relational tasks
- …