477 research outputs found
Modality-Agnostic Variational Compression of Implicit Neural Representations
We introduce a modality-agnostic neural data compression algorithm based on a
functional view of data and parameterised as an Implicit Neural Representation
(INR). Bridging the gap between latent coding and sparsity, we obtain compact
latent representations which are non-linearly mapped to a soft gating mechanism
capable of specialising a shared INR base network to each data item through
subnetwork selection. After obtaining a dataset of such compact latent
representations, we directly optimise the rate/distortion trade-off in this
modality-agnostic space using non-linear transform coding. We term this method
Variational Compression of Implicit Neural Representation (VC-INR) and show
both improved performance given the same representational capacity pre
quantisation while also outperforming previous quantisation schemes used for
other INR-based techniques. Our experiments demonstrate strong results over a
large set of diverse data modalities using the same algorithm without any
modality-specific inductive biases. We show results on images, climate data, 3D
shapes and scenes as well as audio and video, introducing VC-INR as the first
INR-based method to outperform codecs as well-known and diverse as JPEG 2000,
MP3 and AVC/HEVC on their respective modalities
Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
Explaining recommendations enables users to understand whether recommended
items are relevant to their needs and has been shown to increase their trust in
the system. More generally, if designing explainable machine learning models is
key to check the sanity and robustness of a decision process and improve their
efficiency, it however remains a challenge for complex architectures,
especially deep neural networks that are often deemed "black-box". In this
paper, we propose a novel formulation of interpretable deep neural networks for
the attribution task. Differently to popular post-hoc methods, our approach is
interpretable by design. Using masked weights, hidden features can be deeply
attributed, split into several input-restricted sub-networks and trained as a
boosted mixture of experts. Experimental results on synthetic data and
real-world recommendation tasks demonstrate that our method enables to build
models achieving close predictive performances to their non-interpretable
counterparts, while providing informative attribution interpretations.Comment: 14th ACM Conference on Recommender Systems (RecSys '20
Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks
The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, sometimes even better than, the original dense networks. Sparsity promises to reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field
- …