30,828 research outputs found
Concrete Dropout
Dropout is used as a practical tool to obtain uncertainty estimates in large
vision models and reinforcement learning (RL) tasks. But to obtain
well-calibrated uncertainty estimates, a grid-search over the dropout
probabilities is necessary - a prohibitive operation with large models, and an
impossible one with RL. We propose a new dropout variant which gives improved
performance and better calibrated uncertainties. Relying on recent developments
in Bayesian deep learning, we use a continuous relaxation of dropout's discrete
masks. Together with a principled optimisation objective, this allows for
automatic tuning of the dropout probability in large models, and as a result
faster experimentation cycles. In RL this allows the agent to adapt its
uncertainty dynamically as more data is observed. We analyse the proposed
variant extensively on a range of tasks, and give insights into common practice
in the field where larger dropout probabilities are often used in deeper model
layers
Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
When approaching a novel visual recognition problem in a specialized image
domain, a common strategy is to start with a pre-trained deep neural network
and fine-tune it to the specialized domain. If the target domain covers a
smaller visual space than the source domain used for pre-training (e.g.
ImageNet), the fine-tuned network is likely to be over-parameterized. However,
applying network pruning as a post-processing step to reduce the memory
requirements has drawbacks: fine-tuning and pruning are performed
independently; pruning parameters are set once and cannot adapt over time; and
the highly parameterized nature of state-of-the-art pruning methods make it
prohibitive to manually search the pruning parameter space for deep networks,
leading to coarse approximations. We propose a principled method for jointly
fine-tuning and compressing a pre-trained convolutional network that overcomes
these limitations. Experiments on two specialized image domains (remote sensing
images and describable textures) demonstrate the validity of the proposed
approach.Comment: BMVC 2017 ora
- …