12 research outputs found
SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
In an effort to improve generalization in deep learning and automate the
process of learning rate scheduling, we propose SALR: a sharpness-aware
learning rate update technique designed to recover flat minimizers. Our method
dynamically updates the learning rate of gradient-based optimizers based on the
local sharpness of the loss function. This allows optimizers to automatically
increase learning rates at sharp valleys to increase the chance of escaping
them. We demonstrate the effectiveness of SALR when adopted by various
algorithms over a broad range of networks. Our experiments indicate that SALR
improves generalization, converges faster, and drives solutions to
significantly flatter regions