This is the final version. Available from Transactions on Machine Learning Research via the link in this recordWe make contributions towards improving adaptive-optimizer performance. Our improvements are
based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly, we
show that the particular placement of the parameter ϵ within the update expressions of AdaBelief
reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum.
Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve
the above goal, we perform mutual layerwise vector projections between the gradient gt and its
first momentum mt before using them to estimate the second momentum. The new optimization
method is referred to as Aida. Thirdly, extensive experimental results show that Aida outperforms
nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image
classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods
when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher
validation accuracies than AdaBelief for training ResNet18 over ImageNet. Our implementation is
available at https://github.com/guoqiang-zhang-x/Aida-OptimizerNippon Telegraph and Telephone Corporatio
