In this paper, we consider the problem of bit allocation in neural video
compression (NVC). Due to the frame reference structure, current NVC methods
using the same R-D (Rate-Distortion) trade-off parameter λ for all
frames are suboptimal, which brings the need for bit allocation. Unlike
previous methods based on heuristic and empirical R-D models, we propose to
solve this problem by gradient-based optimization. Specifically, we first
propose a continuous bit implementation method based on Semi-Amortized
Variational Inference (SAVI). Then, we propose a pixel-level implicit bit
allocation method using iterative optimization by changing the SAVI target.
Moreover, we derive the precise R-D model based on the differentiable trait of
NVC. And we show the optimality of our method by proofing its equivalence to
the bit allocation with precise R-D model. Experimental results show that our
approach significantly improves NVC methods and outperforms existing bit
allocation methods. Our approach is plug-and-play for all differentiable NVC
methods, and it can be directly adopted on existing pre-trained models