In this paper, we investigate the problem of bit allocation in Neural Video
Compression (NVC). First, we reveal that a recent bit allocation approach
claimed to be optimal is, in fact, sub-optimal due to its implementation.
Specifically, we find that its sub-optimality lies in the improper application
of semi-amortized variational inference (SAVI) on latent with non-factorized
variational posterior. Then, we show that the corrected version of SAVI on
non-factorized latent requires recursively applying back-propagating through
gradient ascent, based on which we derive the corrected optimal bit allocation
algorithm. Due to the computational in-feasibility of the corrected bit
allocation, we design an efficient approximation to make it practical.
Empirical results show that our proposed correction significantly improves the
incorrect bit allocation in terms of R-D performance and bitrate error, and
outperforms all other bit allocation methods by a large margin. The source code
is provided in the supplementary material