Lossy video compression is commonly used when transmitting and storing video
data. Unified video codecs (e.g., H.264 or H.265) remain the de facto standard,
despite the availability of advanced (neural) compression approaches.
Transmitting videos in the face of dynamic network bandwidth conditions
requires video codecs to adapt to vastly different compression strengths. Rate
control modules augment the codec's compression such that bandwidth constraints
are satisfied and video distortion is minimized. While, both standard video
codes and their rate control modules are developed to minimize video distortion
w.r.t. human quality assessment, preserving the downstream performance of deep
vision models is not considered. In this paper, we present the first end-to-end
learnable deep video codec control considering both bandwidth constraints and
downstream vision performance, while not breaking existing standardization. We
demonstrate for two common vision tasks (semantic segmentation and optical flow
estimation) and on two different datasets that our deep codec control better
preserves downstream performance than using 2-pass average bit rate control
while meeting dynamic bandwidth constraints and adhering to standardizations.Comment: 22 pages, 26 figures, 6 table