47 research outputs found
Complexity Analysis Of Next-Generation VVC Encoding and Decoding
While the next generation video compression standard, Versatile Video Coding
(VVC), provides a superior compression efficiency, its computational complexity
dramatically increases. This paper thoroughly analyzes this complexity for both
encoder and decoder of VVC Test Model 6, by quantifying the complexity
break-down for each coding tool and measuring the complexity and memory
requirements for VVC encoding/decoding. These extensive analyses are performed
for six video sequences of 720p, 1080p, and 2160p, under Low-Delay (LD),
Random-Access (RA), and All-Intra (AI) conditions (a total of 320
encoding/decoding). Results indicate that the VVC encoder and decoder are 5x
and 1.5x more complex compared to HEVC in LD, and 31x and 1.8x in AI,
respectively. Detailed analysis of coding tools reveals that in LD on average,
motion estimation tools with 53%, transformation and quantization with 22%, and
entropy coding with 7% dominate the encoding complexity. In decoding, loop
filters with 30%, motion compensation with 20%, and entropy decoding with 16%,
are the most complex modules. Moreover, the required memory bandwidth for VVC
encoding/decoding are measured through memory profiling, which are 30x and 3x
of HEVC. The reported results and insights are a guide for future research and
implementations of energy-efficient VVC encoder/decoder.Comment: IEEE ICIP 202
A comprehensive video codec comparison
In this paper, we compare the video codecs AV1 (version 1.0.0-2242 from August 2019), HEVC (HM and x265), AVC (x264), the exploration software JEM which is based on HEVC, and the VVC (successor of HEVC) test model VTM (version 4.0 from February 2019) under two fair and balanced configurations: All Intra for the assessment of intra coding and Maximum Coding Efficiency with all codecs being tuned for their best coding efficiency settings. VTM achieves the highest coding efficiency in both configurations, followed by JEM and AV1. The worst coding efficiency is achieved by x264 and x265, even in the placebo preset for highest coding efficiency. AV1 gained a lot in terms of coding efficiency compared to previous versions and now outperforms HM by 24% BD-Rate gains. VTM gains 5% over AV1 in terms of BD-Rates. By reporting separate numbers for JVET and AOM test sequences, it is ensured that no bias in the test sequences exists. When comparing only intra coding tools, it is observed that the complexity increases exponentially for linearly increasing coding efficiency
Multiscale Motion-Aware and Spatial-Temporal-Channel Contextual Coding Network for Learned Video Compression
Recently, learned video compression has achieved exciting performance.
Following the traditional hybrid prediction coding framework, most learned
methods generally adopt the motion estimation motion compensation (MEMC) method
to remove inter-frame redundancy. However, inaccurate motion vector (MV)
usually lead to the distortion of reconstructed frame. In addition, most
approaches ignore the spatial and channel redundancy. To solve above problems,
we propose a motion-aware and spatial-temporal-channel contextual coding based
video compression network (MASTC-VC), which learns the latent representation
and uses variational autoencoders (VAEs) to capture the characteristics of
intra-frame pixels and inter-frame motion. Specifically, we design a multiscale
motion-aware module (MS-MAM) to estimate spatial-temporal-channel consistent
motion vector by utilizing the multiscale motion prediction information in a
coarse-to-fine way. On the top of it, we further propose a
spatial-temporal-channel contextual module (STCCM), which explores the
correlation of latent representation to reduce the bit consumption from
spatial, temporal and channel aspects respectively. Comprehensive experiments
show that our proposed MASTC-VC is surprior to previous state-of-the-art (SOTA)
methods on three public benchmark datasets. More specifically, our method
brings average 10.15\% BD-rate savings against H.265/HEVC (HM-16.20) in PSNR
metric and average 23.93\% BD-rate savings against H.266/VVC (VTM-13.2) in
MS-SSIM metric.Comment: 12pages,12 figure
Offline and Online Optical Flow Enhancement for Deep Video Compression
Video compression relies heavily on exploiting the temporal redundancy
between video frames, which is usually achieved by estimating and using the
motion information. The motion information is represented as optical flows in
most of the existing deep video compression networks. Indeed, these networks
often adopt pre-trained optical flow estimation networks for motion estimation.
The optical flows, however, may be less suitable for video compression due to
the following two factors. First, the optical flow estimation networks were
trained to perform inter-frame prediction as accurately as possible, but the
optical flows themselves may cost too many bits to encode. Second, the optical
flow estimation networks were trained on synthetic data, and may not generalize
well enough to real-world videos. We address the twofold limitations by
enhancing the optical flows in two stages: offline and online. In the offline
stage, we fine-tune a trained optical flow estimation network with the motion
information provided by a traditional (non-deep) video compression scheme, e.g.
H.266/VVC, as we believe the motion information of H.266/VVC achieves a better
rate-distortion trade-off. In the online stage, we further optimize the latent
features of the optical flows with a gradient descent-based algorithm for the
video to be compressed, so as to enhance the adaptivity of the optical flows.
We conduct experiments on a state-of-the-art deep video compression scheme,
DCVC. Experimental results demonstrate that the proposed offline and online
enhancement together achieves on average 12.8% bitrate saving on the tested
videos, without increasing the model or computational complexity of the decoder
side.Comment: 9 pages, 6 figure
Improved CNN-based Learning of Interpolation Filters for Low-Complexity Inter Prediction in Video Coding
The versatility of recent machine learning approaches makes them ideal for
improvement of next generation video compression solutions. Unfortunately,
these approaches typically bring significant increases in computational
complexity and are difficult to interpret into explainable models, affecting
their potential for implementation within practical video coding applications.
This paper introduces a novel explainable neural network-based inter-prediction
scheme, to improve the interpolation of reference samples needed for fractional
precision motion compensation. The approach requires a single neural network to
be trained from which a full quarter-pixel interpolation filter set is derived,
as the network is easily interpretable due to its linear structure. A novel
training framework enables each network branch to resemble a specific
fractional shift. This practical solution makes it very efficient to use
alongside conventional video coding schemes. When implemented in the context of
the state-of-the-art Versatile Video Coding (VVC) test model, 0.77%, 1.27% and
2.25% BD-rate savings can be achieved on average for lower resolution sequences
under the random access, low-delay B and low-delay P configurations,
respectively, while the complexity of the learned interpolation schemes is
significantly reduced compared to the interpolation with full CNNs.Comment: IEEE Open Journal of Signal Processing Special Issue on Applied AI
and Machine Learning for Video Coding and Streaming, June 202
Fusion-Based Versatile Video Coding Intra Prediction Algorithm with Template Matching and Linear Prediction
The new generation video coding standard Versatile Video Coding (VVC) has adopted many novel technologies to improve compression performance, and consequently, remarkable results have been achieved. In practical applications, less data, in terms of bitrate, would reduce the burden of the sensors and improve their performance. Hence, to further enhance the intra compression performance of VVC, we propose a fusion-based intra prediction algorithm in this paper. Specifically, to better predict areas with similar texture information, we propose a fusion-based adaptive template matching method, which directly takes the error between reference and objective templates into account. Furthermore, to better utilize the correlation between reference pixels and the pixels to be predicted, we propose a fusion-based linear prediction method, which can compensate for the deficiency of single linear prediction. We implemented our algorithm on top of the VVC Test Model (VTM) 9.1. When compared with the VVC, our proposed fusion-based algorithm saves a bitrate of 0.89%, 0.84%, and 0.90% on average for the Y, Cb, and Cr components, respectively. In addition, when compared with some other existing works, our algorithm showed superior performance in bitrate savings