International audienceThe promising compression efficiency improvement of Versatile Video Coding (VVC) compared to High Efficiency Video Coding (HEVC) [1] comes at the cost of a non-negligible encoder-side complexity. The largely increased complexity overhead is a possible obstacle towards its industrial implementation. Many papers have proposed acceleration methods for VVC. Still, a better understanding of VVC complexity, especially related to new partitions and coding tools, is desirable to help the design of new and better acceleration methods. For this purpose, statistical analyses have been conducted, with a focus on Coding Unit (CU) sizes and inter coding modes

Abdoli, Mohsen

Guillemot, Christine

Guionnet, Thomas

Liu, Yiqun

Roumy, Aline

INRIA a CCSD electronic archive server

STATISTICAL ANALYSIS OF INTER CODING IN VVC TEST MODEL (VTM)Yiqun Liu⋆† Mohsen Abdoli⋆ Thomas Guionnet⋆ Christine Guillemot† Aline Roumy†⋆ ATEME, Rennes, France† INRIA, Rennes, FranceABSTRACTThe promising compression efficiency improvement of Versa-tile Video Coding (VVC) compared to High Efficiency VideoCoding (HEVC) [1] comes at the cost of a non-negligibleencoder-side complexity. The largely increased complexityoverhead is a possible obstacle towards its industrial imple-mentation. Many papers have proposed acceleration methodsfor VVC. Still, a better understanding of VVC complexity, es-pecially related to new partitions and coding tools, is desirableto help the design of new and better acceleration methods. Forthis purpose, statistical analyses have been conducted, with afocus on Coding Unit (CU) sizes and inter coding modes.Index Terms— Versatile Video Coding, Inter Coding,Rate Distortion Optimization, Complexity Analysis1. INTRODUCTIONStandardization of VVC in 2020 has brought significant im-provement to the capacity of video compression in terms ofbitrate saving. Precisely, it offers 50% compression efficiency[2] compared to one of the most efficient video compressionstandards HEVC. This improvement by VVC is principallyowed to newly adopted coding techniques. Most decoder de-vices could afford the additional complexity brought by thesenovel coding techniques taking into account current hardwarecapacities of these devices. Precisely, studies in [3] has shownthat the relative decoder complexity of VVC is from 150% to200% compared to HEVC in different configurations.On the contrary, it is far from affordable for real time ap-plication for VVC encoder side since industrial encoding ap-plications have strict limitations in terms of resources and ex-ecution time. Tests in [4] on VVC reference software VVCTest Model (VTM) 7 show that the encoding time of VVCis 5x, 7x and 37x times of HEVC encoding in Low-Delay(LD), Random-Access (RA) and All-Intra (AI) configurationsrespectively. Hence, it is vital to develop acceleration algo-rithms or methods to largely reduce the encoding complexitywhile preserving the majority of encoding efficiency. Papersof complexity analysis could help researchers to have before-hand a clear understanding of what is happening inside a VVCencoder (e.g. VTM), and what potentially interests them fortheir design process of acceleration method.Various papers have contributed to the complexity analy-sis of VVC. In [5], a detailed complexity analysis based onVVC intra prediction tools has been performed. Pakdaman etal. in [3] have broke down the encoding process into encod-ing modules such as motion estimation, intra prediction, en-tropy coding etc. and then analysed the complexity partitionof modules in multiple encoding configurations. [6] reviewscomplexity aspects of the different modules of the VVC stan-dard and provide a complexity breakdown of these modules ina more precise way. In [4], VVC and HEVC are compared interms of rate-distortion and complexity analysis. These afore-mentioned papers present complexity analysis at the level ofencoding modules for inter coding. Our paper is the first toprovide an analysis from CU sizes and coding modes perspec-tive for inter coding in VVC.(sss, Intra)(sss, Affine)(sssss, Inter)(sss, RegMerge)……IntraInterRegMergeMergeRegSkipAffineMergeAffineMergeAffineSkipMergeGeoMergeGeoSkipMergeCiipVTMshortcut?Partitioning Test modesVTM RDO BitstreamFig. 1. High-level view of the RDO process involving parti-tioning, test modes and possible VTM shortcuts.In this paper, a statistical analysis of the Rate-DistortionOptimization (RDO) process in inter coding of VTM-15.0 ispresented. The main focus is put on the statistics of two fac-tors: CU sizes and inter coding modes. The goal is to pro-vide useful information for related works aiming at speedingup inter coding in VVC. The rest of this paper is organizedas follows. Section 2 presents a summary of VVC specifica-tion in terms of available block sizes and coding modes. InSection 3, all statistical observations are presented, which arelater analyzed and concluded in Section 4.2. RDO OF INTER CODING IN VTMTo depict the RDO process in VTM, there exists numerouscoding parameters such as Intra Prediction Mode (IPM) ofintra coding, Motion Vector (MV) representation mode andchoice of transform etc. Nevertheless, if we ignore these triv-ial parameters, the RDO process could be described as thesearch for the best trade-off between bit rate and distortion.More precisely, this search is executed on different codingmodes of different CU sizes. Therefore, in this paper thestatistics of block sizes and coding modes are jointly consid-ered.As presented in Fig.1, various CU sizes are the result ofpartitioning in the RDO process. The partitioning consistsof splitting the CU of size 128x128 recursively by five splitmodes, namely Quaternary Tree (QT) split, Horizontal BinaryTree (HBT) split, Vertical Binary Tree (VBT) split, Horizon-tal Tenary Tree (HTT) split, Vertical Tenary Tree (VTT) split.Comparing to codec HEVC in which only QT is available forpartitioning, the added directional splits gives rise to a largervariety of CU sizes. In VVC block size is authorised if itswidths and heights are any power of two between 4 and 128,except for sizes 128×4, 128×8, 128×16 and 128×32. It isworth-noting that same CU size could be obtained by differ-ent series of split modes.For each CU, eleven coding modes are available. Two ofthese modes, namely hash-based inter prediction and palettemodes are not enabled in the Joint Video Exploration Team(JVET) Common Test Condition (CTC). Hence, they arenot included in the analysis of this paper. Three principalmodes coding modes are intra (“intra”), inter mode with newMV (“me”) and inter mode with MV of multi resolutions(“aff”). The remaining modes are merged modes indicat-ing the coded MV are shared among neighboring blocks.These modes could be then divided into two groups of skipmodes and non-skip modes. The residual would not be en-coded and transmitted under skip mode. The non-skip modesinclude regular merge (“mrgreg”), merge with geometricpartitioning (“mrggeo”), merge with Combined Intra-InterPrediction (CIIP) (“mrgciip”) and merge with affine (“mr-gaff”), while the merge modes consist of regular merge withskip(“mrgregskp”) merge with skipped geometric partition-ing (“mrggeoskp”) and finally merge with skipped affine MV(“mrgaffskp”).Though VVC is computationally expensive, JVET grouphas already adopted diverse shortcuts or conditional early ex-its as presented in [7] in VTM. We have deactivated existingshortcuts in VTM-15.0 and evaluated its performance. Theassessment was that the complexity will increase by 138%.Moreover, the performance of the tested encoder (i.e. with-out shortcuts) is 0.76% better than the reference encoder (i.e.with shortcuts), in terms of BD-rate. This trade-off might beinterpreted as an indicator that the shortcuts in VTM are ef-ficient in terms of identifying useless tests and partitioningdepths. Many of shortcuts are based on history of the testedsplit modes. Nonetheless, aspects of CU sizes and codingmodes are overlooked.02040608010012022 27 32 37PercentageQP values2160p 1080p 720pFig. 2. Encoding time in different QPs comparing to QP 223. STATISTICSOur major purpose in the following analysis is to find the CUsizes and/or coding modes with relatively high complexityoccupation and low selection rate in RDO process. Fromthe perspective of encoder acceleration, CU sizes or cod-ing modes with higher complexity portion and significantlylower selection rate are more favorable to the design of blocksize/coding mode based acceleration rules. The size or modewith larger complexity portion has more potential in acceler-ating. Lower selection rate indicates it is less likely to makewrong decisions when skipping the RDO of current CU sizeor mode.All our experiments and analyses are conducted on thefirst 64 frames of CTC sequences in RandomAccess Group OfPicture 32 (RAGOP32) configuration in which intra framesare excluded. Exceptionally, Fig.2 is based on sequences inClass A, B, E of CTC.02468101214161820Portion of encoding time (%)CU sizesQP 22 QP 37Fig. 3. Complexity distribution for block sizes in QP22 andQP37The encoding complexity for Chrominance channel onlyaccounts for a small part comparing to Luminance channel.Thus we focus on Luminance channel in the remaining ofthe paper. From a high-level view, the encoding complexityof VTM significantly depends on the selected QuantizationParameter (QP). Particularly, larger QP values tend to havefaster encoding with VTM. Figure 2 is obtained by measur-ing encoding times of sequences of resolution 2160p, 1080pand 720p in QP 22, 27, 32 and 37. Then the average ratiobetween the encoding time of each QP and that of QP 22 iscalculated. It shows that encoding time at QP 22 could be fivetimes as much as QP 37.Fig.3 shows the percentage distribution of encoding timespent on different block sizes in QP 22 and QP 37. Besidesthe fact that the overall encoding time is higher for QP 22, itcan be observed that relatively higher portion of the time inQP 22 is passed on smaller block sizes. This could partly beexplained by existing shortcuts in VTM disallowing exces-sively small blocks in QP 37. We could declare that largerblock sizes are in general more crucial to speeding up the par-titioning process, especially block 64x64 and 128x128 whichtake in total from 20% complexity in QP 22 to 30% in QP 37.In another test, the selection percentage of different blocksizes are computed. This metric is defined as the ratio be-tween the total number of times it is selected and the totalnumber of times a block size is tested. Fig. 4 shows the valuesof this metric in QP 22 and 37. As we can see from this figure,larger CU sizes correspond to larger selection rates comparedwith smaller blocks. Another worth-noting phenomenon isthat the selection rate of 128x128 increase dramatically from14% in QP 22 to 37% QP 37.Fig. 4. Selection rate for different block sizesCombining the above figures, it is observed that CU sizessuch as 16x8, 8x16 and 16x16 are sizes with low selectionrate and high complexity. For example, 16x16 blocks has thesame level of complexity, while its selection rate is half of32x32 blocks in QP 22.Fig. 5. Pie chart of complexities of inter coding modesTo take one step further in the statistical analysis of intercoding, we present how different inter coding modes are in-volved in RDO search. The first experiment in Fig.5 presentthe distribution of encoding time at inter coding mode level inQP 22 and QP 37.In general, intra, mrgaff, mrgaffskip mrgreg and mr-gregskip are main contributors to encoding time. Fig.6 pro-vide selection rates as the ratio between number of selectedinter coding modes and number of tested modes. We couldremark that mrgaff, mrgaffskip and mrgreg these three modeshave minor selection rates though they are accounting fornearly half of the complexity.Fig. 6. Selection rate of inter coding modesFig.7 shows the distribution of inter modes for encodedCUs of different sizes. The fact that the aforementioned threecoding modes are less chosen could also be proved by thisfigure. We could find out that the number of mrgregskip isdominant for most CU sizes and that the number of thesethree modes are relatively small, which is coherent with Fig.6.From Fig.7 we could also observe that smaller blocks tendto be encoded with intra mode. Another remark is that theskip modes are more frequently selected for larger CUs. It isprobably because the residual of larger CUs is more expen-sive to be encoded. In addition, mrgreg mode have higherchance to be selected in smaller QP which is in contrast tomrgaffskip and mrggeoskip. For some CU sizes, we couldmake shortcuts or condition for early termination for RDO ofinter modes which are rarely selected to speed up the encod-ing process, such as mrgaff mode with merely 1.9% selectedfor bloc 128x128.4. ANALYSIS AND CONCLUSIONIn this study, complexity analysis of CU sizes and inter codingmodes has been combined with selection rate analysis. FromCU size perspective, block sizes with high complexity gener-ally correspond to high selection rate. Block sizes 128x128and 64x64 are responsible for one third of the complexity.Fig. 7. Stacked chart of selected inter modes in different block sizesIn addition, block sizes like 16x8, 8x16, 16x16 exhibit rela-tively low selection rate while requiring a significant share ofthe overall complexity. Therefore, these block sizes are rel-evant targets for acceleration algorithms. From coding modeperspective, mrgaff, mrgaffskip and mrgreg tend to be lesslikely to be selected. Thus, shortcuts dedicated to skippingadaptively these coding modes might be promising. Shortcuton coding modes and partitioning acceleration method are indifferent scopes. The former focus on reducing number ofCU for RDO. The latter speeds up the RDO for CU of certainsizes. The combination of these two could lead to a largerspeed-up of encoding.5. REFERENCES[1] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, andThomas Wiegand, “Overview of the high efficiency videocoding (hevc) standard,” IEEE Transactions on circuitsand systems for video technology, vol. 22, no. 12, pp.1649–1668, 2012.[2] B. Bross et al., “Overview of the versatile video coding(VVC) standard and its applications,” IEEE Trans. onCircuits and Systems for Video Technology, vol. 31, no.10, pp. 3736–3764, 2021.[3] Farhad Pakdaman, Mohammad Ali Adelimanesh, Mon-cef Gabbouj, and Mahmoud Reza Hashemi, “Complex-ity analysis of next-generation vvc encoding and decod-ing,” in 2020 IEEE International Conference on ImageProcessing (ICIP). IEEE, 2020, pp. 3134–3138.[4] Alexandre Mercat, Arttu Mäkinen, Joose Sainio, AriLemmetti, Marko Viitanen, and Jarno Vanne, “Compar-ative rate-distortion-complexity analysis of vvc and hevcvideo codecs,” IEEE Access, vol. 9, pp. 67813–67828,2021.[5] Mário Saldanha, Gustavo Sanchez, César Marcon, andLuciano Agostini, “Complexity analysis of vvc intra cod-ing,” in 2020 IEEE International Conference on ImageProcessing (ICIP). IEEE, 2020, pp. 3119–3123.[6] Frank Bossen, Karsten Sühring, Adam Wieckowski, andShan Liu, “Vvc complexity and software implementationanalysis,” IEEE Transactions on Circuits and Systems forVideo Technology, vol. 31, no. 10, pp. 3765–3778, 2021.[7] Adam Wieckowski, Jackie Ma, Heiko Schwarz, DetlevMarpe, and Thomas Wiegand, “Fast partitioning deci-sion strategies for the upcoming versatile video coding(vvc) standard,” in 2019 IEEE International Conferenceon Image Processing (ICIP). IEEE, 2019, pp. 4130–4134.

Statistical analysis of inter coding in vvc test model (VTM)

https://hal.inria.fr/hal-03808292v1/file/ICIP2022_paper.pdf

Statistical analysis of inter coding in vvc test model (VTM)

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server