This work presents a hardware-aware search algorithm for HEVC motion estimation. Implications of several decisions in search algorithm are considered with respect to their hardware implementation costs (in terms of area and bandwidth). Proposed algorithm provides 3X logic area in integer motion estimation, 16% on-chip reference buffer area and 47X maximum off-chip bandwidth savings when compared to HM-3.0 fast search algorithm.Texas Instruments Incorporate

Chandrakasan, Anantha P.

Sinangil, Mahmut

Sinangil, Mahmut E.

Sze, Vivienne

Zhou, Minhua

Mahmut E. Sinangil

Anantha P. Chandrakasan

Vivienne Sze

Minhua Zhou

Crossref

Hardware-aware motion estimation search algorithm development for high-efficiency video coding (HEVC) standard

DSpace@MIT

HARDWARE-AWARE MOTION ESTIMATION SEARCH ALGORITHM DEVELOPMENTFOR HIGH-EFFICIENCY VIDEO CODING (HEVC) STANDARDMahmut E. Sinangil∗Anantha P. ChandrakasanMassachusetts Institute of TechnologyCambridge MAVivienne SzeMinhua ZhouTexas Instruments Inc.Dallas TXABSTRACTThis work presents a hardware-aware search algorithm forHEVC motion estimation. Implications of several decisionsin search algorithm are considered with respect to their hard-ware implementation costs (in terms of area and bandwidth).Proposed algorithm provides 3X logic area in integer motionestimation, 16% on-chip reference buffer area and 47X max-imum off-chip bandwidth savings when compared to HM-3.0fast search algorithm.Index Terms— HEVC, motion estimation, hardware-aware, motion search algorithm1. INTRODUCTIONHigh-Efficiency Video Coding (HEVC) is a new video com-pression standard being standardized by the JCT-VC (jointcollaborative team on video coding) established by ISO/IEOMPEG and ITU-T [1]. HEVC has a design goal of achieving50% coding gain over AVC/H.264 High Profile. For this pur-pose, several coding efficiency enhancement tools have beenproposed to this new standard.Motion estimation is one of the most critical blocks invideo encoding. Search algorithm performed in motionestimation determines coding efficiency as well as powerconsumption through number of computations and on-chipand off-chip bandwidths. Searching among a larger num-ber of candidates leads to finding better motion vectors andimproves coding efficiency. However, larger number of can-didates often results in more switching activity and databandwidth and hence more active power consumption. More-over, for a fixed cycle budget, searching through a largernumber of candidates require a higher level of parallelismand results in larger hardware area. It is also important tonote that searching strategy can have multiple stages wherethe output of the previous stage can affect where and howthe next stages will be performed. This can also limit thethroughput for a fixed cycle budget and is not desirable forhardware implementation. Hence, search algorithm develop-ment should consider its coding efficiency impact as well as∗Funding provided by Texas Instruments Inc.its implications on hardware implementation.Among many new tools in HEVC, coding quad-tree struc-ture is designed for providing a modular coding structure. InHEVC, a frame is divided into LCUs (largest coding units)and an LCU is further divided into CUs (coding units) in aquad-tree structure. Currently, LCU size can be as large as64x64 pixels and SCU (smallest coding unit) size can be assmall as 8x8. If a CU is not divided into smaller CUs, it ispredicted with one of several PU (prediction unit) types.For inter prediction, PU types can be 2Nx2N, 2NxN,Nx2N or NxN where 2Nx2N corresponds to the size of theCU. If asymmetric motion partitions (AMP) are used, non-square PUs for inter also include 2NxnU, 2NxnD, nLx2N andnRx2N. NxN is only allowed at the SCU level not to presentredundancy.This paper presents a HEVC motion estimation search al-gorithm suitable for hardware implementation. First, a searchpattern consisting of two parallel stages is implemented to en-able parallel evaluation of candidates. Secondly, a scheme al-lowing a single shared search range across parallel enginesis proposed to reduce on- and off-chip bandwidth. Lastly,a pre-fetch scheme is implemented to offset the latency ofthe off-chip memory holding reference frame data. The restof this paper is organized as follows: Section 2 provides anoverview of the motion estimation architecture suitable forHEVC and its quad-tree structure. Section 3 presents the par-allelized search strategy and its implementation. Section 4focuses on the search range sharing between engines and sec-tion 5 talks about the pre-fetching strategy to offset latency ofthe off-chip memory. Finally, section 6 summarizes the ben-efits of the proposed search algorithm in terms of hardwarearea and data bandwidth and the cost in terms of coding loss.2. OVERVIEW OF HEVC MOTION ESTIMATIONARCHITECTUREFor maximum coding efficiency in motion estimation, allthe block sizes (CUs and PUs in CUs) should undergo in-dependent motion searches sequentially as implemented inthe HEVC test model (HM) [2]. In hardware implementa-tion, however, various simplifications have been discussedfor previous standards to allow search range and cost calcu-lations to be shared between different block sizes [3, 4, 5].These changes cause motion vector predictor calculations tobe based on estimations rather than correct motion vectorsand cost coding efficiency degradation.In this work, an architecture suitable for HEVC motionestimation allowing sequential processing of blocks is con-sidered to provide highest coding efficiency. Search algo-rithm development in this work focuses on reducing num-ber of computations and data bandwidth. However, proposedmodifications do not require a change in sequential process-ing of blocks and AMVP (advanced motion vector predictor)calculations continue to be exact.Since cost calculations and motion search for differentblocks are separate and independent, separate and indepen-dent engines for different block sizes are necessary in imple-mentation. Then, to find the best combination of CU sizes andPU types, output of each engine can be combined to make sizeand type decisions starting at smallest CU and then going tolarger CU sizes. Fig. 1 shows the architecture of the HEVCmotion estimation module considered in this work. It consistsof four parallel CU engines supporting only 2Nx2N PU type.Hence, supported block sizes are 64x64, 32x32, 16x16 and8x8. This architecture can be generalized to support differentPU types and consequently different block sizes as well.Fig. 1. Architecture of the HEVC motion estimation moduleconsidered in this work. Four parallel CU engines from 64x64down to 8x8 are included with only 2Nx2N PU type.3. SEARCH STRATEGYFast search strategy used in HM-3.0 starts the search aroundthe best AMVP and involves many inter-dependent stages ininteger motion estimation (IME). For example, the result ofthe initial diamond search determines if a sub-sampled rastersearch is performed or not. In hardware implementation, thisdependency increases complexity and often results in extracycles or extra hardware to account for the worst-case con-ditions. For, fractional motion estimation (FME) refinement,best integer motion vector is taken as search center and thenhalf and quarter pixel refinements are performed respectively.Recent work focused on search algorithms that can be par-allelized in hardware implementation [6, 7]. In this work, atwo-stage search strategy is used for IME where each stagecan be independently performed in parallel. Fig. 2 showsIME search patterns used in each of the stages. First, searchcenter is decided by comparing AMVP list entries (up to threeentries) and [0,0]. During this comparison, SAD (sum of ab-solute differences) cost is used. After search center is deter-mined, two stage search is started.8px±64±64±64±644pxCompare CostsDetermine Search CenterFinal MVAMVP List[0,0]3Fig. 2. Two stage search approach used in this work. Stagesare independent of each other and can be performed in parallelin hardware.The first stage consists of a coarse search covering ±64by checking every 8th candidate in each direction. This stagecan capture a change in motion or irregular motion patternsthat cannot be tracked by AMVP. The second stage performsa more localized three step search around the ±7 window ofthe search center. This stage can capture regular motion. Itis important to note that the AMVP calculation for all blocksuses exact motion vectors of the neighbors and AMVP is ac-curate and hence can track motion well in most cases.The proposed IME search strategy checks a total of 285candidates for each block as opposed to up to 850 candidatesthat are checked in fast search strategy in HM-3.0. This re-sults in roughly 3X hardware area reduction in IME for thesame throughput constraint. Actual savings might be largerin implementation because of the additional complexity dueto inter-dependent stages of HM-3.0 algorithm.Lastly, for FME, search strategy of HM is used where re-finement is performed around the best integer motion vector.Separate Buf.(4 x 1R1W)Shared Buf.(1 x 4R1W)Memory Size 109.2KB 39KBEst. Array Area 0.75mm2 0.61mm2Est. Perip. Area 0.5mm2 0.44mm2Est. Total Area 1.25mm2 1.05mm2Table 1. Area comparison of shared and separate referencebuffers. Estimates are based on a generic 65nm CMOS tech-nology.4. SHARED SEARCH RANGE ACROSS PARALLELENGINESAs explained in Section 2, for maximum coding efficiency,each engine in Fig. 1 is running independently and has sep-arate reference buffers holding pixels for independent searchwindows. This approach is expensive in terms of area andexternal memory bandwidth. Sharing the on-chip referencebuffer across parallel engines can be significantly more effi-cient for practical implementations. This section will focuson benefits and cost of sharing search range across parallelengines.4.1. Area Impact of Reference BuffersIn the case of separate reference buffers with ±64 searchrange for each engine, the implementation in Fig. 1 requiresfour 1R1W port memories with 39KB, 27.5KB, 22.5KB and20.2KB sizes for 64x64, 32x32, 16x16 and 8x8 engines re-spectively. The sizes of these memories are determined bythe block size, search range and the extra pixels at the edgesof the search range for FME. In practical implementations, alarger size for each buffer might be used to allow data sharingbetween consecutive blocks. Total area consumed by thesefour memories will be roughly 1.25mm2 in a 65nm CMOStechnology as shown in Table 1. It should be noted that thisarea is for storing the pixels on the chip for a single directionand single reference frame.In the case of a shared reference buffer with ±64 searchrange, the size is determined by the largest block size. Inthis case a single 39KB memory is needed with 4R1W ports.Although the bit-cell area and some peripheral componentsneed to be expanded to support multiple read ports, the overallarea can be smaller as shown in Table 1. Hence, shared searchrange across parallel engines result in 16% area savings forthe implementation considered in Fig. 1.4.2. Data Bandwidth Impact of Reference BuffersWith independent motion searches, each engine might havedifferent search centers and consequently access differentparts of the reference frame as the search window. Table 2shows maximum and average off-chip bandwidth for 64x64,Block Size Max. Off-Chip BW Avg. Off-Chip BW64x64 2.2GB/s 1.49GB/s32x32 6.4GB/s 3.64GB/s16x16 20.9GB/s 7.62GB/s8x8 75.6GB/s 17.47GB/sTotal 105.1GB/s 30.22GB/sTable 2. Maximum and average off-chip bandwidth require-ment for different block sizes (search range is ±64).32x32, 16x16 and 8x8 engines. The upper limit on thebandwidth is calculated by assuming that the entire on-chipreference buffer needs to be updated between consecutiveblocks and hence no data re-use is possible. The total maxi-mum off-chip bandwidth is 112 GB/s for supporting 4Kx2Kresolution at 30fps assuming a search range of ±64. Aver-age bandwidth number assumes close to 100% data re-usebetween consecutive LCUs. However, it should be noted that100% data re-use is not possible due to sequential processingof the blocks.In the case of a shared reference window across engines,the maximum bandwidth is equal to the maximum bandwidthof the 64x64 block since the size of the shared search windowis determined by the largest block size. Hence, sharing thesearch window provides 47X and 20X savings in terms of themaximum and average bandwidth requirements.4.3. Strategy for Sharing Search WindowIn order to minimize the coding efficiency impact of sharingsearch window across engines, a good representative shouldbe selected for the motion of the LCU and CUs within theLCU. AMVP of the LCU is observed to provide a good centerpoint for the shared search window. Fig. 3 shows the densitymap for the relative location of the pixels from best matchingblocks with respect to the AMVP of the LCU for two dif-ferent sequences. Best matching blocks are calculated withthe original HM-3.0 search algorithm and the search range is±64 pixels in each direction. For both sequences, more than99% of the best matching pixels lie in the ±64 vicinity of theAMVP of the LCU. This indicates that AMVP of the LCUcan be used as the search window center without introducingsignificant coding efficiency loss.For smaller blocks that have different AMVPs and conse-quently different search centers, the search window is modi-fied to fit in the shared window. It is important to note thatalthough the search window is modified, original AMVP ofthe block is used in cost calculations. Moreover, total num-ber of candidates for smaller blocks stay the same regardlessof the search window being modified or not. This providessimplicity in hardware implementation.Dist. from AMVP of LCU in x-dirDist. from AMVP of LCU in y-dir 99.4% lies within ±64(a) PeopleOnStreetDist. from AMVP of LCU in x-dirDist. from AMVP of LCU in y-dir 99.9% lies within ±64(b) TrafficFig. 3. Density maps for the relative location of pixels frombest-matching blocks with respect to the AMVP of the LCU.More than 99% of the pixels lie within ±64 of the AMVPof the LCU (2560x1600 sequences with QP=22 in random-access configuration).5. PRE-FETCHING STRATEGYFor a practical hardware implementation, it is necessary torequest the data from off-chip memories in advance as thelatency of these memories can be on the order of thousands ofcycles. To address this, a pre-fetching strategy is developedfor the search algorithm proposed in this work.AMVP of the LCU is used to open the shared searchrange. However, AMVP calculation for current LCU dependson its left neighbor’s motion data and cannot start until leftneighbor’s motion search is finalized. In this work, top (T),top-right (TR) and top-left (TL) neighbors of the current LCUare used to predict current LCU’s AMVP and pre-fetch cor-responding data from the off-chip memory. The procedure topredict current LCU’s AMVP is as follows:• If none of the neighbors is available, data is pre-fetchedfrom [0,0] location.• If only one of the neighbors is available, AMVP of theavailable neighboring LCU is used to pre-fetch data.• If two of the neighbors are available, AMVP of oneof the available neighboring LCUs is used to pre-fetchdata in the following precedence order: T > TR > TL• If all neighbors are available, median of the AMVPs ofthree neighbors is calculated and used to pre-fetch data.With this strategy, request can be sent to an off-chip mem-ory as soon as the motion search of TR neighbor is completed.6. SIMULATION RESULTSProposed changes in search algorithm are implemented inHM-3.0 and simulations are performed under common testconditions defined by JCT-VC [1]. Table 3 shows codingLD LDP RA Avg Max MinHM-3.0 Anchor 0 0 0 0 0 0Prop. Search 0.6 0.8 1.6 1.0 3.1 0.1Prop. Search&Shared Window0.6 1.0 2.9 1.5 7.4 0.2Prop. Search&Shared Window&Pre-fetch0.9 1.0 2.9 1.6 7.3 0.2Table 3. Simulation results of the coding efficiency changewith the proposed changes with respect to HM-3.0.efficiency change with respect to the HM-3.0 fast search al-gorithm after each of the algorithm modifications proposedin this work. Columns LD, LDP and RA stands for low-delay,low-delay with P and random-access test conditions as de-fined by JCT-VC [1]. Avg column is the average of LD, LDPand RA. Lastly, Max and Min columns are the maximum andminimum rate change for all tested sequences respectively.The average rate increase due to the proposed changes is1.6%. Random-access test conditions result in the largest cod-ing efficiency degradation as the distance between the refer-ence frame and the coded frame is longer.7. SUMMARY AND CONCLUSIONSThis work presents a search algorithm for HEVC motion esti-mation. Search patterns, shared search window and pre-fetchschemes are designed to minimize hardware costs. Proposedchanges provide 3X IME logic area, 16% on-chip memoryarea and 47X maximum off-chip bandwidth savings at theexpense of 1.6% average bit-rate increase when compared toHM-3.0 fast search algorithm.8. REFERENCES[1] “Joint Call for Proposals on Video Compression Technology,” ITU-TSG16/Q6, 39th VCEG Meeting: Kyoto, 17-22 Jan. 2010, Doc. VCEG-AM91.[2] “JCT-VC Reference Software HM-3.0,” ISO/IEO MPEG and ITU-T.[3] Yu-Wen Huang et al., “Hardware Architecture Design for Variable BlockSize Motion Estimation in MPEG-4 AVC/JVT/ITU-T H.264,” in ISCAS,May 2003, vol. 2, pp. II–796 – II–799.[4] Swee Yeow Yap et al., “A VLSI Architecture for Variable Block SizeVideo Motion Estimation,” Circuits and Systems II: Express Briefs,IEEE Tran. on, vol. 51, no. 7, pp. 384 – 389, July 2004.[5] M. Haller et al., “Robust global motion estimation using motion vectorsof variable size blocks and automatic motion model selection,” in IEEEICIP, Sept. 2010, pp. 737 –740.[6] Chia-Chun Lin et al., “PMRME: A Parallel Multi-Resolution MotionEstimation Algorithm and Architecture for HDTV Sized H.264 VideoCoding,” in IEEE ICASSP, April 2007, vol. 2, pp. II–385 –II–388.[7] Hsiu-Cheng Chang et al., “A 7mW-to-183mW Dynamic Quality-Scalable H.264 Video Encoder Chip,” in IEEE ISSCC, Feb. 2007, pp.280 –603.

http://dspace.mit.edu/bitstream/1721.1/95987/1/Sinangil_ICIP_2012_algorithm.pdf

Hardware-aware motion estimation search algorithm development for high-efficiency video coding (HEVC) standard

Abstract

Similar works

Full text

Available Versions

Crossref

DSpace@MIT