A comparison of CABAC throughput for HEVC/H.265 VS. AVC/H.264

The CABAC entropy coding engine is a well known throughput bottleneck in the AVC/H.264 video codec. It was redesigned to achieve higher throughput for the latest video coding standard HEVC/H.265. Various improvements were made including reduction in context coded bins, reduction in total bins and grouping of bypass bins. This paper discusses and quantifies the impact of these techniques and introduces a new metric called Bjontegaard delta cycles (BD-cycle) to compare the CABAC throughput of HEVC vs. AVC. BD-cycle uses the Bjontegaard delta measurement method to compute the average difference between the cycles vs. bit-rate curves of HEVC and AVC. This metric is useful for estimating the throughput of an HEVC CABAC engine from an existing AVC CABAC design for a given bit-rate. Under the common conditions set by the JCT-VC standardization body, HEVC CABAC has an average BD-cycle reduction of 31.1% for all intra, 24.3% for low delay, and 25.9% for random ac-cess, when processing up to 8 bypass bins per cycle

A Deeply Pipelined CABAC Decoder for HEVC Supporting Level 6.2 High-tier Applications

High Efficiency Video Coding (HEVC) is the latest video coding standard that specifies video resolutions up to 8K Ultra-HD (UHD) at 120 fps to support the next decade of video applications. This results in high-throughput requirements for the context adaptive binary arithmetic coding (CABAC) entropy decoder, which was already a well-known bottleneck in H.264/AVC. To address the throughput challenges, several modifications were made to CABAC during the standardization of HEVC. This work leverages these improvements in the design of a high-throughput HEVC CABAC decoder. It also supports the high-level parallel processing tools introduced by HEVC, including tile and wavefront parallel processing. The proposed design uses a deeply pipelined architecture to achieve a high clock rate. Additional techniques such as the state prefetch logic, latched-based context memory, and separate finite state machines are applied to minimize stall cycles, while multibypass- bin decoding is used to further increase the throughput. The design is implemented in an IBM 45nm SOI process. After place-and-route, its operating frequency reaches 1.6 GHz. The corresponding throughputs achieve up to 1696 and 2314 Mbin/s under common and theoretical worst-case test conditions, respectively. The results show that the design is sufficient to decode in real-time high-tier video bitstreams at level 6.2 (8K UHD at 120 fps), or main-tier bitstreams at level 5.1 (4K UHD at 60 fps) for applications requiring sub-frame latency, such as video conferencing

Decoder Hardware Architecture for HEVC

This chapter provides an overview of the design challenges faced in the implementation of hardware HEVC decoders. These challenges can be attributed to the larger and diverse coding block sizes and transform sizes, the larger interpolation filter for motion compensation, the increased number of steps in intra prediction and the introduction of a new in-loop filter. Several solutions to address these implementation challenges are discussed. As a reference, results for an HEVC decoder test chip are also presented.Texas Instruments Incorporate

Algorithms and Hardware Co-Design of HEVC Intra Encoders

Digital video is becoming extremely important nowadays and its importance has greatly increased in the last two decades. Due to the rapid development of information and communication technologies, the demand for Ultra-High Definition (UHD) video applications is becoming stronger. However, the most prevalent video compression standard H.264/AVC released in 2003 is inefficient when it comes to UHD videos. The increasing desire for superior compression efficiency to H.264/AVC leads to the standardization of High Efficiency Video Coding (HEVC). Compared with the H.264/AVC standard, HEVC offers a double compression ratio at the same level of video quality or substantial improvement of video quality at the same video bitrate. Yet, HE-VC/H.265 possesses superior compression efficiency, its complexity is several times more than H.264/AVC, impeding its high throughput implementation. Currently, most of the researchers have focused merely on algorithm level adaptations of HEVC/H.265 standard to reduce computational intensity without considering the hardware feasibility. What’s more, the exploration of efficient hardware architecture design is not exhaustive. Only a few research works have been conducted to explore efficient hardware architectures of HEVC/H.265 standard. In this dissertation, we investigate efficient algorithm adaptations and hardware architecture design of HEVC intra encoders. We also explore the deep learning approach in mode prediction. From the algorithm point of view, we propose three efficient hardware-oriented algorithm adaptations, including mode reduction, fast coding unit (CU) cost estimation, and group-based CABAC (context-adaptive binary arithmetic coding) rate estimation. Mode reduction aims to reduce mode candidates of each prediction unit (PU) in the rate-distortion optimization (RDO) process, which is both computation-intensive and time-consuming. Fast CU cost estimation is applied to reduce the complexity in rate-distortion (RD) calculation of each CU. Group-based CABAC rate estimation is proposed to parallelize syntax elements processing to greatly improve rate estimation throughput. From the hardware design perspective, a fully parallel hardware architecture of HEVC intra encoder is developed to sustain UHD video compression at 4K@30fps. The fully parallel architecture introduces four prediction engines (PE) and each PE performs the full cycle of mode prediction, transform, quantization, inverse quantization, inverse transform, reconstruction, rate-distortion estimation independently. PU blocks with different PU sizes will be processed by the different prediction engines (PE) simultaneously. Also, an efficient hardware implementation of a group-based CABAC rate estimator is incorporated into the proposed HEVC intra encoder for accurate and high-throughput rate estimation. To take advantage of the deep learning approach, we also propose a fully connected layer based neural network (FCLNN) mode preselection scheme to reduce the number of RDO modes of luma prediction blocks. All angular prediction modes are classified into 7 prediction groups. Each group contains 3-5 prediction modes that exhibit a similar prediction angle. A rough angle detection algorithm is designed to determine the prediction direction of the current block, then a small scale FCLNN is exploited to refine the mode prediction

Evaluation of multiple slices and tiles in HEVC video encoding

Σημείωση: διατίθεται συμπληρωματικό υλικό σε ξεχωριστό αρχείο

High-Level Synthesis Based VLSI Architectures for Video Coding

High Efficiency Video Coding (HEVC) is state-of-the-art video coding standard. Emerging applications like free-viewpoint video, 360degree video, augmented reality, 3D movies etc. require standardized extensions of HEVC. The standardized extensions of HEVC include HEVC Scalable Video Coding (SHVC), HEVC Multiview Video Coding (MV-HEVC), MV-HEVC+ Depth (3D-HEVC) and HEVC Screen Content Coding. 3D-HEVC is used for applications like view synthesis generation, free-viewpoint video. Coding and transmission of depth maps in 3D-HEVC is used for the virtual view synthesis by the algorithms like Depth Image Based Rendering (DIBR). As first step, we performed the profiling of the 3D-HEVC standard. Computational intensive parts of the standard are identified for the efficient hardware implementation. One of the computational intensive part of the 3D-HEVC, HEVC and H.264/AVC is the Interpolation Filtering used for Fractional Motion Estimation (FME). The hardware implementation of the interpolation filtering is carried out using High-Level Synthesis (HLS) tools. Xilinx Vivado Design Suite is used for the HLS implementation of the interpolation filters of HEVC and H.264/AVC. The complexity of the digital systems is greatly increased. High-Level Synthesis is the methodology which offers great benefits such as late architectural or functional changes without time consuming in rewriting of RTL-code, algorithms can be tested and evaluated early in the design cycle and development of accurate models against which the final hardware can be verified

Feasibility Study of High-Level Synthesis : Implementation of a Real-Time HEVC Intra Encoder on FPGA

High-Level Synthesis (HLS) on automatisoitu suunnitteluprosessi, joka pyrkii parantamaan tuottavuutta perinteisiin suunnittelumenetelmiin verrattuna, nostamalla suunnittelun abstraktiota rekisterisiirtotasolta (RTL) käyttäytymistasolle. Erilaisia kaupallisia HLS-työkaluja on ollut markkinoilla aina 1990-luvulta lähtien, mutta vasta äskettäin ne ovat alkaneet saada hyväksyntää teollisuudessa sekä akateemisessa maailmassa. Hidas käyttöönottoaste on johtunut pääasiassa huonommasta tulosten laadusta (QoR) kuin mitä on ollut mahdollista tavanomaisilla laitteistokuvauskielillä (HDL). Uusimmat HLS-työkalusukupolvet ovat kuitenkin kaventaneet QoR-aukkoa huomattavasti. Tämä väitöskirja tutkii HLS:n soveltuvuutta videokoodekkien kehittämiseen. Se esittelee useita HLS-toteutuksia High Efficiency Video Coding (HEVC) -koodaukselle, joka on keskeinen mahdollistava tekniikka lukuisille nykyaikaisille mediasovelluksille. HEVC kaksinkertaistaa koodaustehokkuuden edeltäjäänsä Advanced Video Coding (AVC) -standardiin verrattuna, saavuttaen silti saman subjektiivisen visuaalisen laadun. Tämä tyypillisesti saavutetaan huomattavalla laskennallisella lisäkustannuksella. Siksi reaaliaikainen HEVC vaatii automatisoituja suunnittelumenetelmiä, joita voidaan käyttää rautatoteutus- (HW ) ja varmennustyön minimoimiseen. Tässä väitöskirjassa ehdotetaan HLS:n käyttöä koko enkooderin suunnitteluprosessissa. Dataintensiivisistä koodaustyökaluista, kuten intra-ennustus ja diskreetit muunnokset, myös enemmän kontrollia vaativiin kokonaisuuksiin, kuten entropiakoodaukseen. Avoimen lähdekoodin Kvazaar HEVC -enkooderin C-lähdekoodia hyödynnetään tässä työssä referenssinä HLS-suunnittelulle sekä toteutuksen varmentamisessa. Suorituskykytulokset saadaan ja raportoidaan ohjelmoitavalla porttimatriisilla (FPGA). Tämän väitöskirjan tärkein tuotos on HEVC intra enkooderin prototyyppi. Prototyyppi koostuu Nokia AirFrame Cloud Server palvelimesta, varustettuna kahdella 2.4 GHz:n 14-ytiminen Intel Xeon prosessorilla, sekä kahdesta Intel Arria 10 GX FPGA kiihdytinkortista, jotka voidaan kytkeä serveriin käyttäen joko peripheral component interconnect express (PCIe) liitäntää tai 40 gigabitin Ethernettiä. Prototyyppijärjestelmä saavuttaa reaaliaikaisen 4K enkoodausnopeuden, jopa 120 kuvaa sekunnissa. Lisäksi järjestelmän suorituskykyä on helppo skaalata paremmaksi lisäämällä järjestelmään käytännössä minkä tahansa määrän verkkoon kytkettäviä FPGA-kortteja. Monimutkaisen HEVC:n tehokas mallinnus ja sen monipuolisten ominaisuuksien mukauttaminen reaaliaikaiselle HW HEVC enkooderille ei ole triviaali tehtävä, koska HW-toteutukset ovat perinteisesti erittäin aikaa vieviä. Tämä väitöskirja osoittaa, että HLS:n avulla pystytään nopeuttamaan kehitysaikaa, tarjoamaan ennen näkemätöntä suunnittelun skaalautuvuutta, ja silti osoittamaan kilpailukykyisiä QoR-arvoja ja absoluuttista suorituskykyä verrattuna olemassa oleviin toteutuksiin.High-Level Synthesis (HLS) is an automated design process that seeks to improve productivity over traditional design methods by increasing design abstraction from register transfer level (RTL) to behavioural level. Various commercial HLS tools have been available on the market since the 1990s, but only recently they have started to gain adoption across industry and academia. The slow adoption rate has mainly stemmed from lower quality of results (QoR) than obtained with conventional hardware description languages (HDLs). However, the latest HLS tool generations have substantially narrowed the QoR gap. This thesis studies the feasibility of HLS in video codec development. It introduces several HLS implementations for High Efficiency Video Coding (HEVC) , that is the key enabling technology for numerous modern media applications. HEVC doubles the coding efficiency over its predecessor Advanced Video Coding (AVC) standard for the same subjective visual quality, but typically at the cost of considerably higher computational complexity. Therefore, real-time HEVC calls for automated design methodologies that can be used to minimize the HW implementation and verification effort. This thesis proposes to use HLS throughout the whole encoder design process. From data-intensive coding tools, like intra prediction and discrete transforms, to more control-oriented tools, such as entropy coding. The C source code of the open-source Kvazaar HEVC encoder serves as a design entry point for the HLS flow, and it is also utilized in design verification. The performance results are gathered with and reported for field programmable gate array (FPGA) . The main contribution of this thesis is an HEVC intra encoder prototype that is built on a Nokia AirFrame Cloud Server equipped with 2.4 GHz dual 14-core Intel Xeon processors and two Intel Arria 10 GX FPGA Development Kits, that can be connected to the server via peripheral component interconnect express (PCIe) generation 3 or 40 Gigabit Ethernet. The proof-of-concept system achieves real-time. 4K coding speed up to 120 fps, which can be further scaled up by adding practically any number of network-connected FPGA cards. Overcoming the complexity of HEVC and customizing its rich features for a real-time HEVC encoder implementation on hardware is not a trivial task, as hardware development has traditionally turned out to be very time-consuming. This thesis shows that HLS is able to boost the development time, provide previously unseen design scalability, and still result in competitive performance and QoR over state-of-the-art hardware implementations

Evaluación de la QoE en un sistema de streaming adaptativo de vídeo 3D basado en DASH

[ES] La distribución de contenidos multimedia, y en particular el streaming de vídeo, domina actualmente el tráfico global de Internet y su importancia será incluso mayor en el futuro. Miles de títulos se agregan mensualmente a los principales proveedores de servicios, como Netflix, YouTube y Amazon. Y de la mano del consumo de contenidos de alta definición que se convierte en la principal tendencia, se puede observar nuevamente un incremento en el consumo de contenidos 3D. Esto ha hecho que las temáticas relacionadas con la producción de contenidos, codificación, transmisión, Calidad de Servicio (QoS) y Calidad de Experiencia (QoE) percibidas por los usuarios de los sistemas de distribución de vídeo 3D sean un tema de investigación con numerosas contribuciones en los últimos años. Esta tesis aborda el problema de la prestación de servicios de transmisión de vídeo 3D bajo condiciones de red de ancho de banda variable. En este sentido, presenta los resultados de la evaluación de la QoE percibida por los usuarios de los sistemas de vídeo 3D, analizando principalmente el impacto de los efectos introducidos en dos de los elementos de la cadena de procesamiento de vídeo 3D: la etapa de codificación y el proceso de transmisión. Para analizar los efectos de la codificación en la calidad del vídeo 3D, en la primera etapa se aborda la evaluación objetiva y subjetiva de la calidad del vídeo, comparando el rendimiento de diferentes estándares y métodos de codificación, con el fin de identificar aquellos que logran la mejor relación entre calidad, tasa de bits y tiempo de codificación. Así mismo, en el contexto de la transmisión en un entorno simulcast, se evalúa la eficacia de la utilización de las codificaciones asimétricas para la transmisión de vídeo 3D, como una alternativa para la reducción del ancho de banda manteniendo la calidad global. En segundo lugar, para el estudio del impacto y el rendimiento del proceso de transmisión, se ha trabajado sobre la base de un sistema de transmisión dinámica adaptativa sobre HTTP (DASH) en el contexto de la transmisión de vídeo tanto 2D como 3D, utilizando diferentes escenarios de variación de ancho de banda. El objetivo ha sido el desarrollo de un marco de referencia para la evaluación de la QoE en escenarios de transmisión adaptativa de vídeo 3D, que permite analizar el impacto en la QoE del usuario frente a diferentes patrones de variación del ancho de banda, así como el rendimiento del algoritmo de adaptación frente a estos escenarios. El trabajo se enfoca en identificar el impacto en la Calidad de Experiencia del usuario que tienen aspectos como: la frecuencia, el tipo, el alcance y la ubicación temporal de los eventos de variación del ancho de banda. El sistema propuesto permite realizar mediciones de rendimiento de forma automatizada y sistemática para la evaluación de los sistemas DASH en el servicio de distribución de vídeo 2D y 3D. Se ha utilizado Puppeteer, la librería Node.js desarrollada por Google, que proporciona una API de alto nivel, para automatizar acciones en el protocolo Chrome Devtools, como iniciar la reproducción, provocar cambios de ancho de banda y guardar los resultados de los procesos de cambio de calidad, marcas de tiempo, paradas, etc. A partir de estos datos, se realiza un procesamiento que permite la reconstrucción del vídeo visualizado, así como la extracción de métricas de calidad y la evaluación de la QoE de los usuarios utilizando la recomendación ITU-T P.1203.[CA] La distribució de continguts multimèdia, i en particular el streaming de vídeo, domina actualment el trànsit global d'Internet i la seua importància serà fins i tot mes gran en el futur. Milers de títols s'afegeixen mensualment als principals proveïdors de serveis, com ara Netflix, YouTube i Amazon. I de la mà del consum de continguts d'alta definició que es converteix en la tendència principal, es pot observar novament un increment en el consum de continguts 3D. Això ha fet que les temàtiques relacionades amb la producció de continguts, codificació, transmissió, Qualitat de Servei (QoS) i Qualitat d'Experiència (QoE) percebudes pels usuaris dels sistemes de distribució de vídeo 3D siguen un tema de recerca amb nombroses contribucions en els últims anys. Aquesta tesi aborda el problema de la prestació de serveis de transmissió de vídeo 3D sota condicions de xarxa d'ample de banda variable. En aquest sentit, presenta els resultats de l'avaluació de la QoE percebuda pels usuaris dels sistemes de vídeo 3D, analitzant principalment l'impacte dels efectes introduïts en dos dels elements de la cadena de processament de vídeo 3D: l'etapa de codificació i el procés de transmissió. Per analitzar els efectes de la codificació en la qualitat del vídeo 3D, a la primera etapa s'aborda l'avaluació objectiva i subjectiva de la qualitat del vídeo, comparant el rendiment de diferents estàndards i mètodes de codificació, per tal d'identificar aquells que aconsegueixen la millor relació entre qualitat, taxa de bits i temps de codificació. Així mateix, en el context de la transmissió en un entorn simulcast, s'avalua l'eficàcia de la utilització de les codificacions asimètriques per la transmissió de vídeo 3D, com una alternativa per la reducció de l'ampleada de banda mantenint la qualitat global. En segon lloc, per a l'estudi de l'impacte i el rendiment del procés de transmissió, s'ha treballat sobre la base d'un sistema de transmissió dinàmica adaptativa sobre HTTP (DASH) en el context de la transmissió de vídeo tant 2D com 3D, utilitzant diferents escenaris de variació d'ample de banda. L'objectiu ha estat el desenvolupament d'un marc de referència per a l'avaluació de la QoE en escenaris de transmissió adaptativa de vídeo 3D, que permet analitzar l'impacte en la QoE de l'usuari davant de diferents patrons de variació de l'ample de banda; així com el rendiment de l'algorisme d'adaptació davant d'aquests escenaris. El treball s'enfoca a identificar l'impacte a la Qualitat d'Experiència de l'usuari que tenen aspectes com ara: la freqüència, el tipus, l'abast i la ubicació temporal dels esdeveniments de variació de l'ample de banda. El sistema proposat permet realitzar mesuraments de rendiment de manera automatitzada i sistemàtica per a l'avaluació dels sistemes DASH en el servei de distribució de vídeo 2D i 3D. S'ha utilitzat Puppeteer, la llibreria Node.js desenvolupada per Google, que proporciona una API d'alt nivell, per automatitzar accions al protocol Chrome Devtools, com iniciar la reproducció, provocar canvis d'ample de banda i desar els resultats dels processos de canvi de qualitat, marques de temps, parades, etc. A partir d'aquestes dades, es fa un processament que permet la reconstrucció del vídeo visualitzat, així com l'extracció de mètriques de qualitat i l'avaluació de la QoE dels usuaris fent servir la recomanació ITU-T P.1203.[EN] The distribution of multimedia content, and in particular video streaming, currently dominates global Internet traffic and will become even more important in the future. Thousands of titles are added monthly to major service providers such as Netflix, YouTube and Amazon. In addition to the consumption of high-definition content becoming the main trend, an increase in the consumption of 3D content can be observed again. This fact has caused that issues related to content production, encoding, transmission, Quality of Service (QoS) and Quality of Experience (QoE) perceived by users of 3D video distribution systems became a research topic with numerous contributions in recent years. This thesis addresses the problem of providing 3D video streaming services under variable bandwidth network conditions. In this sense, it presents the results of the evaluation of the QoE perceived by the users of 3D video systems, analyzing mainly the impact of the effects introduced in two of the elements of the 3D video processing chain: the encoding stage and the transmission process. To analyze the effects of the encoding process on the quality of 3D video, the first stage deals with the objective and subjective evaluation of video quality, comparing the performance of different encoding standards and methods, in order to identify those that achieve the best ratio between quality, bit rate and encoding time. Also, in the context of transmission in a simulcast environment, the advantages of using asymmetric coding for 3D video transmission is evaluated as an alternative for bandwidth reduction while maintaining overall quality. Secondly, for the study of the impact and performance of the transmission process, the work has been carried out on the basis of an adaptive dynamic over HTTP (DASH) transmission system in the context of both 2D and 3D video transmission, using different bandwidth variation scenarios. The aim has been to develop a framework for the evaluation of QoE in 3D adaptive video streaming scenarios, which allows analyzing the impact on the user's QoE against different bandwidth variation patterns, as well as the performance of the adaptation algorithm under these scenarios. The work focuses on identifying the impact on the user's Quality of Experience in aspects such as: frequency, type, range and temporal location of bandwidth variation events. The proposed system allows to perform performance measurements in an automated and systematic way for the evaluation of DASH systems in the 2D and 3D video distribution service. The tool Puppeteer, the Node.js library developed by Google, has been used, which provides a high-level API to automate actions in the Chrome Devtools protocol, such as starting playback, causing bandwidth changes and saving the results of the quality change processes, timestamps, stops, etc. From this data, a further processing is performed that allows the reconstruction of the displayed video, as well as the extraction of quality metrics and the evaluation of the QoE of the users using the ITU-T P.1203 recommendation.Guzmán Castillo, PF. (2022). Evaluación de la QoE en un sistema de streaming adaptativo de vídeo 3D basado en DASH [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/186354TESI

End to end Multi-Objective Optimisation of H.264 and HEVC Codecs

All multimedia devices now incorporate video CODECs that comply with international video coding standards such as H.264 / MPEG4-AVC and the new High Efficiency Video Coding Standard (HEVC) otherwise known as H.265. Although the standard CODECs have been designed to include algorithms with optimal efficiency, large number of coding parameters can be used to fine tune their operation, within known constraints of for e.g., available computational power, bandwidth, consumer QoS requirements, etc. With large number of such parameters involved, determining which parameters will play a significant role in providing optimal quality of service within given constraints is a further challenge that needs to be met. Further how to select the values of the significant parameters so that the CODEC performs optimally under the given constraints is a further important question to be answered. This thesis proposes a framework that uses machine learning algorithms to model the performance of a video CODEC based on the significant coding parameters. Means of modelling both the Encoder and Decoder performance is proposed. We define objective functions that can be used to model the performance related properties of a CODEC, i.e., video quality, bit-rate and CPU time. We show that these objective functions can be practically utilised in video Encoder/Decoder designs, in particular in their performance optimisation within given operational and practical constraints. A Multi-objective Optimisation framework based on Genetic Algorithms is thus proposed to optimise the performance of a video codec. The framework is designed to jointly minimize the CPU Time, Bit-rate and to maximize the quality of the compressed video stream. The thesis presents the use of this framework in the performance modelling and multi-objective optimisation of the most widely used video coding standard in practice at present, H.264 and the latest video coding standard, H.265/HEVC. When a communication network is used to transmit video, performance related parameters of the communication channel will impact the end-to-end performance of the video CODEC. Network delays and packet loss will impact the quality of the video that is received at the decoder via the communication channel, i.e., even if a video CODEC is optimally configured network conditions will make the experience sub-optimal. Given the above the thesis proposes a design, integration and testing of a novel approach to simulating a wired network and the use of UDP protocol for the transmission of video data. This network is subsequently used to simulate the impact of packet loss and network delays on optimally coded video based on the framework previously proposed for the modelling and optimisation of video CODECs. The quality of received video under different levels of packet loss and network delay is simulated, concluding the impact on transmitted video based on their content and features