HEVC ENCODER OPTIMISATIONS USING ADAPTIVE CODING UNIT VISITING ORDER

This research utilised Queen Mary’s MidPlus computational facilities, supported by QMUL Research-IT and funded by EPSRC grant EP/K000128/1

End to end Multi-Objective Optimisation of H.264 and HEVC Codecs

All multimedia devices now incorporate video CODECs that comply with international video coding standards such as H.264 / MPEG4-AVC and the new High Efficiency Video Coding Standard (HEVC) otherwise known as H.265. Although the standard CODECs have been designed to include algorithms with optimal efficiency, large number of coding parameters can be used to fine tune their operation, within known constraints of for e.g., available computational power, bandwidth, consumer QoS requirements, etc. With large number of such parameters involved, determining which parameters will play a significant role in providing optimal quality of service within given constraints is a further challenge that needs to be met. Further how to select the values of the significant parameters so that the CODEC performs optimally under the given constraints is a further important question to be answered. This thesis proposes a framework that uses machine learning algorithms to model the performance of a video CODEC based on the significant coding parameters. Means of modelling both the Encoder and Decoder performance is proposed. We define objective functions that can be used to model the performance related properties of a CODEC, i.e., video quality, bit-rate and CPU time. We show that these objective functions can be practically utilised in video Encoder/Decoder designs, in particular in their performance optimisation within given operational and practical constraints. A Multi-objective Optimisation framework based on Genetic Algorithms is thus proposed to optimise the performance of a video codec. The framework is designed to jointly minimize the CPU Time, Bit-rate and to maximize the quality of the compressed video stream. The thesis presents the use of this framework in the performance modelling and multi-objective optimisation of the most widely used video coding standard in practice at present, H.264 and the latest video coding standard, H.265/HEVC. When a communication network is used to transmit video, performance related parameters of the communication channel will impact the end-to-end performance of the video CODEC. Network delays and packet loss will impact the quality of the video that is received at the decoder via the communication channel, i.e., even if a video CODEC is optimally configured network conditions will make the experience sub-optimal. Given the above the thesis proposes a design, integration and testing of a novel approach to simulating a wired network and the use of UDP protocol for the transmission of video data. This network is subsequently used to simulate the impact of packet loss and network delays on optimally coded video based on the framework previously proposed for the modelling and optimisation of video CODECs. The quality of received video under different levels of packet loss and network delay is simulated, concluding the impact on transmitted video based on their content and features

Video compression algorithms for HEVC and beyond

PhDDue to the increasing number of new services and devices that allow the creation, distribution and consumption of video content, the amount of video information being transmitted all over the world is constantly growing. Video compression technology is essential to cope with the ever increasing volume of digital video data being distributed in today's networks, as more e cient video compression techniques allow support for higher volumes of video data under the same memory/bandwidth constraints. This is especially relevant with the introduction of new and more immersive video formats associated with signi cantly higher amounts of data. In this thesis, novel techniques for improving the e ciency of current and future video coding technologies are investigated. Several aspects that in uence the way conventional video coding methods work are considered. In particular, the properties and limitations of the Human Visual System are exploited to tune the performance of video encoders towards better subjective quality. Additionally, it is shown how the visibility of speci c types of visual artefacts can be prevented during the video encoding process, in order to avoid subjective quality degradations in the compressed content. Techniques for higher video compression e ciency are also explored, targeting to improve the compression capabilities of state-of-the-art video coding standards. Finally, the application of video coding technologies to practical use-cases is considered. Accurate estimation models are devised to control the encoding time and bit rate associated with compressed video signals, in order to meet speci c encoding time and transmission time restrictions

Optimisation du codage HEVC par des moyens de pré-analyse et/ou de pré-codage du contenu

La compression vidéo HEVC standardisée en 2013 offre des gains de compression supérieurs dépassant les 50% par rapport au standard de compression précédent MPEG4-AVC/H.264. Ces gains de compression se paient par une augmentation très importante de la complexité de codage. Si on ajoute à cela l’augmentation de complexité générée par l’accroissement de résolution et de fréquence images du signal vidéo d’entrée pour passer de la Haute Définition (HD) à l’Ultra Haute Définition (UHD), on comprend vite l’intérêt de techniques de réduction de complexité pour le développement de codeurs économiquement viables. En premier lieu, un effort particulier a été réalisé pour réduirela complexité des images Intra. Nous proposons une méthode d’inférence des modes de codage à partir d’un pré-codage d’un version réduite en HD de la vidéo UHD. Ensuite, nous proposons une méthode de partitionnement rapide basée sur la pré-analyse du contenu. La première méthode offre une réduction de complexité d’un facteur 3 et la deuxième, d’un facteur 6, contre une perte de compression proche de 5%. En second lieu, nous avons traité le codage des images Inter. En mettant en oeuvre une solution d’inférence des modes de codage UHD à partir d’un pré-codage au format HD, la complexité de codage est réduite d’un facteur 3 en considérant les 2 flux produits et d’un facteur 9.2 sur le seul flux UHD, pour une perte en compression proche de 3%. Appliqué à une configuration de codage proche d’un système réellement déployé, l’apport de notre algorithme reste intéressant puisqu’il réduit la complexité de codage du flux UHD d’un facteur proche de 2 pour une perte de compression limitée à 4%. Les stratégies de réduction de complexité mises en oeuvre au cours de cette thèse pour le codage Intra et Inter offrent des perspectives intéressantes pour le développement de codeurs HEVC UHD plus économes en ressources de calculs. Elles sont particulièrement adaptées au domaine de la WebTV/OTT qui prend une part croissante dans la diffusion de la vidéo et pour lequel le signal vidéo est codé à des résolutions multiples pour adresser des réseaux et des terminaux de capacités variées.The High Efficiency Video Coding (HEVC) standard was released in 2013 which reduced network bandwidth by a factor of 2 compared to the prior standard H.264/AVC. These gains are achieved by a very significant increase in the encoding complexity. Especially with the industrial demand to shift in format from High Definition (HD) to Ultra High Definition (UHD), one can understand the relevance of complexity reduction techniques to develop cost-effective encoders. In our first contribution, we attempted new strategies to reduce the encoding complexity of Intra-pictures. We proposed a method with inference rules on the coding modes from the modes obtained with pre-encoding of the UHD video down-sampled in HD. We, then, proposed a fast partitioning method based on a preanalysis of the content. The first method reduced the complexity by a factor of 3x and the second one, by a factor of 6, with a loss of compression efficiency of 5%. As a second contribution, we adressedthe Inter-pictures. By implementing inference rules in the UHD encoder, from a HD pre-encoding pass, the encoding complexity is reduced by a factor of 3x when both HD and UHD encodings are considered, and by 9.2x on just the UHD encoding, with a loss of compression efficiency of 3%. Combined with an encoding configuration imitating a real system, our approach reduces the complexity by a factor of close to 2x with 4% of loss. These strategies built during this thesis offer encouraging prospects for implementation of low complexity HEVC UHD encoders. They are fully adapted to the WebTV/OTT segment that is playing a growing part in the video delivery, in which the video signal is encoded with different resolution to reach heterogeneous devices and network capacities

Error resilience and concealment techniques for high-efficiency video coding

This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC). After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against network losses with significant impact on video quality degradation. Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. The problem of error concealment-aware video coding is also investigated to enhance the overall error robustness. A new approach based on scalable coding and optimally error concealment selection is proposed, where the optimal error concealment modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. Moreover, recovery residual information is encoded using a rate-controlled enhancement layer. Both are transmitted to the decoder to be used in case of data loss. Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods

Learned-based Intra Coding Tools for Video Compression.

PhD Theses.The increase in demand for video rendering in 4K and beyond displays, as well as immersive video formats, requires the use of e cient compression techniques. In this thesis novel methods for enhancing the e ciency of current and next generation video codecs are investigated. Several aspects that in uence the way conventional video coding methods work are considered. The methods proposed in this thesis utilise Neural Networks (NNs) trained for regression tasks in order to predict data. In particular, Convolutional Neural Networks (CNNs) are used to predict Rate-Distortion (RD) data for intra-coded frames. Moreover, a novel intra-prediction methods are proposed with the aim of providing new ways to exploit redundancies overlooked by traditional intraprediction tools. Additionally, it is shown how such methods can be simpli ed in order to derive less resource-demanding tools

Error control strategies in H.265|HEVC video transmission

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonWith the rapid development in video coding technologies in the last decade, high-resolution video delivery suffers from packet loss due to unreliable transmission channels (time-varying characteristics). The error Resilience approaches at channel coding level are less efficient to implement in real time video transmission as the encoded video samples are in variable code length. Therefore, error resilience in video coding standard plays a vital role to reduce the effect of error propagation and improve the perceived visual quality. The main work in this thesis is to develop an efficient error resilience mechanism for H.265|HEVC video coding standard to reduce the effects of error propagation in error-prone conditions. In this thesis, two error resilience algorithms are proposed. The first one is Adaptive Slice Encoding (ASE) error resilience algorithm. The concept of this algorithm is to extract and protect the most active slices in the coded bitstream based on the adaptive search window. This algorithm can be applied in low delay video transmission with and without using a feedback channel. It is also designed to be compatible with reference coding software manual (HM16) for H.265|HEVC coding standard. The second proposed algorithm is a joint encoder-decoder error resilience called Error resilience based on Supplemental Enhancement Information (ERSEI) algorithm. A feedback message status is used from the decoder to notify the encoder to start encoding clean random-access picture adaptively based on the decoded picture hash message status from the decoder. At the same time, the decoder will be notified to start the error concealment process whilst waiting to receive correct video data. A recovery point message from the decoder feedback channel is used to update the encoder with error messages. In this thesis, extensive experimental work, evaluation, and comparison with state-of-the-art related algorithms have been conducted to evaluate the proposed algorithms. Furthermore, the best trade-off between the coding efficiency of the proposed error resilience algorithms and error resilience performance has been considered at the design stage. The experimental work evaluation includes both encoding conditions, i.e. error-free and error-prone. The results achieved from the experiments show significant improvements, in (Y-PSNR) results and subjective quality of the decoded bitstream, using the proposed algorithm in error-prone conditions with a variety of packet loss rates. Moreover, experimental work is conducted to test the algorithms complexity in terms of required processing execution time at both encoding and decoding stages. Additionally, the video coding standard performance for both H.264|AVC and H.265|HEVC coding standards are evaluated in error-free and error-prone environments. For ASE algorithm and when compared with improved region of interest (IROI) and region of interest (ROI) algorithms, a significant improvement in visual quality was the most obvious finding from the obtained results with PLRs of 2-18 (%). For ERSEI algorithm and when compared with the default HM16 with pixel copy concealment and motion compensated error concealment (MCEC) techniques, the evaluation results indicate clear visual quality enhancement under different packet loss rates PLRs (1,2 6, 8) %.The Ministry of Higher Education and Scientific Research in Ira

Nouvelles méthodes de prédiction inter-images pour la compression d’images et de vidéos

Due to the large availability of video cameras and new social media practices, as well as the emergence of cloud services, images and videosconstitute today a significant amount of the total data that is transmitted over the internet. Video streaming applications account for more than 70% of the world internet bandwidth. Whereas billions of images are already stored in the cloud and millions are uploaded every day. The ever growing streaming and storage requirements of these media require the constant improvements of image and video coding tools. This thesis aims at exploring novel approaches for improving current inter-prediction methods. Such methods leverage redundancies between similar frames, and were originally developed in the context of video compression. In a first approach, novel global and local inter-prediction tools are associated to improve the efficiency of image sets compression schemes based on video codecs. By leveraging a global geometric and photometric compensation with a locally linear prediction, significant improvements can be obtained. A second approach is then proposed which introduces a region-based inter-prediction scheme. The proposed method is able to improve the coding performances compared to existing solutions by estimating and compensating geometric and photometric distortions on a semi-local level. This approach is then adapted and validated in the context of video compression. Bit-rate improvements are obtained, especially for sequences displaying complex real-world motions such as zooms and rotations. The last part of the thesis focuses on deep learning approaches for inter-prediction. Deep neural networks have shown striking results for a large number of computer vision tasks over the last years. Deep learning based methods proposed for frame interpolation applications are studied here in the context of video compression. Coding performance improvements over traditional motion estimation and compensation methods highlight the potential of these deep architectures.En raison de la grande disponibilité des dispositifs de capture vidéo et des nouvelles pratiques liées aux réseaux sociaux, ainsi qu’à l’émergence desservices en ligne, les images et les vidéos constituent aujourd’hui une partie importante de données transmises sur internet. Les applications de streaming vidéo représentent ainsi plus de 70% de la bande passante totale de l’internet. Des milliards d’images sont déjà stockées dans le cloud et des millions y sont téléchargés chaque jour. Les besoins toujours croissants en streaming et stockage nécessitent donc une amélioration constante des outils de compression d’image et de vidéo. Cette thèse vise à explorer des nouvelles approches pour améliorer les méthodes actuelles de prédiction inter-images. De telles méthodes tirent parti des redondances entre images similaires, et ont été développées à l’origine dans le contexte de la vidéo compression. Dans une première partie, de nouveaux outils de prédiction inter globaux et locaux sont associés pour améliorer l’efficacité des schémas de compression de bases de données d’image. En associant une compensation géométrique et photométrique globale avec une prédiction linéaire locale, des améliorations significatives peuvent être obtenues. Une seconde approche est ensuite proposée qui introduit un schéma deprédiction inter par régions. La méthode proposée est en mesure d’améliorer les performances de codage par rapport aux solutions existantes en estimant et en compensant les distorsions géométriques et photométriques à une échelle semi locale. Cette approche est ensuite adaptée et validée dans le cadre de la compression vidéo. Des améliorations en réduction de débit sont obtenues, en particulier pour les séquences présentant des mouvements complexes réels tels que des zooms et des rotations. La dernière partie de la thèse se concentre sur l’étude des méthodes d’apprentissage en profondeur dans le cadre de la prédiction inter. Ces dernières années, les réseaux de neurones profonds ont obtenu des résultats impressionnants pour un grand nombre de tâches de vision par ordinateur. Les méthodes basées sur l’apprentissage en profondeur proposéesà l’origine pour de l’interpolation d’images sont étudiées ici dans le contexte de la compression vidéo. Des améliorations en terme de performances de codage sont obtenues par rapport aux méthodes d’estimation et de compensation de mouvements traditionnelles. Ces résultats mettent en évidence le fort potentiel de ces architectures profondes dans le domaine de la compression vidéo

Técnica de aprendizagem automática aplicada a um codificador HEVC em tempo real.

O padrão HEVC (High Efficiency Video Coding) é o mais recente padrão para codificação de vídeos e tem uma complexidade computacional muito maior do que seu antecessor, o padrão H.264. A grande eficiência de codificação atingida pelo codificador HEVC é obtida com um custo computacional bastante elevado. Esta tese aborda oportunidades de reduzir essa carga computacional. Dessa forma, um algoritmo de decisão prematura de divisão de uma unidade de codificação é proposto para o codificador HEVC, terminando prematuramente o processo de busca pelo melhor particionamento baseado em um modelo de classificação adaptativo, criado em tempo de execução. Esse modelo é gerado por um processo de aprendizado online baseado no algoritmo Pegasos, que é uma implementação que aplica a resolução do gradiente estocástico ao algoritmo SVM (Support Vector Machine). O método proposto foi implementado e integrado ao codificador de referência HM 16.7. Os resultados experimentais mostraram que o codificador modificado reduziu o custo computacional do processo de codificação em até 50%, em alguns casos, e aproximadamente 30% em média, com perdas de qualidade desprezíveis para os usuários. De modo geral, esse processo resulta em reduzidas perdas de qualidade, no entanto, alguns resultados mostraram pequenos ganhos em eficiência de compressão quando comparados com os resultados do codificador HM 16.7.The most recent video coding standard, the High Efficiency Video Coding (HEVC), has a higher encoding complexity when compared with H.264/AVC, which means a higher computational cost. This thesis presents a review of the recent literature and proposes an algorithm that reduces such complexity. Therefore, a fast CU (Coding Unit) splitting algorithm is proposed for the HEVC encoder, which terminates the CU partitioning process at an early phase, based on an adaptive classification model. This model is generated by an online learning method based on the Primal Estimated sub-GrAdient SOlver for SVM (Pegasos) algorithm. The proposed method is implemented and integrated in the HEVC reference source code on its version 16.7. Experimental results show that the proposed method reduces the computational complexity of the HEVC encoder, up to 50% in some cases, with negligible losses, and shows an average computational reduction of 30%. This process results in reduced coding efficiency losses, however, some results showed a nearby 1% of BD-Rate (Bjontegaard Delta) gains in the Low Delay B configuration, without using an offline training phase