Algorithms and methods for video transcoding.

Video transcoding is the process of dynamic video adaptation. Dynamic video adaptation can be defined as the process of converting video from one format to another, changing the bit rate, frame rate or resolution of the encoded video, which is mainly necessitated by the end user requirements. H.264 has been the predominantly used video compression standard for the last 15 years. HEVC (High Efficiency Video Coding) is the latest video compression standard finalised in 2013, which is an improvement over H.264 video compression standard. HEVC performs significantly better than H.264 in terms of the Rate-Distortion performance. As H.264 has been widely used in the last decade, a large amount of video content exists in H.264 format. There is a need to convert H.264 video content to HEVC format to achieve better Rate-Distortion performance and to support legacy video formats on newer devices. However, the computational complexity of HEVC encoder is 2-10 times higher than that of H.264 encoder. This makes it necessary to develop low complexity video transcoding algorithms to transcode from H.264 to HEVC format. This research work proposes low complexity algorithms for H.264 to HEVC video transcoding. The proposed algorithms reduce the computational complexity of H.264 to HEVC video transcoding significantly, with negligible loss in Rate-Distortion performance. This work proposes three different video transcoding algorithms. The MV-based mode merge algorithm uses the block mode and MV variances to estimate the split/non-split decision as part of the HEVC block prediction process. The conditional probability-based mode mapping algorithm models HEVC blocks of sizes 16×16 and lower as a function of H.264 block modes, H.264 and HEVC Quantisation Parameters (QP). The motion-compensated MB residual-based mode mapping algorithm makes the split/non-split decision based on content-adaptive classification models. With a combination of the proposed set of algorithms, the computational complexity of the HEVC encoder is reduced by around 60%, with negligible loss in Rate-Distortion performance, outperforming existing state-of-art algorithms by 20-25% in terms of computational complexity. The proposed algorithms can be used in computation-constrained video transcoding applications, to support video format conversion in smart devices, migration of large-scale H.264 video content from host servers to HEVC, cloud computing-based transcoding applications, and also to support high quality videos over bandwidth-constrained networks

Filling the gaps in video transcoder deployment in the cloud

Cloud-based deployment of content production and broadcast workflows has continued to disrupt the industry after the pandemic. The key tools required for unlocking cloud workflows, e.g., transcoding, metadata parsing, and streaming playback, are increasingly commoditized. However, as video traffic continues to increase there is a need to consider tools which offer opportunities for further bitrate/quality gains as well as those which facilitate cloud deployment. In this paper we consider preprocessing, rate/distortion optimisation and cloud cost prediction tools which are only just emerging from the research community. These tools are posed as part of the per-clip optimisation approach to transcoding which has been adopted by large streaming media processing entities but has yet to be made more widely available for the industry.Comment: Camera-ready version of BEIT Conference at NAB 202

Transcodificação em tempo real de vídeo digital H.264 para codecs de nova geração

A utilização do vídeo digital tem crescido bastante nos últimos anos em diversos dispositivos. A oferta e a procura de conteúdos audiovisuais com mais qualidade devem-se muito às melhorias significativas no poder de processamento nos diversos dispositivos e ao aumento da largura de banda na Internet. A utilização de codecs de vídeo torna-se inevitável para tornar a transmissão de vídeo digital mais eficiente. O codec de vídeo H.264 continua ainda hoje a ser utilizado em diversas plataformas relacionadas com o vídeo, sendo que em alguns casos é sacrificada a qualidade de vídeo para preservar a largura de banda utilizada. Novos codecs de vídeo surgiram após o H.264 para ultrapassarem problemas que este não consegue solucionar face às exigências atuais. Este trabalho analisa detalhadamente os codecs de vídeo H.265, VP9 e AV1 que visam substituir o H.264 no âmbito de transmissões televisivas através da implementação de um sistema apto para processar em tempo real as streams produzidas pelos estúdios de televisão, com o objetivo de reduzir a largura de banda necessária para a transmissão de conteúdos audiovisuais sem sacrificar a qualidade de vídeo. Propõe-se a implementação de um sistema que transcodifica as streams de vídeo enviadas pelos estúdios de televisão para codecs de vídeo mais recentes ao invés de substituir os equipamentos necessários em cada estação televisiva. Esta implementação detalha técnicas e ferramentas de software utilizadas num protótipo experimental, seguindo-se uma fase de testes realizados para validar o propósito da utilização deste tipo de sistema. No final, conclui-se que a utilização deste sistema permite em alguns casos reduzir o débito binário do mesmo vídeo de forma considerável, mantendo o mesmo nível de qualidade de imagem.The use of digital video has grown considerably in recent years in various devices. The supply and demand for higher-quality audiovisual content is due to significant improvements in the processing power of the various devices and the increase in bandwidth on the Internet. The use of video codecs becomes inevitable to make digital video transmission more efficient. The H.264 video codec is still used in several platforms related to the video, and in some cases the quality of video is sacrificed to preserve the bandwidth. New video codecs have emerged after H.264 to overcome problems that it can’t solve today. This work analyzes in detail the video codecs H.265, VP9 and AV1 that aim to replace the H.264 in the scope of television transmissions through the implementation of a system able to process in real time the streams produced by the television studios, with the objective to reduce the bandwidth required for the transmission of audiovisual content without sacrificing video quality. It is proposed to implement a system that transcodes video streams sent by video studios to newer video codecs instead of replacing the necessary equipment on each television station. This implementation details techniques and software tools used in an experimental prototype, followed by a phase of tests performed to validate the purpose of the use of this type of system. In the end, it is concluded that the use of this system allows in some cases to reduce the bitrate of the same video while maintaining the same level of image quality

Transcodificação em tempo real de vídeo digital H.264 para codecs de nova geração

A utilização do vídeo digital tem crescido bastante nos últimos anos em diversos dispositivos. A oferta e a procura de conteúdos audiovisuais com mais qualidade devem-se muito às melhorias significativas no poder de processamento nos diversos dispositivos e ao aumento da largura de banda na Internet. A utilização de codecs de vídeo torna-se inevitável para tornar a transmissão de vídeo digital mais eficiente. O codec de vídeo H.264 continua ainda hoje a ser utilizado em diversas plataformas relacionadas com o vídeo, sendo que em alguns casos é sacrificada a qualidade de vídeo para preservar a largura de banda utilizada. Novos codecs de vídeo surgiram após o H.264 para ultrapassarem problemas que este não consegue solucionar face às exigências atuais. Este trabalho analisa detalhadamente os codecs de vídeo H.265, VP9 e AV1 que visam substituir o H.264 no âmbito de transmissões televisivas através da implementação de um sistema apto para processar em tempo real as streams produzidas pelos estúdios de televisão, com o objetivo de reduzir a largura de banda necessária para a transmissão de conteúdos audiovisuais sem sacrificar a qualidade de vídeo. Propõe-se a implementação de um sistema que transcodifica as streams de vídeo enviadas pelos estúdios de televisão para codecs de vídeo mais recentes ao invés de substituir os equipamentos necessários em cada estação televisiva. Esta implementação detalha técnicas e ferramentas de software utilizadas num protótipo experimental, seguindo-se uma fase de testes realizados para validar o propósito da utilização deste tipo de sistema. No final, conclui-se que a utilização deste sistema permite em alguns casos reduzir o débito binário do mesmo vídeo de forma considerável, mantendo o mesmo nível de qualidade de imagem.The use of digital video has grown considerably in recent years in various devices. The supply and demand for higher-quality audiovisual content is due to significant improvements in the processing power of the various devices and the increase in bandwidth on the Internet. The use of video codecs becomes inevitable to make digital video transmission more efficient. The H.264 video codec is still used in several platforms related to the video, and in some cases the quality of video is sacrificed to preserve the bandwidth. New video codecs have emerged after H.264 to overcome problems that it can’t solve today. This work analyzes in detail the video codecs H.265, VP9 and AV1 that aim to replace the H.264 in the scope of television transmissions through the implementation of a system able to process in real time the streams produced by the television studios, with the objective to reduce the bandwidth required for the transmission of audiovisual content without sacrificing video quality. It is proposed to implement a system that transcodes video streams sent by video studios to newer video codecs instead of replacing the necessary equipment on each television station. This implementation details techniques and software tools used in an experimental prototype, followed by a phase of tests performed to validate the purpose of the use of this type of system. In the end, it is concluded that the use of this system allows in some cases to reduce the bitrate of the same video while maintaining the same level of image quality

Implementing a Video Framework based on IIIF: A Customized Approach from Long-Term Preservation Video Formats to Conversion on Demand

This paper addresses the issue of elaborating a structure for digital video assets based on the International Image Interoperability Framework (IIIF) concepts for the use in archival environments. With a view to tailoring a solution to fit the end user's needs, the dissemination copies of video material could be automatically converted on demand from their master files. Such a reduced data structure simplifies access to digital video sources but leads as well to simplified preservation due to reduced data volume and data complexity. Dissemination copies do not require specific dispositions for digital archiving anymore. Memory institutions would greatly benefit from a technology that can be integrated into a Web-based infrastructure. In such a way video content can for example be embedded into flexible Virtual Research Environments which allow scholars to work and cite more accurately video resources using IIIF

Deep-learning based precoding techniques for next-generation video compression

Several research groups worldwide are currently investigating how deep learning may advance the state-of-the-art in image and video coding. An open question is how to make deep neural networks work in conjunction with existing (and upcoming) video codecs, such as MPEG AVC/H.264, HEVC, VVC, Google VP9 and AOMedia AV1, as well as existing container and transport formats. Such compatibility is a crucial aspect, as the video content industry and hardware manufacturers are expected to remain committed to supporting these standards for the foreseeable future. We propose deep neural networks as precoding components for current and future codec ecosystems. In our current deployments for DASH/HLS adaptive streaming, this comprises downscaling neural networks. Precoding via deep learning allows for full compatibility to current and future codec and transport standards while providing for significant savings. Our results with HD content show that 23%-43% rate reduction takes place under a range of state-of-the-art video codec implementations. The use of precoding can also lead to significant encoding complexity reduction, which is essential for the cloud deployment of complex encoders like AV1 and MPEG VVC. Therefore, beyond bitrate saving, deep-learning based precoding may reduce the required cloud resources for video transcoding and make cloud-based solutions competitive or superior to state-of-the-art captive deployments

End to end Multi-Objective Optimisation of H.264 and HEVC Codecs

All multimedia devices now incorporate video CODECs that comply with international video coding standards such as H.264 / MPEG4-AVC and the new High Efficiency Video Coding Standard (HEVC) otherwise known as H.265. Although the standard CODECs have been designed to include algorithms with optimal efficiency, large number of coding parameters can be used to fine tune their operation, within known constraints of for e.g., available computational power, bandwidth, consumer QoS requirements, etc. With large number of such parameters involved, determining which parameters will play a significant role in providing optimal quality of service within given constraints is a further challenge that needs to be met. Further how to select the values of the significant parameters so that the CODEC performs optimally under the given constraints is a further important question to be answered. This thesis proposes a framework that uses machine learning algorithms to model the performance of a video CODEC based on the significant coding parameters. Means of modelling both the Encoder and Decoder performance is proposed. We define objective functions that can be used to model the performance related properties of a CODEC, i.e., video quality, bit-rate and CPU time. We show that these objective functions can be practically utilised in video Encoder/Decoder designs, in particular in their performance optimisation within given operational and practical constraints. A Multi-objective Optimisation framework based on Genetic Algorithms is thus proposed to optimise the performance of a video codec. The framework is designed to jointly minimize the CPU Time, Bit-rate and to maximize the quality of the compressed video stream. The thesis presents the use of this framework in the performance modelling and multi-objective optimisation of the most widely used video coding standard in practice at present, H.264 and the latest video coding standard, H.265/HEVC. When a communication network is used to transmit video, performance related parameters of the communication channel will impact the end-to-end performance of the video CODEC. Network delays and packet loss will impact the quality of the video that is received at the decoder via the communication channel, i.e., even if a video CODEC is optimally configured network conditions will make the experience sub-optimal. Given the above the thesis proposes a design, integration and testing of a novel approach to simulating a wired network and the use of UDP protocol for the transmission of video data. This network is subsequently used to simulate the impact of packet loss and network delays on optimally coded video based on the framework previously proposed for the modelling and optimisation of video CODECs. The quality of received video under different levels of packet loss and network delay is simulated, concluding the impact on transmitted video based on their content and features