3D high definition video coding on a GPU-based heterogeneous system

H.264/MVC is a standard for supporting the sensation of 3D, based on coding from 2 (stereo) to N views. H.264/MVC adopts many coding options inherited from single view H.264/AVC, and thus its complexity is even higher, mainly because the number of processing views is higher. In this manuscript, we aim at an efficient parallelization of the most computationally intensive video encoding module for stereo sequences. In particular, inter prediction and its collaborative execution on a heterogeneous platform. The proposal is based on an efficient dynamic load balancing algorithm and on breaking encoding dependencies. Experimental results demonstrate the proposed algorithm's ability to reduce the encoding time for different stereo high definition sequences. Speed-up values of up to 90× were obtained when compared with the reference encoder on the same platform. Moreover, the proposed algorithm also provides a more energy-efficient approach and hence requires less energy than the sequential reference algorith

Reducing 3D video coding complexity through more efficient disparity estimation

3D video coding for transmission exploits the Disparity Estimation (DE) to remove the inter-view redundancies present within both the texture and the depth map multi-view videos. Good estimation accuracy can be achieved by partitioning the macro-block into smaller subblocks partitions. However, the DE process must be performed on each individual sub-block to determine the optimal mode and their disparity vectors, in terms of ratedistortion efficiency. This vector estimation process is heavy on computational resources, thus, the coding computational cost becomes proportional to the number of search points and the inter-view modes tested during the rate-distortion optimization. In this paper, a solution that exploits the available depth map data, together with the multi-view geometry, is proposed to identify a better DE search area; such that it allows a reduction in its search points. It also exploits the number of different depth levels present within the current macro-block to determine which modes can be used for DE to further reduce its computations. Simulation results demonstrate that this can save up to 95% of the encoding time, with little influence on the coding efficiency of the texture and the depth map multi-view video coding. This makes 3D video coding more practical for any consumer devices, which tend to have limited computational power.peer-reviewe

Exploiting depth information for fast multi-view video coding

This research work is partially funded by the Strategic Educational Pathways Scholarship Scheme (STEPS-Malta). This scholarship is partly financed by the European Union – European Social Fund (ESF 1.25).Multi-view video coding exploits inter-view redundancies to compress the video streams and their associated depth information. These techniques utilize disparity estimation techniques to obtain disparity vectors (DVs) across different views. However, these methods contribute to the majority of the computational power needed for multi-view video encoding. This paper proposes a solution for fast disparity estimation based on multi-view geometry and depth information. A DV predictor is first calculated followed by an iterative or a fast search estimation process which finds the optimal DV in the search area dictated by the predictor. Simulation results demonstrate that this predictor is reliable enough to determine the area of the optimal DVs to allow a smaller search range. Furthermore, results show that the proposed approach achieves a speedup of 2.5 while still preserving the original rate-distortion performance.peer-reviewe

Fast multi-view video plus depth coding with hierarchical bi-prediction

This research work is partially funded by STEPS-Malta and partially by the EU–ESF 1.25.The Multi-view Video Coding (MVC) standard was developed for efficient encoding of multi-view videos. Part of it requires the calculation of both disparity and motion estimations using a bi-prediction structure. These estimations involve an exhaustive search for the optimal compensation vectors from multiple forward and backward reference frames which, while being very efficient in terms of compression, results in high computational costs. This paper proposes a solution that utilizes the multi-view geometry along with the available depth data, to calculate more accurate predictors for both motion and disparity estimations, and for both directions of the prediction structure. Simulation results demonstrate that this technique is reliable enough to allow a substantial reduction in the search areas in all the reference frames. This in turn results in a significant speed-up gain of 3.2 times with a negligible influence on the coding efficiency, while encoding both the color and the depth MVVs.peer-reviewe

Exploiting depth information for fast motion and disparity estimation in multi-view video coding

This research work is partially funded by the Strategic Educational Pathways Scholarship Scheme (STEPS-Malta). This scholarship is partly financed by the European Union – European Social Fund (ESF 1.25).Multi-view Video Coding (MVC) employs both motion and disparity estimation within the encoding process. These provide a significant increase in coding efficiency at the expense of a substantial increase in computational requirements. This paper presents a fast motion and disparity estimation technique that utilizes the multi-view geometry together with the depth information and the corresponding encoded motion vectors from the reference view, to produce more reliable motion and disparity vector predictors for the current view. This allows for a smaller search area which reduces the computational cost of the multi-view encoding system. Experimental results confirm that the proposed techniques can provide a speed-up gain of up to 4.2 times, with a negligible loss in the rate-distortion performance for both the color and the depth MVC.peer-reviewe

High Performance Multiview Video Coding

Following the standardization of the latest video coding standard High Efficiency Video Coding in 2013, in 2014, multiview extension of HEVC (MV-HEVC) was published and brought significantly better compression performance of around 50% for multiview and 3D videos compared to multiple independent single-view HEVC coding. However, the extremely high computational complexity of MV-HEVC demands significant optimization of the encoder. To tackle this problem, this work investigates the possibilities of using modern parallel computing platforms and tools such as single-instruction-multiple-data (SIMD) instructions, multi-core CPU, massively parallel GPU, and computer cluster to significantly enhance the MVC encoder performance. The aforementioned computing tools have very different computing characteristics and misuse of the tools may result in poor performance improvement and sometimes even reduction. To achieve the best possible encoding performance from modern computing tools, different levels of parallelism inside a typical MVC encoder are identified and analyzed. Novel optimization techniques at various levels of abstraction are proposed, non-aggregation massively parallel motion estimation (ME) and disparity estimation (DE) in prediction unit (PU), fractional and bi-directional ME/DE acceleration through SIMD, quantization parameter (QP)-based early termination for coding tree unit (CTU), optimized resource-scheduled wave-front parallel processing for CTU, and workload balanced, cluster-based multiple-view parallel are proposed. The result shows proposed parallel optimization techniques, with insignificant loss to coding efficiency, significantly improves the execution time performance. This , in turn, proves modern parallel computing platforms, with appropriate platform-specific algorithm design, are valuable tools for improving the performance of computationally intensive applications

Loss-resilient Coding of Texture and Depth for Free-viewpoint Video Conferencing

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is commonly synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this end, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will erase errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Experimental results show that the proposed scheme can outperform the use of a traditional feedback channel by up to 0.82 dB on average at 8% packet loss rate, and by as much as 3 dB for particular frames

Overview of 3D Video: Coding Algorithms, Implementations and Standardization

Projecte final de carrera fet en col.laboració amb Linköping Institute of TechnologyEnglish: 3D technologies have aroused a great interest over the world in the last years. Television, cinema and videogames are introducing, little by little, 3D technologies into the mass market. This comes as a result of the research done in the 3D field, solving many of its limitations such as quality, contents creation or 3D displays. This thesis focus on 3D video, considering concepts that concerns the coding issues and the video formats. The aim is to provide an overview of the current state of 3D video, including the standardization and some interesting implementations and alternatives that exist. In the report necessary background information is presented in order to understand the concepts developed: compression techniques, the different video formats, their standardization and some advances or alternatives to the processes previously explained. Finally, a comparison between the different concepts is presented to complete the overview, ending with some conclusions and proposed ideas for future works.Castellano: Las tecnologías 3D han despertado un gran interés en todo el mundo en los últimos años. Televisión, cine y videojuegos están introduciendo, poco a poco, ésta tecnología en el mercado. Esto es resultado de la investigación realizada en el campo de las 3D, solucionando muchas de sus limitaciones, como la calidad, la creación de contenidos o las pantallas 3D. Este proyecto se centra en el video 3D, considerando los conceptos relacionados con la codificación y los formatos de vídeo. El objetivo es proporcionar una visión del estado actual del vídeo 3D, incluyendo los estándares y algunas de las implementaciones más interesantes que existen. En la memoria, se presenta información adicional para facilitar el seguimiento de los conceptos desarrollados: técnicas de compresión, formatos de vídeo, su estandarización y algunos avances o alternativas a los procesos explicados. Finalmente, se presentan diferentes comparaciones entre los conceptos tratados, acabando el documento con las conclusiones obtenidas e ideas propuestas para futuros trabajos.Català: Les tecnologies 3D han despertat un gran interès a tot el món en els últims anys. Televisió, cinema i videojocs estan introduint, lentament, aquesta tecnologia en el mercat. Això és resultat de la investigació portada a terme en el camp de les 3D, solucionant moltes de les seves limitacions, com la qualitat, la creació de continguts o les pantalles 3D. Aquest proyecte es centra en el video 3D, considerant els conceptes relacionats amb la codificació i els formats de video. L'objectiu és proporcionar una visió de l'estat actual del video 3D, incloent-hi els estandàrds i algunes de les implementacions més interessants que existeixen. A la memòria, es presenta informació adicional per facilitar el seguiment dels conceptes desenvolupats: tècniques de compressió, formats de video, la seva estandardització i alguns avenços o alternatives als procesos explicats. Finalment, es presenten diferents comparacions entre els conceptes tractats i les conclusions obtingudes, juntament amb propostes per a futurs treballs

Computational Complexity Optimization on H.264 Scalable/Multiview Video Coding

The H.264/MPEG-4 Advanced Video Coding (AVC) standard is a high efficiency and flexible video coding standard compared to previous standards. The high efficiency is achieved by utilizing a comprehensive full search motion estimation method. Although the H.264 standard improves the visual quality at low bitrates, it enormously increases the computational complexity. The research described in this thesis focuses on optimization of the computational complexity on H.264 scalable and multiview video coding. Nowadays, video application areas range from multimedia messaging and mobile to high definition television, and they use different type of transmission systems. The Scalable Video Coding (SVC) extension of the H.264/AVC standard is able to scale the video stream in order to adapt to a variety of devices with different capabilities. Furthermore, a rate control scheme is utilized to improve the visual quality under the constraints of capability and channel bandwidth. However, the computational complexity is increased. A simplified rate control scheme is proposed to reduce the computational complexity. In the proposed scheme, the quantisation parameter can be computed directly instead of using the exhaustive Rate-Quantization model. The linear Mean Absolute Distortion (MAD) prediction model is used to predict the scene change, and the quantisation parameter will be increased directly by a threshold when the scene changes abruptly; otherwise, the comprehensive Rate-Quantisation model will be used. Results show that the optimized rate control scheme is efficient on time saving. Multiview Video Coding (MVC) is efficient on reducing the huge amount of data in multiple-view video coding. The inter-view reference frames from the adjacent views are exploited for prediction in addition to the temporal prediction. However, due to the increase in the number of reference frames, the computational complexity is also increased. In order to manage the reference frame efficiently, a phase correlation algorithm is utilized to remove the inefficient inter-view reference frame from the reference list. The dependency between the inter-view reference frame and current frame is decided based on the phase correlation coefficients. If the inter-view reference frame is highly related to the current frame, it is still enabled in the reference list; otherwise, it will be disabled. The experimental results show that the proposed scheme is efficient on time saving and without loss in visual quality and increase in bitrate. The proposed optimization algorithms are efficient in reducing the computational complexity on H.264/AVC extension. The low computational complexity algorithm is useful in the design of future video coding standards, especially on low power handheld devices