Directional Transforms for Video Coding Based on Lifting on Graphs

In this work we describe and optimize a general scheme based on lifting transforms on graphs for video coding. A graph is constructed to represent the video signal. Each pixel becomes a node in the graph and links between nodes represent similarity between them. Therefore, spatial neighbors and temporal motion-related pixels can be linked, while nonsimilar pixels (e.g., pixels across an edge) may not be. Then, a lifting-based transform, in which filterin operations are performed using linked nodes, is applied to this graph, leading to a 3-dimensional (spatio-temporal) directional transform which can be viewed as an extension of wavelet transforms for video. The design of the proposed scheme requires four main steps: (i) graph construction, (ii) graph splitting, (iii) filte design, and (iv) extension of the transform to different levels of decomposition. We focus on the optimization of these steps in order to obtain an effective transform for video coding. Furthermore, based on this scheme, we propose a coefficien reordering method and an entropy coder leading to a complete video encoder that achieves better coding performance than a motion compensated temporal filterin wavelet-based encoder and a simple encoder derived from H.264/AVC that makes use of similar tools as our proposed encoder (reference software JM15.1 configu ed to use 1 reference frame, no subpixel motion estimation, 16 × 16 inter and 4 × 4 intra modes).This work was supported in part by NSF under grant CCF-1018977 and by Spanish Ministry of Economy and Competitiveness under grants TEC2014-53390-P and TEC2014-52289-R.Publicad

Lifting transforms on graphs and their application to video coding

Compact representations of data are very useful in many applications such as coding, denoising or feature extraction. “Classical” transforms such as Discrete Cosine Transforms (DCT) or Discrete Wavelets Transforms (DWT) provide sparse approximations of smooth signals, but lose efficiency when they are applied to signals with large discontinuities. In such cases, directional transforms, which are able to adapt their basis functions to the underlying signal structure, improve the performance of “classical” transforms. In this PhD Thesis we describe a general class of lifting transforms on graphs that can be seen as N-dimensional directional transforms. Graphs are constructed so that every node corresponds to a specific sample point of a discrete N-dimensional signal and links between nodes represent correlation between samples. Therefore, non-correlated samples (e.g., samples across a large discontinuity in the signal) should not be linked. We propose a lifting-based directional transform that can be applied to any undirected graph. In this transform, filtering operations are performed following highcorrelation directions (indicated by the links between nodes), thus avoiding filtering across large discontinuities that give rise to large high-pass coefficients in those locations. In this way, the transform efficiently exploits the correlation that exists between data on the graph, leading to a more compact representation. We mainly focus on the design and optimization of these lifting transforms on graphs, studying and discussing the three main steps required to obtain an invertible and critically sampled transform: (i) graph construction, (ii) design of “good” graph bipartitions, and (iii) filter design. We also explain how to extend the transform to J levels of decomposition, obtaining a multiresolution analysis of the original N-dimensional signal. The proposed transform has many desirable properties, such as perfect reconstruction, critically-sampled, easy generalization to N-dimensional domains, non-separable and one-dimensional filtering operations, localization in frequency and in the original domain, and the ability to choose any filtering direction. As an application, we develop a graph-based video encoder where the goal is to obtain a compact representation of the original video sequence. To this end, we first propose a graph-representation of the video sequence and then design a 3-dimensional (spatio-temporal) non-separable directional transform. This can be viewed as an extension of wavelet transform-based video encoders that operate in the spatial and in the temporal domains independently. Our transform yields better compaction ability (in terms of non-linear approximation) than a state of the art motion-compensated temporal filtering transform (which can be interpreted as a temporal wavelet transform) and a comparable hybrid Discrete Cosine Transform (DCT)-based video encoder (which is the basis of the latest video coding standards). In order to obtain a complete video encoder, the transform coefficients and the side information (needed to obtain an invertible scheme) should be entropy coded and sent to the decoder. Therefore, we also propose a coefficient-reordering method based on the information of the graph which allows to improve the compression ability of the entropy encoder. Furthermore, we design two different low-cost approaches which aim to reduce the extensive computational complexity of the proposed system without causing significant losses of compression performance. The proposed complete system leads to an efficient encoder which significantly outperforms a comparable hybrid DCT-based encoder in rate-distortion terms. Finally, we investigate how rate-distortion optimization can be applied to the proposed coding scheme.La representación compacta de señales resulta útil en diversas aplicaciones, tales como compresión, reducción de ruido, o extracción de características. Transformadas “clásicas” como la Transformada Discreta del Coseno (DCT) o la TransformadaWavelet Discreta (DWT) logran aproximaciones compactas de señales suaves, pero pierden su eficiencia al ser aplicadas sobre se˜nales que contienen grandes discontinuidades. En estos casos, las transformadas direccionales, capaces de adaptar sus funciones base a la estructura de la señal a analizar, mejoran la eficiencia de las transformadas “clásicas”. En esta tesis nos centramos en el diseño y optimización de transformadas “lifting” sobre grafos, las cuales pueden ser interpretadas como transformadas direccionales N-dimensionales. Los grafos son construidos demanera que cada nodo se corresponde con una muestra específica de una señal discreta N-dimensional, y los enlaces entre los nodos representan correlación entre muestras. Así, muestras no correlacionadas (por ejemplo, muestras que se encuentran a ambos lados de una discontinuidad) no deberían estar unidas. Sobre el grafo formado aplicaremos transformadas basadas en el esquema “lifting”, en las que las operaciones de filtrado se realizan siguiendo las direcciones indicadas por los enlaces entre nodos (direcciones de alta correlación). De esta manera, evitaremos filtrar cruzando a través de largas discontinuidades (lo que resultaría en coeficientes con alto valor en dichas discontinuidades), dando lugar a una transformada direccional que explota la correlación que existe entre las muestras de la señal en el grafo, obteniendo una representación compacta de dicha señal. En esta tesis nos centramos, principalmente, en investigar los tres principales pasos requeridos para obtener una transformada direccional basada en el esquema “lifting” aplicado en grafos: (i) la construcción del grafo, (ii) el diseño de biparticiones del grafo, y (iii) la definición de los filtros. El buen diseño de estos tres procesos determinará, entre otras cosas, la capacidad para compactar la energía de la transformada. También explicamos cómo extender este tipo de transformadas a J niveles de descomposición, obteniendo un análisis multi-resolución de la señal N-dimensional original. La transformada propuesta tiene muchas propiedades deseables, tales como reconstrucción perfecta, muestreo crítico, fácil generalización a dominios N-dimensionales, operaciones de filtrado no separables y unidimensionales, localización en frecuencia y en el dominio original, y capacidad de elegir cualquier dirección de filtrado. Como aplicación, desarrollamos un codificador de vídeo basado en grafos donde el objetivo es obtener una versión compacta de la señal de vídeo original. Para ello, primero proponemos una representación en grafos de la secuencia de vídeo y luego diseñamos transformadas no separables direccionales 3-dimensionales (espacio-tiempo). Nuestro codificador puede interpretarse como una extensión de los codificadores de vídeo basados en “wavelets”, los cuales operan independientemente (de forma separable) en el dominio espacial y en el temporal. La transformada propuesta consigue mejores resultados (en términos de aproximación no lineal) que un método del estado del arte basado en “wavelets” temporales compensadas en movimiento, y un codificador DCT comparable (base de los últimos estándares de codificación de vídeo). Para conseguir un codificador de vídeo completo, los coeficientes resultantes de la transformada y la información secundaria (necesaria para obtener un esquema invertible) deben ser codificados entrópicamente y enviados al decodificador. Por ello, también proponemos en esta tesis un método de reordenación de los coeficientes basado en la información del grafo que permite mejorar la capacidad de compresión del codificador entrópico. El esquema de codificación propuesto mejora significativamente la eficiencia de un codificador híbrido basado en DCT en términos de tasa-distorsión. Sin embargo, nuestro método tiene la desventaja de su gran complejidad computacional. Para tratar de paliar este problema, diseñamos dos algoritmos que tratan de reducir dicha complejidad sin que ello afecte en la capacidad de compresión. Finalmente, investigamos como realizar optimización tasa-distorsión sobre el codificador basado en grafos propuesto

Design and Optimization of Graph Transform for Image and Video Compression

The main contribution of this thesis is the introduction of new methods for designing adaptive transforms for image and video compression. Exploiting graph signal processing techniques, we develop new graph construction methods targeted for image and video compression applications. In this way, we obtain a graph that is, at the same time, a good representation of the image and easy to transmit to the decoder. To do so, we investigate different research directions. First, we propose a new method for graph construction that employs innovative edge metrics, quantization and edge prediction techniques. Then, we propose to use a graph learning approach and we introduce a new graph learning algorithm targeted for image compression that defines the connectivities between pixels by taking into consideration the coding of the image signal and the graph topology in rate-distortion term. Moreover, we also present a new superpixel-driven graph transform that uses clusters of superpixel as coding blocks and then computes the graph transform inside each region. In the second part of this work, we exploit graphs to design directional transforms. In fact, an efficient representation of the image directional information is extremely important in order to obtain high performance image and video coding. In this thesis, we present a new directional transform, called Steerable Discrete Cosine Transform (SDCT). This new transform can be obtained by steering the 2D-DCT basis in any chosen direction. Moreover, we can also use more complex steering patterns than a single pure rotation. In order to show the advantages of the SDCT, we present a few image and video compression methods based on this new directional transform. The obtained results show that the SDCT can be efficiently applied to image and video compression and it outperforms the classical DCT and other directional transforms. Along the same lines, we present also a new generalization of the DFT, called Steerable DFT (SDFT). Differently from the SDCT, the SDFT can be defined in one or two dimensions. The 1D-SDFT represents a rotation in the complex plane, instead the 2D-SDFT performs a rotation in the 2D Euclidean space

Fully Scalable Video Coding Using Redundant-Wavelet Multihypothesis and Motion-Compensated Temporal Filtering

In this dissertation, a fully scalable video coding system is proposed. This system achieves full temporal, resolution, and fidelity scalability by combining mesh-based motion-compensated temporal filtering, multihypothesis motion compensation, and an embedded 3D wavelet-coefficient coder. The first major contribution of this work is the introduction of the redundant-wavelet multihypothesis paradigm into motion-compensated temporal filtering, which is achieved by deploying temporal filtering in the domain of a spatially redundant wavelet transform. A regular triangle mesh is used to track motion between frames, and an affine transform between mesh triangles implements motion compensation within a lifting-based temporal transform. Experimental results reveal that the incorporation of redundant-wavelet multihypothesis into mesh-based motion-compensated temporal filtering significantly improves the rate-distortion performance of the scalable coder. The second major contribution is the introduction of a sliding-window implementation of motion-compensated temporal filtering such that video sequences of arbitrarily length may be temporally filtered using a finite-length frame buffer without suffering from severe degradation at buffer boundaries. Finally, as a third major contribution, a novel 3D coder is designed for the coding of the 3D volume of coefficients resulting from the redundant-wavelet based temporal filtering. This coder employs an explicit estimate of the probability of coefficient significance to drive a nonadaptive arithmetic coder, resulting in a simple software implementation. Additionally, the coder offers the possibility of a high degree of vectorization particularly well suited to the data-parallel capabilities of modern general-purpose processors or customized hardware. Results show that the proposed coder yields nearly the same rate-distortion performance as a more complicated coefficient coder considered to be state of the art