The CCSDS 123.0-B-2 Low-Complexity Lossless and Near-Lossless Multispectral and Hyperspectral Image Compression Standard: A comprehensive review

The Consultative Committee for Space Data Systems (CCSDS) published the CCSDS 123.0-B-2, “Low- Complexity Lossless and Near-Lossless Multispectral and Hyperspectral Image Compression” standard. This standard extends the previous issue, CCSDS 123.0-B-1, which supported only lossless compression, while maintaining backward compatibility. The main novelty of the new issue is support for near-lossless compression, i.e., lossy compression with user-defined absolute and/or relative error limits in the reconstructed images. This new feature is achieved via closed-loop quantization of prediction errors. Two further additions arise from the new near lossless support: first, the calculation of predicted sample values using sample representatives that may not be equal to the reconstructed sample values, and, second, a new hybrid entropy coder designed to provide enhanced compression performance for low-entropy data, prevalent when non lossless compression is used. These new features enable significantly smaller compressed data volumes than those achievable with CCSDS 123.0-B-1 while controlling the quality of the decompressed images. As a result, larger amounts of valuable information can be retrieved given a set of bandwidth and energy consumption constraints

Lossless compression of hyperspectral images

Band ordering and the prediction scheme are the two major aspects of hyperspectral imaging which have been studied to improve the performance of the compression system. In the prediction module, we propose spatio-spectral prediction methods. Two non-linear spectral prediction methods have been proposed in this thesis. NPHI (Non-linear Prediction for Hyperspectral Images) is based on a band look-ahead technique wherein a reference band is included in the prediction of pixels in the current band. The prediction technique estimates the variation between the contexts of the two bands to modify the weights computed in the reference band to predict the pixels in the current band. EPHI (Edge-based Prediction for Hyperspectral Images) is the modified NPHI technique wherein an edge-based analysis is used to classify the pixels into edges and non-edges in order to perform the prediction of the pixel in the current band. Three ordering methods have been proposed in this thesis. The first ordering method computes the local and global features in each band to group the bands. The bands in each group are ordered by estimating the compression ratios achieved between the entire band in the group and then ordering them using Kruskal\u27s algorithm. The other two methods of ordering compute the compression ratios between b-neighbors in performing the band ordering

Fast and Lightweight Rate Control for Onboard Predictive Coding of Hyperspectral Images

Predictive coding is attractive for compression of hyperspecral images onboard of spacecrafts in light of the excellent rate-distortion performance and low complexity of recent schemes. In this letter we propose a rate control algorithm and integrate it in a lossy extension to the CCSDS-123 lossless compression recommendation. The proposed rate algorithm overhauls our previous scheme by being orders of magnitude faster and simpler to implement, while still providing the same accuracy in terms of output rate and comparable or better image quality

A fully embedded two-stage coder for hyperspectral near-lossless compression

This letter proposes a near-lossless coder for hyperspectral images. The coding technique is fully embedded and minimizes the distortion in the l2 norm initially and in the l∞ norm subsequently. Based on a two-stage near-lossless compression scheme, it includes a lossy and a near-lossless layer. The novelties are: the observation of the convergence of the entropy of the residuals in the original domain and in the spectral-spatial transformed domain; and an embedded near-lossless layer. These contributions enable a progressive transmission while optimising both SNR and PAE performance. The embeddedness is accomplished by bitplane encoding plus arithmetic encoding. Experimental results suggest that the proposed method yields a highly competitive coding performance for hyperspectral images, outperforming multi-component JPEG2000 for l∞ norm and pairing its performance for l2 norm, and also outperforming M-CALIC in the near-lossless case -for PAE ≥5-

Multiband and Lossless Compression of Hyperspectral Images

Hyperspectral images are widely used in several real-life applications. In this paper, we investigate on the compression of hyperspectral images by considering different aspects, including the optimization of the computational complexity in order to allow implementations on limited hardware (i.e., hyperspectral sensors, etc.). We present an approach that relies on a three-dimensional predictive structure. Our predictive structure, 3D-MBLP, uses one or more previous bands as references to exploit the redundancies among the third dimension. The achieved results are comparable, and often better, with respect to the other state-of-art lossless compression techniques for hyperspectral images

Statistical Atmospheric Parameter Retrieval Largely Benefits from Spatial-Spectral Image Compression

The Infrared Atmospheric Sounding Interferometer (IASI) is flying on board of the Metop satellite series, which is part of the EUMETSAT Polar System (EPS). Products obtained from IASI data represent a significant improvement in the accuracy and quality of the measurements used for meteorological models. Notably, IASI collects rich spectral information to derive temperature and moisture profiles –among other relevant trace gases–, essential for atmospheric forecasts and for the understanding of weather. Here, we investigate the impact of near-lossless and lossy compression on IASI L1C data when statistical retrieval algorithms are later applied. We search for those compression ratios that yield a positive impact on the accuracy of the statistical retrievals. The compression techniques help reduce certain amount of noise on the original data and, at the same time, incorporate spatial-spectral feature relations in an indirect way without increasing the computational complexity. We observed that compressing images, at relatively low bitrates, improves results in predicting temperature and dew point temperature, and we advocate that some amount of compression prior to model inversion is beneficial. This research can benefit the development of current and upcoming retrieval chains in infrared sounding and hyperspectral sensors

АЛГОРИТМ СЖАТИЯ ГИПЕРСПЕКТРАЛЬНЫХ ДАННЫХ ДИСТАНЦИОННОГО ЗОНДИРОВАНИЯ ЗЕМЛИ

The evaluation results of hyperspectral data correlation in spatial and spectral domains are presented by the example of the hypercube AVIRIS Moffett Field, and the key features of hyperspectral data are formulated. The basic approaches to lossless compression and the algorithms, which can be applied in Earth remote sensing, are considerеd. They are the prediction (linear prediction, fast lossless, spectral oriented least squares, correlation-based conditional average prediction, M-CALIC), the lookup tables (lookup table, locally averaged interband scaling lookup tables), the 3D wavelets (3D-SPECK). A compression algorithm of hyperspectral data is proposed with regard to the advantages and disadvantages of specific implementations of the analyzed algorithms in remote sensing. The main algorithm stages are the preprocessing (for each spectral channel, it is executed independently), the reduction of a correlation level in the spectral area and the entropy coder. The test results of the developed algorithm are given in comparison to the alternative codecs on the AVIRIS test set (Cuprite, Jasper Ridge, Low Altitude, Moffet Field) that prove the efficiency of the proposed algorithm: parallel processing, low computing cost (low latency instructions are used, no division and multiplication), small random access memory requirements (the memory is used only for storage of the hypercube). In the context of the above advantages, the hardware implementation of the algorithm is allowed for on board the aircraft. Представлены результаты оценки корреляции гиперспектральных данных в пространственной и спектральной областях на примере гиперкуба AVIRIS Moffett Field. На их основе сформулированы ключевые особенности гиперспектральных данных. Приведены основные подходы к сжатию без потерь, выделены алгоритмы, относящиеся к тому или иному классу и применяемые в области дистанционного зондирования, показаны достоинства и недостатки конкретных реализаций на основе предсказания (linear prediction, fast lossless, spectral oriented least squares, correlation-based сonditional аverage рrediction, M-CALIC), поиска по таблице (lookup table, locally averaged interband scaling lookup tables) и вей- влет-преобразования (3D-SPECK). С учетом выявленных недостатков разработан алгоритм сжатия гиперспектральных данных, включающий следующие этапы обработки: предобработка (для каждого спектрального канала выполняется независимо), понижение степени корреляции в спектральной области и энтропийный кодер. Приведены результаты тестирования предложенного алгоритма в сравнении с альтернативными кодеками. В качестве тестовых данных использовались гиперкубы, входящие в тестовый набор AVIRIS (Cuprite, Jasper Ridge, Low Altitude, Moffet Field), который является общепризнанным стандартом при исследовании гиперспектральных данных. Полученные результаты свидетельствуют о соответствии разработанного алгоритма альтернативным подходам к сжатию без потерь, применяемым в дистанционном зондировании Земли. Достоинствами указанного алгоритма являются обеспечение параллельной обработки, вычислительная простота (отсутствие операций с высокой латентностью, например, умножения и деления), минимальные требования к объему оперативной памяти (память используется только для хранения гиперкуба и соответствует его объему). С учетом всего вышесказанного допускается схемотехническая реализация алгоритма на борту летательного аппарата.

Diffusion-based inpainting for coding remote-sensing data

Inpainting techniques based on partial differential equations (PDEs) such as diffusion processes are gaining growing importance as a novel family of image compression methods. Nevertheless, the application of inpainting in the field of hyperspectral imagery has been mainly focused on filling in missing information or dead pixels due to sensor failures. In this paper we propose a novel PDE-based inpainting algorithm to compress hyperspectral images. The method inpaints separately the known data in the spatial and in the spectral dimensions. Then it applies a prediction model to the final inpainting solution to obtain a representation much closer to the original image. Experimental results over a set of hyperspectral images indicate that the proposed algorithm can perform better than a recent proposed extension to prediction-based standard CCSDS-123.0 at low bitrate, better than JPEG 2000 Part 2 with the DWT 9/7 as a spectral transform at all bit-rates, and competitive to JPEG 2000 with principal component analysis (PCA), the optimal spectral decorrelation transform for Gaussian sources