Data Compression in the Petascale Astronomy Era: a GERLUMPH case study

As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -- and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. However, there is still a need to evaluate its potential on other type of astronomical data, such as from numerical simulations. GERLUMPH (the GPU-Enabled High Resolution cosmological MicroLensing parameter survey) represents an example of a data intensive project in theoretical astrophysics. In the next phase of processing, the ~27 terabyte GERLUMPH dataset is set to grow by a factor of 100 -- well beyond the current storage capabilities of the supercomputing facility on which it resides. In order to minimise bandwidth usage, file transfer time, and storage space, this work evaluates several data compression techniques. Specifically, we investigate off-the-shelf and custom lossless compression algorithms as well as the lossy JPEG2000 compression format. Results of lossless compression algorithms on GERLUMPH data products show small compression ratios (1.35:1 to 4.69:1 of input file size) varying with the nature of the input data. Our results suggest that JPEG2000 could be suitable for other numerical datasets stored as gridded data or volumetric data. When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. In our case study, lossy compression and a high compression ratio do not significantly compromise the intended use of the data for constraining quasar source profiles from cosmological microlensing.Comment: 15 pages, 9 figures, 5 tables. Published in the Special Issue of Astronomy & Computing on The future of astronomical data format

Exclusive-or preprocessing and dictionary coding of continuous-tone images.

The field of lossless image compression studies the various ways to represent image data in the most compact and efficient manner possible that also allows the image to be reproduced without any loss. One of the most efficient strategies used in lossless compression is to introduce entropy reduction through decorrelation. This study focuses on using the exclusive-or logic operator in a decorrelation filter as the preprocessing phase of lossless image compression of continuous-tone images. The exclusive-or logic operator is simply and reversibly applied to continuous-tone images for the purpose of extracting differences between neighboring pixels. Implementation of the exclusive-or operator also does not introduce data expansion. Traditional as well as innovative prediction methods are included for the creation of inputs for the exclusive-or logic based decorrelation filter. The results of the filter are then encoded by a variation of the Lempel-Ziv-Welch dictionary coder. Dictionary coding is selected for the coding phase of the algorithm because it does not require the storage of code tables or probabilities and because it is lower in complexity than other popular options such as Huffman or Arithmetic coding. The first modification of the Lempel-Ziv-Welch dictionary coder is that image data can be read in a sequence that is linear, 2-dimensional, or an adaptive combination of both. The second modification of the dictionary coder is that the coder can instead include multiple, dynamically chosen dictionaries. Experiments indicate that the exclusive-or operator based decorrelation filter when combined with a modified Lempel-Ziv-Welch dictionary coder provides compression comparable to algorithms that represent the current standard in lossless compression. The proposed algorithm provides compression performance that is below the Context-Based, Adaptive, Lossless Image Compression (CALIC) algorithm by 23%, below the Low Complexity Lossless Compression for Images (LOCO-I) algorithm by 19%, and below the Portable Network Graphics implementation of the Deflate algorithm by 7%, but above the Zip implementation of the Deflate algorithm by 24%. The proposed algorithm uses the exclusive-or operator in the modeling phase and uses modified Lempel-Ziv-Welch dictionary coding in the coding phase to form a low complexity, reversible, and dynamic method of lossless image compression

Empirical analysis of BWT-based lossless image compression

The Burrows-Wheeler Transformation (BWT) is a text transformation algorithm originally designed to improve the coherence in text data. This coherence can be exploited by compression algorithms such as run-length encoding or arithmetic coding. However, there is still a debate on its performance on images. Motivated by a theoretical analysis of the performance of BWT and MTF, we perform a detailed empirical study on the role of MTF in compressing images with the BWT. This research studies the compression performance of BWT on digital images using different predictors and context partitions. The major interest of the research is in finding efficient ways to make BWT suitable for lossless image compression.;This research studied three different approaches to improve the compression of image data by BWT. First, the idea of preprocessing the image data before sending it to the BWT compression scheme is studied by using different mapping and prediction schemes. Second, different variations of MTF were investigated to see which one works best for Image compression with BWT. Third, the concept of context partitioning for BWT output before it is forwarded to the next stage in the compression scheme.;For lossless image compression, this thesis proposes the removal of the MTF stage from the BWT compression pipeline and the usage of context partitioning method. The compression performance is further improved by using MED predictor on the image data along with the 8-bit mapping of the prediction residuals before it is processed by BWT.;This thesis proposes two schemes for BWT-based image coding, namely BLIC and BLICx, the later being based on the context-ordering property of the BWT. Our methods outperformed other text compression algorithms such as PPM, GZIP, direct BWT, and WinZip in compressing images. Final results showed that our methods performed better than the state of the art lossless image compression algorithms, such as JPEG-LS, JPEG2000, CALIC, EDP and PPAM on the natural images

Burrows Wheeler Compression Algorithm (BWCA) in Lossless Image Compression

The present paper discusses the implementation of BWCA in lossless image compression. BWCA uses Burrows Wheeler Transform (BWT) as its main transform. As one of combinatorial compression algorithm which in particular reordered symbols according to their following context, it becomes one of promising approach in context modeling compression. BWT was initially created for text compression, and here we study the impact of BWCA method and its improvement when applied to image compression. Since this application is quite different from the original method aim, we analyze the pre- and post-processing influences of BWT

Adaptive edge-based prediction for lossless image compression

Many lossless image compression methods have been suggested with established results hard to surpass. However there are some aspects that can be considered to improve the performance further. This research focuses on two-phase prediction-encoding method, separately studying each and suggesting new techniques.;In the prediction module, proposed Edge-Based-Predictor (EBP) and Least-Squares-Edge-Based-Predictor (LS-EBP) emphasizes on image edges and make predictions accordingly. EBP is a gradient based nonlinear adaptive predictor. EBP switches between prediction-rules based on few threshold parameters automatically determined by a pre-analysis procedure, which makes a first pass. The LS-EBP also uses these parameters, but optimizes the prediction for each pre-analysis assigned edge location, thus applying least-square approach only at the edge points.;For encoding module: a novel Burrows Wheeler Transform (BWT) inspired method is suggested, which performs better than applying the BWT directly on the images. We also present a context-based adaptive error modeling and encoding scheme. When coupled with the above-mentioned prediction schemes, the result is the best-known compression performance in the genre of compression schemes with same time and space complexity

A secure image steganography based on burrows wheeler transform and dynamic bit embedding

In modern public communication networks, digital data is massively transmitted through the internet with a high risk of data piracy. Steganography is a technique used to transmit data without arousing suspicion of secret data existence.  In this paper, a color image steganography technique is proposed in spatial domain. The cover image is segmented into non-overlapping blocks which are scattered among image size window using Burrows Wheeler transform before embedding. Secret data is embedded in each block according to its sequence in the Burrows Wheeler transform output. The hiding method is an operation of an exclusive-or between a virtual bit which is generated from the most significant bit and the least significant bits of the cover pixel. Results of the algorithm are analyzed according to its degradation of the output image and embedding capacity. The results are also compared with other existing methods

Inversion Ranks for Lossless Compression of Color Palette Images

Palette images are widely used in World Wide Web (WWW) and game cartridges applications. Many image used in the WWW are stored and transmitted after they are compressed losslessly with the standard graphics interchange format (GIF), or portable network graphic (PNG). Well known two dimensional compression scheme; such as JPEG-LS and CALIC, fails to yield better compression than GIF or PNG, due to the fact that the pixel value represent indices that point to color values in a look-up table. The GIF standard uses Lempel-Ziv compression, which treats the image as a one-dimensional sequence of index values, ignoring two-dimensional nature. Bzip, another universal compressor, yields even better compression gain that the GIF, PNG, JPEG-LS, and CALIC. Variants of block sorting coders, such as Bzip2, utilize Burrows-Wheeler transformation (BWT) by Burrows M. and Wheeler D. J. (1994), followed by move-to-front (MTF) transformation by Bentley J. L. (1986), Elias, P (1987) before using a statistical coder at the final stage. In this paper, we show that the compression performance of block sorting coder can be improved almost 14% on average by utilizing inversion ranks instead of the move-to-front coding