130,696 research outputs found
Compression of interferometric radio-astronomical data
The volume of radio-astronomical data is a considerable burden in the
processing and storing of radio observations with high time and frequency
resolutions and large bandwidths. Lossy compression of interferometric
radio-astronomical data is considered to reduce the volume of visibility data
and to speed up processing.
A new compression technique named "Dysco" is introduced that consists of two
steps: a normalization step, in which grouped visibilities are normalized to
have a similar distribution; and a quantization and encoding step, which rounds
values to a given quantization scheme using a dithering scheme. Several
non-linear quantization schemes are tested and combined with different methods
for normalizing the data. Four data sets with observations from the LOFAR and
MWA telescopes are processed with different processing strategies and different
combinations of normalization and quantization. The effects of compression are
measured in image plane.
The noise added by the lossy compression technique acts like normal system
noise. The accuracy of Dysco is depending on the signal-to-noise ratio of the
data: noisy data can be compressed with a smaller loss of image quality. Data
with typical correlator time and frequency resolutions can be compressed by a
factor of 6.4 for LOFAR and 5.3 for MWA observations with less than 1% added
system noise. An implementation of the compression technique is released that
provides a Casacore storage manager and allows transparent encoding and
decoding. Encoding and decoding is faster than the read/write speed of typical
disks.
The technique can be used for LOFAR and MWA to reduce the archival space
requirements for storing observed data. Data from SKA-low will likely be
compressible by the same amount as LOFAR. The same technique can be used to
compress data from other telescopes, but a different bit-rate might be
required.Comment: Accepted for publication in A&A. 13 pages, 8 figures. Abstract was
abridge
Implicit Neural Multiple Description for DNA-based data storage
DNA exhibits remarkable potential as a data storage solution due to its
impressive storage density and long-term stability, stemming from its inherent
biomolecular structure. However, developing this novel medium comes with its
own set of challenges, particularly in addressing errors arising from storage
and biological manipulations. These challenges are further conditioned by the
structural constraints of DNA sequences and cost considerations. In response to
these limitations, we have pioneered a novel compression scheme and a
cutting-edge Multiple Description Coding (MDC) technique utilizing neural
networks for DNA data storage. Our MDC method introduces an innovative approach
to encoding data into DNA, specifically designed to withstand errors
effectively. Notably, our new compression scheme overperforms classic image
compression methods for DNA-data storage. Furthermore, our approach exhibits
superiority over conventional MDC methods reliant on auto-encoders. Its
distinctive strengths lie in its ability to bypass the need for extensive model
training and its enhanced adaptability for fine-tuning redundancy levels.
Experimental results demonstrate that our solution competes favorably with the
latest DNA data storage methods in the field, offering superior compression
rates and robust noise resilience.Comment: Xavier Pic and Trung Hieu Le are both equal contributors and primary
author
Fast Random Access to Wavelet Compressed Volumetric Data Using Hashing
We present a new approach to lossy storage of the coefficients of wavelet transformed data. While it is common to store the coefficients of largest magnitude (and let all other coefficients be zero), we allow a slightly different set of coefficients to be stored. This brings into play a recently proposed hashing technique that allows space efficient storage and very efficient retrieval of coefficients. Our approach is applied to compression of volumetric data sets. For the ``Visible Man'' volume we obtain up to 80% improvement in compression ratio over previously suggested schemes. Further, the time for accessing a random voxel is quite competitive
A new lossless method of Huffman coding for text data compression and decompression process with FPGA implementation
Digital compression for reducing data size is important because of bandwidth restriction. Compression technique is also named source coding. It defines the process of compressed data using less number of bits than uncompressed form. Compression is the technique for decreasing the amount of information used to represent data without decreasing the quality of the text. It also decreases the number of bits needed to storage or transmission in different media. Compression is a method that makes keeping of data easier for a large size of information. In this study, proposed Huffman design includes encoder and decoder based on new binary tree for improving usage of memory for text compression. A saving percentage of approximately 4°.95% was achieved through the suggested way. In this research, Huffman encoder and decoder were created using Verilog HDL. Huffman design was achieved by using a binary tree. Model Sim simulator tool from Mentor Graphics was used for functional verification and simulation of the design modules. FPGA was used for Huffman implementation
- …