Compression of interferometric radio-astronomical data

The volume of radio-astronomical data is a considerable burden in the processing and storing of radio observations with high time and frequency resolutions and large bandwidths. Lossy compression of interferometric radio-astronomical data is considered to reduce the volume of visibility data and to speed up processing. A new compression technique named "Dysco" is introduced that consists of two steps: a normalization step, in which grouped visibilities are normalized to have a similar distribution; and a quantization and encoding step, which rounds values to a given quantization scheme using a dithering scheme. Several non-linear quantization schemes are tested and combined with different methods for normalizing the data. Four data sets with observations from the LOFAR and MWA telescopes are processed with different processing strategies and different combinations of normalization and quantization. The effects of compression are measured in image plane. The noise added by the lossy compression technique acts like normal system noise. The accuracy of Dysco is depending on the signal-to-noise ratio of the data: noisy data can be compressed with a smaller loss of image quality. Data with typical correlator time and frequency resolutions can be compressed by a factor of 6.4 for LOFAR and 5.3 for MWA observations with less than 1% added system noise. An implementation of the compression technique is released that provides a Casacore storage manager and allows transparent encoding and decoding. Encoding and decoding is faster than the read/write speed of typical disks. The technique can be used for LOFAR and MWA to reduce the archival space requirements for storing observed data. Data from SKA-low will likely be compressible by the same amount as LOFAR. The same technique can be used to compress data from other telescopes, but a different bit-rate might be required.Comment: Accepted for publication in A&A. 13 pages, 8 figures. Abstract was abridge

The importance of detailed epigenomic profiling of different cell types within organs.

The human body consists of hundreds of kinds of cells specified from a single genome overlaid with cell type-specific epigenetic information. Comprehensively profiling the body's distinct epigenetic landscapes will allow researchers to verify cell types used in regenerative medicine and to determine the epigenetic effects of disease, environmental exposures and genetic variation. Key marks/factors that should be investigated include regions of nucleosome-free DNA accessible to regulatory factors, histone marks defining active enhancers and promoters, DNA methylation levels, regulatory RNAs, and factors controlling the three-dimensional conformation of the genome. Here we use the lung to illustrate the importance of investigating an organ's purified cell epigenomes, and outline the challenges and promise of realizing a comprehensive catalog of primary cell epigenomes

A morphological algorithm for improving radio-frequency interference detection

A technique is described that is used to improve the detection of radio-frequency interference in astronomical radio observatories. It is applied on a two-dimensional interference mask after regular detection in the time-frequency domain with existing techniques. The scale-invariant rank (SIR) operator is defined, which is a one-dimensional mathematical morphology technique that can be used to find adjacent intervals in the time or frequency domain that are likely to be affected by RFI. The technique might also be applicable in other areas in which morphological scale-invariant behaviour is desired, such as source detection. A new algorithm is described, that is shown to perform quite well, has linear time complexity and is fast enough to be applied in modern high resolution observatories. It is used in the default pipeline of the LOFAR observatory.Comment: Accepted for publication in A&

Post-correlation radio frequency interference classification methods

We describe and compare several post-correlation radio frequency interference classification methods. As data sizes of observations grow with new and improved telescopes, the need for completely automated, robust methods for radio frequency interference mitigation is pressing. We investigated several classification methods and find that, for the data sets we used, the most accurate among them is the SumThreshold method. This is a new method formed from a combination of existing techniques, including a new way of thresholding. This iterative method estimates the astronomical signal by carrying out a surface fit in the time-frequency plane. With a theoretical accuracy of 95% recognition and an approximately 0.1% false probability rate in simple simulated cases, the method is in practice as good as the human eye in finding RFI. In addition it is fast, robust, does not need a data model before it can be executed and works in almost all configurations with its default parameters. The method has been compared using simulated data with several other mitigation techniques, including one based upon the singular value decomposition of the time-frequency matrix, and has shown better results than the rest.Comment: 14 pages, 12 figures (11 in colour). The software that was used in the article can be downloaded from http://www.astro.rug.nl/rfi-software

Prospects for detecting the 21cm forest from the diffuse intergalactic medium with LOFAR

We discuss the feasibility of the detection of the 21cm forest in the diffuse IGM with the radio telescope LOFAR. The optical depth to the 21cm line has been derived using simulations of reionization which include detailed radiative transfer of ionizing photons. We find that the spectra from reionization models with similar total comoving hydrogen ionizing emissivity but different frequency distribution look remarkably similar. Thus, unless the reionization histories are very different from each other (e.g. a predominance of UV vs. x-ray heating) we do not expect to distinguish them by means of observations of the 21cm forest. Because the presence of a strong x-ray background would make the detection of 21cm line absorption impossible, the lack of absorption could be used as a probe of the presence/intensity of the x-ray background and the thermal history of the universe. Along a random line of sight LOFAR could detect a global suppression of the spectrum from z>12, when the IGM is still mostly neutral and cold, in contrast with the more well-defined, albeit broad, absorption features visible at lower redshift. Sharp, strong absorption features associated with rare, high density pockets of gas could be detected also at z~7 along preferential lines of sight.Comment: 12 pages, 13 figures. MNRAS, in pres

Wide-field LOFAR-LBA power-spectra analyses: Impact of calibration, polarization leakage and ionosphere

Contamination due to foregrounds (Galactic and Extra-galactic), calibration errors and ionospheric effects pose major challenges in detection of the cosmic 21 cm signal in various Epoch of Reionization (EoR) experiments. We present the results of a pilot study of a field centered on 3C196 using LOFAR Low Band (56-70 MHz) observations, where we quantify various wide field and calibration effects such as gain errors, polarized foregrounds, and ionospheric effects. We observe a `pitchfork' structure in the 2D power spectrum of the polarized intensity in delay-baseline space, which leaks into the modes beyond the instrumental horizon (EoR/CD window). We show that this structure largely arises due to strong instrumental polarization leakage ($\sim30\%$) towards {Cas\,A} ($\sim21$ kJy at 81 MHz, brightest source in northern sky), which is far away from primary field of view. We measure an extremely small ionospheric diffractive scale ($r_{\text{diff}} \approx 430$ m at 60 MHz) towards {Cas\,A} resembling pure Kolmogorov turbulence compared to $r_{\text{diff}} \sim3 - 20$ km towards zenith at 150 MHz for typical ionospheric conditions. This is one of the smallest diffractive scales ever measured at these frequencies. Our work provides insights in understanding the nature of aforementioned effects and mitigating them in future Cosmic Dawn observations (e.g. with SKA-low and HERA) in the same frequency window.Comment: 20 pages, 11 figures, accepted for publication in MNRA

Foregrounds for observations of the cosmological 21 cm line: II. Westerbork observations of the fields around 3C196 and the North Celestial Pole

In the coming years a new insight into galaxy formation and the thermal history of the Universe is expected to come from the detection of the highly redshifted cosmological 21 cm line. The cosmological 21 cm line signal is buried under Galactic and extragalactic foregrounds which are likely to be a few orders of magnitude brighter. Strategies and techniques for effective subtraction of these foreground sources require a detailed knowledge of their structure in both intensity and polarization on the relevant angular scales of 1-30 arcmin. We present results from observations conducted with the Westerbork telescope in the 140-160 MHz range with 2 arcmin resolution in two fields located at intermediate Galactic latitude, centred around the bright quasar 3C196 and the North Celestial Pole. They were observed with the purpose of characterizing the foreground properties in sky areas where actual observations of the cosmological 21 cm line could be carried out. The polarization data were analysed through the rotation measure synthesis technique. We have computed total intensity and polarization angular power spectra. Total intensity maps were carefully calibrated, reaching a high dynamic range, 150000:1 in the case of the 3C196 field. [abridged]Comment: 20 pages, 22 figures, accepted for publication in A&A. A version with full resolution figures is available at http://www.astro.rug.nl/~bernardi/NCP_3C196/bernardi.pd