Optimizing Sparse RFI Prediction using Deep Learning

Radio Frequency Interference (RFI) is an ever-present limiting factor among radio telescopes even in the most remote observing locations. When looking to retain the maximum amount of sensitivity and reduce contamination for Epoch of Reionization studies, the identification and removal of RFI is especially important. In addition to improved RFI identification, we must also take into account computational efficiency of the RFI-Identification algorithm as radio interferometer arrays such as the Hydrogen Epoch of Reionization Array grow larger in number of receivers. To address this, we present a Deep Fully Convolutional Neural Network (DFCN) that is comprehensive in its use of interferometric data, where both amplitude and phase information are used jointly for identifying RFI. We train the network using simulated HERA visibilities containing mock RFI, yielding a known "ground truth" dataset for evaluating the accuracy of various RFI algorithms. Evaluation of the DFCN model is performed on observations from the 67 dish build-out, HERA-67, and achieves a data throughput of 1.6$\times 10^{5}$ HERA time-ordered 1024 channeled visibilities per hour per GPU. We determine that relative to an amplitude only network including visibility phase adds important adjacent time-frequency context which increases discrimination between RFI and Non-RFI. The inclusion of phase when predicting achieves a Recall of 0.81, Precision of 0.58, and $F_{2}$ score of 0.75 as applied to our HERA-67 observations.Comment: 11 pages, 7 figure

Mitigating Internal Instrument Coupling for 21 cm Cosmology. II. A Method Demonstration with the Hydrogen Epoch of Reionization Array

We present a study of internal reflection and cross-coupling systematics in Phase I of the Hydrogen Epoch of Reionization Array (HERA). In a companion paper, we outlined the mathematical formalism for such systematics and presented algorithms for modeling and removing them from the data. In this work, we apply these techniques to data from HERA's first observing season as a method demonstration. The data show evidence for systematics that, without removal, would hinder a detection of the 21 cm power spectrum for the targeted Epoch of Reionization (EoR) line-of-sight modes in the range 0.2 h −1 Mpc−1 < ${k}_{\parallel }$ < 0.5 h −1 Mpc−1. In particular, we find evidence for nonnegligible amounts of spectral structure in the raw autocorrelations that overlaps with the EoR window and is suggestive of complex instrumental effects. Through systematic modeling on a single night of data, we find we can recover these modes in the power spectrum down to the integrated noise floor, achieving a dynamic range in the EoR window of 106 in power (mK2 units) with respect to the bright galactic foreground signal. Future work with deeper integrations will help determine whether these systematics can continue to be mitigated down to EoR levels. For future observing seasons, HERA will have upgraded analog and digital hardware to better control these systematics in the field

HI 21cm Cosmology and the Bi-spectrum: Closure Diagnostics in Massively Redundant Interferometric Arrays

New massively redundant low frequency arrays allow for a novel investigation of closure relations in interferometry. We employ commissioning data from the Hydrogen Epoch of Reionization Array to investigate closure quantities in this densely packed grid array of 14m antennas operating at 100 MHz to 200 MHz. We investigate techniques that utilize closure phase spectra for redundant triads to estimate departures from redundancy for redundant baseline visibilities. We find a median absolute deviation from redundancy in closure phase across the observed frequency range of about 4.5deg. This value translates into a non-redundancy per visibility phase of about 2.6deg, using prototype electronics. The median absolute deviations from redundancy decrease with longer baselines. We show that closure phase spectra can be used to identify ill-behaved antennas in the array, independent of calibration. We investigate the temporal behavior of closure spectra. The Allan variance increases after a one minute stride time, due to passage of the sky through the primary beam of the transit telescope. However, the closure spectra repeat to well within the noise per measurement at corresponding local sidereal times (LST) from day to day. In future papers in this series we will develop the technique of using closure phase spectra in the search for the HI 21cm signal from cosmic reionization.Comment: 32 pages. 11 figures. Accepted to Radio Scienc

Detection of Cosmic Structures using the Bispectrum Phase. II. First Results from Application to Cosmic Reionization Using the Hydrogen Epoch of Reionization Array

Characterizing the epoch of reionization (EoR) at $z\gtrsim 6$ via the redshifted 21 cm line of neutral Hydrogen (HI) is critical to modern astrophysics and cosmology, and thus a key science goal of many current and planned low-frequency radio telescopes. The primary challenge to detecting this signal is the overwhelmingly bright foreground emission at these frequencies, placing stringent requirements on the knowledge of the instruments and inaccuracies in analyses. Results from these experiments have largely been limited not by thermal sensitivity but by systematics, particularly caused by the inability to calibrate the instrument to high accuracy. The interferometric bispectrum phase is immune to antenna-based calibration and errors therein, and presents an independent alternative to detect the EoR HI fluctuations while largely avoiding calibration systematics. Here, we provide a demonstration of this technique on a subset of data from the Hydrogen Epoch of Reionization Array (HERA) to place approximate constraints on the brightness temperature of the intergalactic medium (IGM). From this limited data, at $z=7.7$ we infer "$1\sigma$" upper limits on the IGM brightness temperature to be $\le 316$ "pseudo" mK at $\kappa_\parallel=0.33$ "pseudo" $h$ Mpc$^{-1}$ (data-limited) and $\le 1000$ "pseudo" mK at $\kappa_\parallel=0.875$ "pseudo" $h$ Mpc$^{-1}$ (noise-limited). The "pseudo" units denote only an approximate and not an exact correspondence to the actual distance scales and brightness temperatures. By propagating models in parallel to the data analysis, we confirm that the dynamic range required to separate the cosmic HI signal from the foregrounds is similar to that in standard approaches, and the power spectrum of the bispectrum phase is still data-limited (at $\gtrsim 10^6$ dynamic range) indicating scope for further improvement in sensitivity as the array build-out continues.Comment: 22 pages, 12 figures (including sub-figures). Published in PhRvD. Abstract may be slightly abridged compared to the actual manuscript due to length limitations on arXi

Direct Optimal Mapping Image Power Spectrum and its Window Functions

The key to detecting neutral hydrogen during the epoch of reionization (EoR) is to separate the cosmological signal from the dominating foreground radiation. We developed direct optimal mapping (Xu et al. 2022) to map interferometric visibilities; it contains only linear operations, with full knowledge of point spread functions from visibilities to images. Here we present an FFT-based image power spectrum and its window functions based on direct optimal mapping. We use noiseless simulation, based on the Hydrogen Epoch of Reionization Array (HERA) Phase I configuration, to study the image power spectrum properties. The window functions show $<10^{-11}$ power leakage from the foreground-dominated region into the EoR window; the 2D and 1D power spectra also verify the separation between the foregrounds and the EoR. Furthermore, we simulated visibilities from a $uv$-complete array and calculated its image power spectrum. The result shows that the foreground--EoR leakage is further suppressed below $10^{-12}$, dominated by the tapering function sidelobes; the 2D power spectrum does not show signs of the horizon wedge. The $uv$-complete result provides a reference case for future 21cm cosmology array designs.Comment: Submitted to Ap

Characterization Of Inpaint Residuals In Interferometric Measurements of the Epoch Of Reionization

Radio Frequency Interference (RFI) is one of the systematic challenges preventing 21cm interferometric instruments from detecting the Epoch of Reionization. To mitigate the effects of RFI on data analysis pipelines, numerous inpaint techniques have been developed to restore RFI corrupted data. We examine the qualitative and quantitative errors introduced into the visibilities and power spectrum due to inpainting. We perform our analysis on simulated data as well as real data from the Hydrogen Epoch of Reionization Array (HERA) Phase 1 upper limits. We also introduce a convolutional neural network that capable of inpainting RFI corrupted data in interferometric instruments. We train our network on simulated data and show that our network is capable at inpainting real data without requiring to be retrained. We find that techniques that incorporate high wavenumbers in delay space in their modeling are best suited for inpainting over narrowband RFI. We also show that with our fiducial parameters Discrete Prolate Spheroidal Sequences (DPSS) and CLEAN provide the best performance for intermittent ``narrowband'' RFI while Gaussian Progress Regression (GPR) and Least Squares Spectral Analysis (LSSA) provide the best performance for larger RFI gaps. However we caution that these qualitative conclusions are sensitive to the chosen hyperparameters of each inpainting technique. We find these results to be consistent in both simulated and real visibilities. We show that all inpainting techniques reliably reproduce foreground dominated modes in the power spectrum. Since the inpainting techniques should not be capable of reproducing noise realizations, we find that the largest errors occur in the noise dominated delay modes. We show that in the future, as the noise level of the data comes down, CLEAN and DPSS are most capable of reproducing the fine frequency structure in the visibilities of HERA data.Comment: 26 pages, 18 figure

What does an interferometer really measure? Including instrument and data characteristics in the reconstruction of the 21cm power spectrum

Combining the visibilities measured by an interferometer to form a cosmological power spectrum is a complicated process in which the window functions play a crucial role. In a delay-based analysis, the mapping between instrumental space, made of per-baseline delay spectra, and cosmological space is not a one-to-one relation. Instead, neighbouring modes contribute to the power measured at one point, with their respective contributions encoded in the window functions. To better understand the power spectrum measured by an interferometer, we assess the impact of instrument characteristics and analysis choices on the estimator by deriving its exact window functions, outside of the delay approximation. Focusing on HERA as a case study, we find that observations made with long baselines tend to correspond to enhanced low-k tails of the window functions, which facilitate foreground leakage outside the wedge, whilst the choice of bandwidth and frequency taper can help narrow them down. With the help of simple test cases and more realistic visibility simulations, we show that, apart from tracing mode mixing, the window functions can accurately reconstruct the power spectrum estimator of simulated visibilities. We note that the window functions depend strongly on the chromaticity of the beam, and less on its spatial structure - a Gaussian approximation, ignoring side lobes, is sufficient. Finally, we investigate the potential of asymmetric window functions, down-weighting the contribution of low-k power to avoid foreground leakage. The window functions presented in this work correspond to the latest HERA upper limits for the full Phase I data. They allow an accurate reconstruction of the power spectrum measured by the instrument and can be used in future analyses to confront theoretical models and data directly in cylindrical space.Comment: 18 pages, 18 figures, submitted to MNRAS. Comments welcome

Bayesian jackknife tests with a small number of subsets: Application to HERA 21cm power spectrum upper limits

We present a Bayesian jackknife test for assessing the probability that a data set contains biased subsets, and, if so, which of the subsets are likely to be biased. The test can be used to assess the presence and likely source of statistical tension between different measurements of the same quantities in an automated manner. Under certain broadly applicable assumptions, the test is analytically tractable. We also provide an open-source code, CHIBORG, that performs both analytic and numerical computations of the test on general Gaussian-distributed data. After exploring the information theoretical aspects of the test and its performance with an array of simulations, we apply it to data from the Hydrogen Epoch of Reionization Array (HERA) to assess whether different sub-seasons of observing can justifiably be combined to produce a deeper 21 cm power spectrum upper limit. We find that, with a handful of exceptions, the HERA data in question are statistically consistent and this decision is justified. We conclude by pointing out the wide applicability of this test, including to CMB experiments and the H0 tension