In-correlator techniques offer the possibility of identifying and/or excising
radio frequency interference (RFI) from interferometric observations at much
higher time and/or frequency resolution than is generally possible with the
final visibility dataset. Due to the considerable computational requirements of
the correlation procedure, cross-correlators have most commonly been
implemented using high-speed digital signal processing boards, which typically
require long development times and are difficult to alter once complete.
"Software" correlators, on the other hand, make use of commodity server
machines and a correlation algorithm coded in a high-level language. They are
inherently much more flexible and can be developed - and modified - much more
rapidly than purpose-built "hardware" correlators. Software correlators are
thus a natural choice for testing new RFI detection and mitigation techniques
for interferometers. The ease with which software correlators can be adapted to
test RFI detection algorithms is demonstrated by the addition of kurtosis
detection and plotting to the widely used DiFX software correlator, which
highlights previously unknown short -duration RFI at the Hancock VLBA station.Comment: 6 pages, 1 figure, accepted for publication in Proceedings of Science
  [PoS(RFI2010)035]. Presented at RFI2010, the Third Workshop on RFI Mitigation
  in Radio Astronomy, 29-31 March 2010, Groningen, The Netherland

Deller, Adam T.

English

arXiv

In–correlator techniques offer the possibility of identifying and/or excising radio frequency interference (RFI) from interferometric observations at much higher time and/or frequency resolution than is generally possible with the final visibility dataset. Due to the considerable computational requirements of the correlation procedure, cross–correlators have most commonly been implemented using high–speed digital signal processing boards, which typically require long development times and are difficult to alter once complete. “Software" correlators, on the other hand, make use of commodity server machines and a correlation algorithm coded in a high–level language. They are inherently much more flexible and can be developed – and modified – much more rapidly than purpose–built “hardware" correlators. Software correlators are thus a natural choice for testing new RFI detection and mitigation techniques for interferometers. The ease with which software correlators can be adapted to test RFI detection algorithms is demonstrated by the addition of kurtosis detection and plotting to the widely used DiFX software correlator, which highlights previously unknown short–duration RFI at the Hancock VLBA station

Deller, Adam

University of Groningen

   University of GroningenSoftware correlators as testbeds for RFI algorithmsDeller, AdamPublished in:Proceedings of ScienceIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2010Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Deller, A. (2010). Software correlators as testbeds for RFI algorithms. Proceedings of Science.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 12-11-2019arXiv:1012.0325v1  [astro-ph.IM]  1 Dec 2010 Software correlators as testbeds for RFI algorithmsAdam Deller∗National Radio Astronomy ObservatoryE-mail: adeller@nrao.eduIn–correlator techniques offer the possibility of identifying and/or excising radio frequency inter-ference (RFI) from interferometric observations at much higher time and/or frequency resolutionthan is generally possible with the final visibility dataset. Due to the considerable computationalrequirements of the correlation procedure, cross–correlators have most commonly been imple-mented using high–speed digital signal processing boards, which typically require long develop-ment times and are difficult to alter once complete. “Software" correlators, on the other hand,make use of commodity server machines and a correlation algorithm coded in a high–level lan-guage. They are inherently much more flexible and can be developed – and modified – muchmore rapidly than purpose–built “hardware" correlators. Software correlators are thus a naturalchoice for testing new RFI detection and mitigation techniques for interferometers. The ease withwhich software correlators can be adapted to test RFI detection algorithms is demonstrated bythe addition of kurtosis detection and plotting to the widely used DiFX software correlator, whichhighlights previously unknown short–duration RFI at the Hancock VLBA station.RFI mitigation workshop - RFI2010,March 29-31, 2010Groningen, the Netherlands∗Speaker.c© Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. http://pos.sissa.it/Software correlators as testbeds Adam Deller1. Interferometry, cross–correlators and RFIRadiotelescope arrays are inherently more robust against RFI than single dishes, due to thespatial separation between the array elements. However, sources of RFI which are visible to morethan one element will not be rejected during the correlation process and will corrupt the measuredvisibilities between those pairs of elements (baselines) and then the final interferometric image.This problem is particularly severe for short baselines, where interfering sources are least likely tohave been attenuated significantly before being received at neighbouring stations.The cross–correlation algorithm used by radio interferometers to estimate the visibility func-tion has been extensively reviewed elsewhere (see e.g. [1] and [2]) and is not re–derived here. Themost important point of note is that cross–correlators can be built in an FX style (where the sampledbaseband data is segmented and channelised on a per–station basis, and then cross-multiplied) oran XF style (in which the data streams for every baseline are cross–correlated with a range of lags,and the resultant lag spectrum is periodically Fourier transformed to generated the final visibilityspectrum). From the point of view of RFI detection and mitigation, the FX algorithm offers greaterflexibility, since it provides access to the final channelised visibilities at higher time resolution.Previous efforts to mitigate RFI in interferometric arrays have focused on identification andexcision/correction of the RFI either before [3] or after [4] the correlator. Integrating RFI mitigationalgorithms inside the correlator, however, exploits the advantages of both approaches:1. Unlike post–correlation techniques, inspection of the data at high time resolution is possible,providing better signal–to–noise for the detection of faint, short–duration RFI;2. Likewise, if the duty cycle of the RFI is low, excision of the RFI at high time resolution ispossible and will result in the loss of less good data;3. Unlike pre-correlation techniques, no new steps are inserted into the serial signal processingchain, minimising the chance of introducing unwanted effects; and4. Utilising the processing which is already undertaken in the correlator allows potentially morepowerful techniques to be applied.However, incorporating RFI detection and mitigation inside the cross–correlator is more chal-lenging than traditional approaches, since:1. The data can only be inspected on–the–fly;2. The implementation of the RFI algorithms cannot consume resources or otherwise imposeconstraints that would disturb the calculation of the visibility function; and3. In most correlators, access to intermediate data products such as partially integrated visibilityspectra is limited.Since software correlators can expose intermediate data products in a convenient (software–accessible) form, they are ideal for testing new RFI detection and mitigation techniques, with amuch shorter development time than would be required on a hardware correlator. The DiFX soft-ware correlator [5], which is the subject of this investigation, is briefly described below.2Software correlators as testbeds Adam Deller1.1 The DiFX correlatorThe Distributed FX (DiFX) software correlator is described in detail in [5], and its relevantqualities for high time resoution RFI studies are briefly summarised here. As the name suggests,it is an FX style correlator, which distributes data across multiple computing nodes using time–division multiplexing. The length of the time slices (termed “sub–integrations" below) is com-pletely configurable, down to the inverse of the final spectral resolution, and the spectral resolutionitself is completely configurable. DiFX thus offers a very flexible platform for inspecting the RFIcharacteristics of interferometric data. DiFX is used for production correlation of both the VeryLong Baseline Array (VLBA) in the USA and the Long Baseline Array (LBA) in Australia, pro-viding ample opportunities to test RFI algorithms on real data.DiFX is written in C++, but uses optimised C vector libraries to carry out the bulk of corre-lation operations. This allows both a convenient and comprehensible high–level design, and goodperformance. Access to the vectors which hold (for example) the sampled baseband data and thevisibility results is easily provided through pointers, and since the correlator runs in an unclockedfashion (in contrast to purpose–built hardware correlators using Field Programmable Gate Arraysor custom logic, which will generically be referred to as Digital Signal Processing – DSP – boardsbelow) there are no timing constraints on the addition of extra RFI–related processing.2. Kurtosis–based RFI detectionKurtosis measures the “peakedness" of the probability distribution of a real–valued randomvariable. It is defined as the fourth central moment of the variable divided by the square of thesecond central moment (where the second central moment is also the variance). Closely related isthe “kurtosis excess", which is equal to the kurtosis minus 3. The kurtosis of a normally distributedvariable is equal to 3, and hence the kurtosis excess of a normally distributed variable is 0.Since the explicit assumption of radio interferometry is that bona fide radio astronomical sig-nals are normally distributed, the calculation of kurtosis provides a useful diagnostic of the presenceof RFI. A continuously present sinusoid has a kurtosis excess < 0 (its distribution is platykurtic),while impulsive bursts lead to a kurtosis excess > 0 (a leptokurtic distribution). The effects of RFIwith varying duty cycles on kurtosis in a radio astronomy context are investigated in [6].In practice, the central moments of the signal are not known and must be estimated from asample of data. This is commonly achieved using unbiased estimators of the second and fourthcumulants (“k statistics") in place of the second and fourth central moments, respectively.In the ideal case of a perfect filterbank, each output spectral channel can be treated as anindependent time series and standard time–domain kurtosis analysis can be applied. However,this straightforward approach is complicated by the fact that cross-correlators window data in thetime domain, produce complex channelised data, and result in non-negligible leakage between thespectral channels. A more general concept of spectral kurtosis (SK; [7]) has been developed whichaccounts for these complications, while preserving the properties of kurtosis excess (Gaussian inputdata gives SK = 0, impulsive RFI yields SK > 0, and continuous sinusoids yield SK < 0). Thekurtosis calculation implemented in DiFX is somewhat simplified compared to the newly developedSK concept, but illustrates the principles of RFI detection and the ease with which the algorithmcan be incorporated into a software correlator.3Software correlators as testbeds Adam Deller3. Implementation of kurtosis detection in DiFXAs an FX style correlator, DiFX already produced channelised data for each antenna, which isavailable in the code before the cross-multiplication to form visibilities. Over the course of a sub–integration, the 1st, 2nd, 3rd and 4th raw moments (m1, m2, m3 and m4) for each (complex) spectralchannel were maintained. At the end of each sub–integration, the 2nd and 4th central momentsu2 and u4 were calculated from the raw moments, and the kurtosis for each channel calculated asu4/u22. The calculated kurtosis values were written to a text file for later analysis. The durationof the sub–integration can be (and was) varied to times less than a millisecond, using the existingsoftware correlator infrastructure.This simple approach yields a biased estimation of the kurtosis, for the reasons given in Sec-tion 2 above. At the time of implementation, the author was not aware of the SK concept, which iswhy this simple implementation was made. Converting the existing simple implementation to thefull SK calculation will be trivial, however, and is covered in Section 5. Nevertheless, Section 4shows that even a biased estimation of the kurtosis can be used to show the presence and spectralcharacteristics of RFI.The greatest value of the kurtosis implementation in DiFX, however, is as a proof of conceptfor the rapid prototyping of RFI algorithms. In total, ∼100 lines of code were necessary for thekurtosis calculation, bookkeeping and storage. This is much less than would be required on a sys-tem based on DSP boards, due to the abstraction made possible by the use of high level languages.Implementing and testing this additional code required only a few hours of coding. Compared tothe development which would be required on a DSP board correlator, this is extremely rapid.4. ResultsTo test the kurtosis implementation, a VLBA test experiment from 22 June 2009 at 1650 MHzwas correlated using DiFX with kurtosis calculation enabled. The spectrum around 1650 MHz ismoderately contaminated with RFI at many VLBA sites. The results below show a single 8 MHzband of right circular polarisation, spanning 1650.49 MHz to 1658.49 MHz. Auto– and cross–correlation spectra were integrated for a standard value of 2 seconds, while the kurtosis calculationswere integrated for 100 ms, providing a higher time resolution view of the RFI.Figure 1 shows a sample of the results from two stations. Three adjacent 100 ms kurtosis inte-grations are shown for the Fort Davis (left panel) and Hancock (right panel) stations, with the auto-correlations shown for comparison. The autocorrelation spectra are relatively clean for both stations– the spikes at 1 MHz intervals are the injected pulse calibration tones, used for removing instru-mental delays. Small deviations from the expected smooth bandpass are seen for the Hancock sta-tion, but the location and magnitude of the RFI is not obvious. The kurtosis results, however, showseveral strong, rapidly varying RFI sources at Hancock. Movies made from these plots, which showthe varying RFI more clearly, are available at http://www.aoc.nrao.edu/~adeller/rfi/.The source of the RFI at the Hancock station is not known. It is clearly variable on mstimescales and is somewhat spectrally confined, although it has a substantial bandwidth of around0.5 MHz. Other Hancock bands exhibit similar characteristics, including in the protected 1660 –1670 MHz range. When kurtosis–based blanking (see Section 5) is implemented, the data which isclearly RFI–dominated could easily be excised, recovering most of the Hancock data.4Software correlators as testbeds Adam DellerThe measured receiver noise temperature at Hancock has been observed to be erratic at theband encompassing these observations, and the knowledge gained from these tests has been passedon to NRAO staff to aid in the search for potential self–generated RFI, which might possibly orig-inate in the noise injection circuit. Obtaining high signal to noise results on short timescales usingthe kurtosis output of DiFX, which would not possible with the normal cross and/or autocorrela-tions, has provided useful information for diagnosing this problem. 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelationFigure 1: Autocorrelation (green) and kurtosis (red) for several successive 100 ms snapshots during theobservation, time running from top to bottom. The Fort Davis station is shown in the left hand column, withno discernible RFI. The Hancock station is shown in the right hand column, and non-Gaussian signals areclearly present based on the kurtosis results. The autocorrelation amplitudes in green have been arbitrarilyscaled for convenient plotting on the same scale as the kurtosis measurements. The autocorrelation ampli-tudes remain constant, as they are taken from a 2 second integration covering this 300 ms period. Kurtosis(not kurtosis excess) is shown. It is clear that the RFI varies dramatically on timescales shorter than 100 ms.5Software correlators as testbeds Adam Deller5. Conclusions and future workA simple kurtosis estimator has been successfully implemented in the DiFX software cor-relator. This initial foray into high time resolution in–correlator RFI detection was made withminimal effort, requiring only an afternoon of code development and testing, and provided usefulinformation on previously unknown RFI at the Hancock VLBA station. The present simplisticimplementation should soon be updated to the unbiased SK estimator, and a thorough analysis ofthe effects of the coarse front–end quantisation should also be undertaken. Ultimately, it may bepossible to generalise the SK concept to the crosscorrelation outputs of the correlator.With or without these improvements, however, the calculated kurtosis could be used to flagdata on the fly (on timescales shorter than a full integration) or to produce a flag table which couldbe inspected and applied later in offline analysis. The latter case, whilst allowing the flexibility todeselect some flags, loses the ability to save data which has been affected for only a short subsetof a given integration. In either case, this approach is still an excision procedure which ultimatelyresults in data loss. "Kurtosis blanking" and "kurtosis flagging" will be added as DiFX features inthe near feature, and the usability and presentation of the saved kurtosis results will be improved.Development of other forms of in–correlator RFI mitigation are planned for the DiFX corre-lator. One example is the rejection of bright sources outside the desired array field of view, whichcan be achieved through field of view shaping using tapered time integrations (see [8]). This willallow a determination of the relative trade-off between accuracy of the tapering function and theperformance overhead incurred. The success of the kurtosis development detailed in this workindicates that these approaches can be rapidly and effectively implemented.References[1] A. R. Thompson, J. M. Moran, & G. W. Swenson, Jn., Interferometry and synthesis in radioastronomy, 2nd ed., Wiley, New York 1994[2] J. D. Romney, 1999, Cross–correlators, in Synthesis Imaging in Radio Astronomy II, ASP ConferenceSeries, 180, 57[3] W. A. Baan, P. A. Fridman & R. P. Millenaar, 2004, Radio Frequency Interference Mitigation at theWesterbork Synthesis Radio Telescope: Algorithms, Test Observations, and System Implementation,AJ, 128, 933[4] R. Athreya, 2009, A New Approach to Mitigation of Radio Frequency Interference in InterferometricData, ApJ, 696, 885[5] A. Deller, S. J. Tingay, M. Bailes, & C. West, 2007, DiFX: A Software Correlator for Very LongBaseline Interferometry Using Multiprocessor Computing Environments, PASP, 119, 318[6] R. D. de Roo, S. Misra, & C. S. Ruf, 2007, Sensitivity of the Kurtosis Statistic as a Detector of PulsedSinusoidal RFI, IEEE Transactions on Geoscience and Remote Sensing, 45, 1938[7] G. M. Nita, D. E. Gary, Z. Liu, G. J. Hurford, & S. M. White, 2007, Radio Frequency InterferenceExcision Using Spectral-Domain Statistics, PASP, 119, 805[8] C. J. Lonsdale, S. S. Doeleman, & D. Oberoi, 2004, Efficient Imaging Strategies For Next-GenerationRadio Array, Experimental Astronomy, 17, 3456

Software correlators as testbeds for RFI algorithms

ARTS repository - University of Groningen

   University of GroningenSoftware correlators as testbeds for RFI algorithmsDeller, AdamPublished in:Proceedings of ScienceIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2010Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Deller, A. (2010). Software correlators as testbeds for RFI algorithms. Proceedings of Science.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 30-10-2023arXiv:1012.0325v1  [astro-ph.IM]  1 Dec 2010Software correlators as testbeds for RFI algorithmsAdam Deller∗National Radio Astronomy ObservatoryE-mail: adeller@nrao.eduIn–correlator techniques offer the possibility of identifying and/or excising radio frequency inter-ference (RFI) from interferometric observations at much higher time and/or frequency resolutionthan is generally possible with the final visibility dataset. Due to the considerable computationalrequirements of the correlation procedure, cross–correlators have most commonly been imple-mented using high–speed digital signal processing boards,which typically require long develop-ment times and are difficult to alter once complete. “Software" correlators, on the other hand,make use of commodity server machines and a correlation algorithm coded in a high–level lan-guage. They are inherently much more flexible and can be developed – and modified – muchmore rapidly than purpose–built “hardware" correlators. Software correlators are thus a naturalchoice for testing new RFI detection and mitigation techniques for interferometers. The ease withwhich software correlators can be adapted to test RFI detection algorithms is demonstrated bythe addition of kurtosis detection and plotting to the widely used DiFX software correlator, whichhighlights previously unknown short–duration RFI at the Hancock VLBA station.RFI mitigation workshop - RFI2010,March 29-31, 2010Groningen, the Netherlands∗Speaker.c© Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licen. http://pos.sissa.it/Software correlators as testbeds Adam Deller1. Interferometry, cross–correlators and RFIRadiotelescope arrays are inherently more robust against RFI han single dishes, due to thespatial separation between the array elements. However, sources of RFI which are visible to morethan one element will not be rejected during the correlationprocess and will corrupt the measuredvisibilities between those pairs of elements (baselines) and then the final interferometric image.This problem is particularly severe for short baselines, where interfering sources are least likely tohave been attenuated significantly before being received atneighbouring stations.The cross–correlation algorithm used by radio interferometers to estimate the visibility func-tion has been extensively reviewed elsewhere (see e.g. [1] and [2]) and is not re–derived here. Themost important point of note is that cross–correlators can be built in an FX style (where the sampledbaseband data is segmented and channelised on a per–stationbas s, and then cross-multiplied) oran XF style (in which the data streams for every baseline are cross–correlated with a range of lags,and the resultant lag spectrum is periodically Fourier transformed to generated the final visibilityspectrum). From the point of view of RFI detection and mitigation, the FX algorithm offers greaterflexibility, since it provides access to the final channelised visibilities at higher time resolution.Previous efforts to mitigate RFI in interferometric arrayshave focused on identification andexcision/correction of the RFI either before [3] or after [4] the correlator. Integrating RFI mitigationalgorithmsinsidethe correlator, however, exploits the advantages of both appro ches:1. Unlike post–correlation techniques, inspection of the data at high time resolution is possible,providing better signal–to–noise for the detection of faint, short–duration RFI;2. Likewise, if the duty cycle of the RFI is low, excision of the RFI at high time resolution ispossible and will result in the loss of less good data;3. Unlike pre-correlation techniques, no new steps are inserted into the serial signal processingchain, minimising the chance of introducing unwanted effects; and4. Utilising the processing which is already undertaken in the correlator allows potentially morepowerful techniques to be applied.However, incorporating RFI detection and mitigation inside the cross–correlator is more chal-lenging than traditional approaches, since:1. The data can only be inspected on–the–fly;2. The implementation of the RFI algorithms cannot consume resources or otherwise imposeconstraints that would disturb the calculation of the visibility function; and3. In most correlators, access to intermediate data products s h as partially integrated visibilityspectra is limited.Since software correlators can expose intermediate data products in a convenient (software–accessible) form, they are ideal for testing new RFI detection and mitigation techniques, with amuch shorter development time than would be required on a hardware correlator. The DiFX soft-ware correlator [5], which is the subject of this investigaton, is briefly described below.2Software correlators as testbeds Adam Deller1.1 The DiFX correlatorThe Distributed FX (DiFX) software correlator is describedin detail in [5], and its relevantqualities for high time resoution RFI studies are briefly summarised here. As the name suggests,it is an FX style correlator, which distributes data across multiple computing nodes using time–division multiplexing. The length of the time slices (termed “sub–integrations" below) is com-pletely configurable, down to the inverse of the final spectral resolution, and the spectral resolutionitself is completely configurable. DiFX thus offers a very flexible platform for inspecting the RFIcharacteristics of interferometric data. DiFX is used for production correlation of both the VeryLong Baseline Array (VLBA) in the USA and the Long Baseline Array (LBA) in Australia, pro-viding ample opportunities to test RFI algorithms on real data.DiFX is written in C++, but uses optimised C vector librariesto carry out the bulk of corre-lation operations. This allows both a convenient and comprehensible high–level design, and goodperformance. Access to the vectors which hold (for example)th sampled baseband data and thevisibility results is easily provided through pointers, and since the correlator runs in an unclockedfashion (in contrast to purpose–built hardware correlators using Field Programmable Gate Arraysor custom logic, which will generically be referred to as Digital Signal Processing – DSP – boardsbelow) there are no timing constraints on the addition of extra RFI–related processing.2. Kurtosis–based RFI detectionKurtosis measures the “peakedness" of the probability distribution of a real–valued randomvariable. It is defined as the fourth central moment of the variable divided by the square of thesecond central moment (where the second central moment is also the variance). Closely related isthe “kurtosis excess", which is equal to the kurtosis minus 3. The kurtosis of a normally distributedvariable is equal to 3, and hence the kurtosis excess of a normally distributed variable is 0.Since the explicit assumption of radio interferometry is that bona fide radio astronomical sig-nals are normally distributed, the calculation of kurtosisprovides a useful diagnostic of the presenceof RFI. A continuously present sinusoid has a kurtosis excess< 0 (its distribution isplatykurtic),while impulsive bursts lead to a kurtosis excess> 0 (a leptokurticdistribution). The effects of RFIwith varying duty cycles on kurtosis in a radio astronomy context are investigated in [6].In practice, the central moments of the signal are not known and must be estimated from asample of data. This is commonly achieved using unbiased estimators of the second and fourthcumulants (“k statistics") in place of the second and fourthcentral moments, respectively.In the ideal case of a perfect filterbank, each output spectral h nnel can be treated as anindependent time series and standard time–domain kurtosisanalysis can be applied. However,this straightforward approach is complicated by the fact tha cross-correlators window data in thetime domain, produce complex channelised data, and result in non-negligible leakage between thespectral channels. A more general concept of spectral kurtosis (SK; [7]) has been developed whichaccounts for these complications, while preserving the properties of kurtosis excess (Gaussian inputdata gives SK = 0, impulsive RFI yields SK > 0, and continuous sinusoids yield SK < 0). Thekurtosis calculation implemented in DiFX is somewhat simplified compared to the newly developedSK concept, but illustrates the principles of RFI detectionand the ease with which the algorithmcan be incorporated into a software correlator.3Software correlators as testbeds Adam Deller3. Implementation of kurtosis detection in DiFXAs an FX style correlator, DiFX already produced channelised data for each antenna, which isavailable in the code before the cross-multiplication to form visibilities. Over the course of a sub–integration, the 1st, 2nd, 3rd and 4th raw moments (m1, m2, m3 andm4) for each (complex) spectralchannel were maintained. At the end of each sub–integration, the 2nd and 4th central momentsu2 andu4 were calculated from the raw moments, and the kurtosis for each channel calculated asu4/u22. The calculated kurtosis values were written to a text file for later analysis. The durationof the sub–integration can be (and was) varied to times less than a millisecond, using the existingsoftware correlator infrastructure.This simple approach yields a biased estimation of the kurtosis, for the reasons given in Sec-tion 2 above. At the time of implementation, the author was not aware of the SK concept, which iswhy this simple implementation was made. Converting the existing simple implementation to thefull SK calculation will be trivial, however, and is coveredin Section 5. Nevertheless, Section 4shows that even a biased estimation of the kurtosis can be used to show the presence and spectralcharacteristics of RFI.The greatest value of the kurtosis implementation in DiFX, however, is as a proof of conceptfor the rapid prototyping of RFI algorithms. In total,∼100 lines of code were necessary for thekurtosis calculation, bookkeeping and storage. This is much less than would be required on a sys-tem based on DSP boards, due to the abstraction made possibleby th use of high level languages.Implementing and testing this additional code required only a few hours of coding. Compared tothe development which would be required on a DSP board correlator, this is extremely rapid.4. ResultsTo test the kurtosis implementation, a VLBA test experimentfrom 22 June 2009 at 1650 MHzwas correlated using DiFX with kurtosis calculation enabled. The spectrum around 1650 MHz ismoderately contaminated with RFI at many VLBA sites. The results below show a single 8 MHzband of right circular polarisation, spanning 1650.49 MHz to 1658.49 MHz. Auto– and cross–correlation spectra were integrated for a standard value of2 seconds, while the kurtosis calculationswere integrated for 100 ms, providing a higher time resolutin view of the RFI.Figure 1 shows a sample of the results from two stations. Three adjacent 100 ms kurtosis inte-grations are shown for the Fort Davis (left panel) and Hancock (right panel) stations, with the auto-correlations shown for comparison. The autocorrelation spectra are relatively clean for both stations– the spikes at 1 MHz intervals are the injected pulse calibration tones, used for removing instru-mental delays. Small deviations from the expected smooth bandpass are seen for the Hancock sta-tion, but the location and magnitude of the RFI is not obvious. The kurtosis results, however, showseveral strong, rapidly varying RFI sources at Hancock. Movies made from these plots, which showthe varying RFI more clearly, are available atht p://www.aoc.nrao.edu/~adeller/rfi/.The source of the RFI at the Hancock station is not known. It isclearly variable on mstimescales and is somewhat spectrally confined, although ithas a substantial bandwidth of around0.5 MHz. Other Hancock bands exhibit similar characteristics, ncluding in the protected 1660 –1670 MHz range. When kurtosis–based blanking (see Section 5) is implemented, the data which isclearly RFI–dominated could easily be excised, recoveringmost of the Hancock data.4Software correlators as testbeds Adam DellerThe measured receiver noise temperature at Hancock has beeno s rved to be erratic at theband encompassing these observations, and the knowledge gained from these tests has been passedon to NRAO staff to aid in the search for potential self–generated RFI, which might possibly orig-inate in the noise injection circuit. Obtaining high signalto noise results on short timescales usingthe kurtosis output of DiFX, which would not possible with the normal cross and/or autocorrela-tions, has provided useful information for diagnosing thisproblem. 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelationFigure 1: Autocorrelation (green) and kurtosis (red) for several successive 100 ms snapshots during theobservation, time running from top to bottom. The Fort Davisstation is shown in the left hand column, withno discernible RFI. The Hancock station is shown in the righthand column, and non-Gaussian signals areclearly present based on the kurtosis results. The autocorrelation amplitudes in green have been arbitrarilyscaled for convenient plotting on the same scale as the kurtosis measurements. The autocorrelation ampli-tudes remain constant, as they are taken from a 2 second integration covering this 300 ms period. Kurtosis(not kurtosis excess) is shown. It is clear that the RFI varies dramatically on timescales shorter than 100 ms.5Software correlators as testbeds Adam Deller5. Conclusions and future workA simple kurtosis estimator has been successfully implemented in the DiFX software cor-relator. This initial foray into high time resolution in–correlator RFI detection was made withminimal effort, requiring only an afternoon of code development and testing, and provided usefulinformation on previously unknown RFI at the Hancock VLBA station. The present simplisticimplementation should soon be updated to the unbiased SK estimator, and a thorough analysis ofthe effects of the coarse front–end quantisation should also be undertaken. Ultimately, it may bepossible to generalise the SK concept to the crosscorrelation outputs of the correlator.With or without these improvements, however, the calculated kurtosis could be used to flagdata on the fly (on timescales shorter than a full integration) or to produce a flag table which couldbe inspected and applied later in offline analysis. The latter case, whilst allowing the flexibility todeselect some flags, loses the ability to save data which has been affected for only a short subsetof a given integration. In either case, this approach is still an excision procedure which ultimatelyresults in data loss. "Kurtosis blanking" and "kurtosis flaggin " will be added as DiFX features inthe near feature, and the usability and presentation of the sav d kurtosis results will be improved.Development of other forms of in–correlator RFI mitigationare planned for the DiFX corre-lator. One example is the rejection of bright sources outside the desired array field of view, whichcan be achieved through field of view shaping using tapered time integrations (see [8]). This willallow a determination of the relative trade-off between accura y of the tapering function and theperformance overhead incurred. The success of the kurtosisdevelopment detailed in this workindicates that these approaches can be rapidly and effectively implemented.References[1] A. R. Thompson, J. M. Moran, & G. W. Swenson, Jn.,I terferometry and synthesis in radioastronomy, 2nd ed., Wiley, New York 1994[2] J. D. Romney, 1999,Cross–correlators, in Synthesis Imaging in Radio Astronomy II, ASP ConferenceSeries,180, 57[3] W. A. Baan, P. A. Fridman & R. P. Millenaar, 2004,Radio Frequency Interference Mitigation at theWesterbork Synthesis Radio Telescope: Algorithms, Test Observations, and System Implementation,AJ, 128, 933[4] R. Athreya, 2009,A New Approach to Mitigation of Radio Frequency Interferencin InterferometricData, ApJ, 696, 885[5] A. Deller, S. J. Tingay, M. Bailes, & C. West, 2007,DiFX: A Software Correlator for Very LongBaseline Interferometry Using Multiprocessor Computing Environments, PASP, 119, 318[6] R. D. de Roo, S. Misra, & C. S. Ruf, 2007,Sensitivity of the Kurtosis Statistic as a Detector of PulsedSinusoidal RFI, IEEE Transactions on Geoscience and Remote Sensing, 45, 1938[7] G. M. Nita, D. E. Gary, Z. Liu, G. J. Hurford, & S. M. White, 2007,Radio Frequency InterferenceExcision Using Spectral-Domain Statistics, PASP, 119, 805[8] C. J. Lonsdale, S. S. Doeleman, & D. Oberoi, 2004,Efficient Imaging Strategies For Next-GenerationRadio Array, Experimental Astronomy, 17, 3456

   University of GroningenSoftware correlators as testbeds for RFI algorithmsDeller, AdamPublished in:Proceedings of ScienceIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2010Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Deller, A. (2010). Software correlators as testbeds for RFI algorithms. Proceedings of Science.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 08-09-2023arXiv:1012.0325v1  [astro-ph.IM]  1 Dec 2010Software correlators as testbeds for RFI algorithmsAdam Deller∗National Radio Astronomy ObservatoryE-mail: adeller@nrao.eduIn–correlator techniques offer the possibility of identifying and/or excising radio frequency inter-ference (RFI) from interferometric observations at much higher time and/or frequency resolutionthan is generally possible with the final visibility dataset. Due to the considerable computationalrequirements of the correlation procedure, cross–correlators have most commonly been imple-mented using high–speed digital signal processing boards,which typically require long develop-ment times and are difficult to alter once complete. “Software" correlators, on the other hand,make use of commodity server machines and a correlation algorithm coded in a high–level lan-guage. They are inherently much more flexible and can be developed – and modified – muchmore rapidly than purpose–built “hardware" correlators. Software correlators are thus a naturalchoice for testing new RFI detection and mitigation techniques for interferometers. The ease withwhich software correlators can be adapted to test RFI detection algorithms is demonstrated bythe addition of kurtosis detection and plotting to the widely used DiFX software correlator, whichhighlights previously unknown short–duration RFI at the Hancock VLBA station.RFI mitigation workshop - RFI2010,March 29-31, 2010Groningen, the Netherlands∗Speaker.c© Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licen. http://pos.sissa.it/Software correlators as testbeds Adam Deller1. Interferometry, cross–correlators and RFIRadiotelescope arrays are inherently more robust against RFI han single dishes, due to thespatial separation between the array elements. However, sources of RFI which are visible to morethan one element will not be rejected during the correlationprocess and will corrupt the measuredvisibilities between those pairs of elements (baselines) and then the final interferometric image.This problem is particularly severe for short baselines, where interfering sources are least likely tohave been attenuated significantly before being received atneighbouring stations.The cross–correlation algorithm used by radio interferometers to estimate the visibility func-tion has been extensively reviewed elsewhere (see e.g. [1] and [2]) and is not re–derived here. Themost important point of note is that cross–correlators can be built in an FX style (where the sampledbaseband data is segmented and channelised on a per–stationbas s, and then cross-multiplied) oran XF style (in which the data streams for every baseline are cross–correlated with a range of lags,and the resultant lag spectrum is periodically Fourier transformed to generated the final visibilityspectrum). From the point of view of RFI detection and mitigation, the FX algorithm offers greaterflexibility, since it provides access to the final channelised visibilities at higher time resolution.Previous efforts to mitigate RFI in interferometric arrayshave focused on identification andexcision/correction of the RFI either before [3] or after [4] the correlator. Integrating RFI mitigationalgorithmsinsidethe correlator, however, exploits the advantages of both appro ches:1. Unlike post–correlation techniques, inspection of the data at high time resolution is possible,providing better signal–to–noise for the detection of faint, short–duration RFI;2. Likewise, if the duty cycle of the RFI is low, excision of the RFI at high time resolution ispossible and will result in the loss of less good data;3. Unlike pre-correlation techniques, no new steps are inserted into the serial signal processingchain, minimising the chance of introducing unwanted effects; and4. Utilising the processing which is already undertaken in the correlator allows potentially morepowerful techniques to be applied.However, incorporating RFI detection and mitigation inside the cross–correlator is more chal-lenging than traditional approaches, since:1. The data can only be inspected on–the–fly;2. The implementation of the RFI algorithms cannot consume resources or otherwise imposeconstraints that would disturb the calculation of the visibility function; and3. In most correlators, access to intermediate data products s h as partially integrated visibilityspectra is limited.Since software correlators can expose intermediate data products in a convenient (software–accessible) form, they are ideal for testing new RFI detection and mitigation techniques, with amuch shorter development time than would be required on a hardware correlator. The DiFX soft-ware correlator [5], which is the subject of this investigaton, is briefly described below.2Software correlators as testbeds Adam Deller1.1 The DiFX correlatorThe Distributed FX (DiFX) software correlator is describedin detail in [5], and its relevantqualities for high time resoution RFI studies are briefly summarised here. As the name suggests,it is an FX style correlator, which distributes data across multiple computing nodes using time–division multiplexing. The length of the time slices (termed “sub–integrations" below) is com-pletely configurable, down to the inverse of the final spectral resolution, and the spectral resolutionitself is completely configurable. DiFX thus offers a very flexible platform for inspecting the RFIcharacteristics of interferometric data. DiFX is used for production correlation of both the VeryLong Baseline Array (VLBA) in the USA and the Long Baseline Array (LBA) in Australia, pro-viding ample opportunities to test RFI algorithms on real data.DiFX is written in C++, but uses optimised C vector librariesto carry out the bulk of corre-lation operations. This allows both a convenient and comprehensible high–level design, and goodperformance. Access to the vectors which hold (for example)th sampled baseband data and thevisibility results is easily provided through pointers, and since the correlator runs in an unclockedfashion (in contrast to purpose–built hardware correlators using Field Programmable Gate Arraysor custom logic, which will generically be referred to as Digital Signal Processing – DSP – boardsbelow) there are no timing constraints on the addition of extra RFI–related processing.2. Kurtosis–based RFI detectionKurtosis measures the “peakedness" of the probability distribution of a real–valued randomvariable. It is defined as the fourth central moment of the variable divided by the square of thesecond central moment (where the second central moment is also the variance). Closely related isthe “kurtosis excess", which is equal to the kurtosis minus 3. The kurtosis of a normally distributedvariable is equal to 3, and hence the kurtosis excess of a normally distributed variable is 0.Since the explicit assumption of radio interferometry is that bona fide radio astronomical sig-nals are normally distributed, the calculation of kurtosisprovides a useful diagnostic of the presenceof RFI. A continuously present sinusoid has a kurtosis excess< 0 (its distribution isplatykurtic),while impulsive bursts lead to a kurtosis excess> 0 (a leptokurticdistribution). The effects of RFIwith varying duty cycles on kurtosis in a radio astronomy context are investigated in [6].In practice, the central moments of the signal are not known and must be estimated from asample of data. This is commonly achieved using unbiased estimators of the second and fourthcumulants (“k statistics") in place of the second and fourthcentral moments, respectively.In the ideal case of a perfect filterbank, each output spectral h nnel can be treated as anindependent time series and standard time–domain kurtosisanalysis can be applied. However,this straightforward approach is complicated by the fact tha cross-correlators window data in thetime domain, produce complex channelised data, and result in non-negligible leakage between thespectral channels. A more general concept of spectral kurtosis (SK; [7]) has been developed whichaccounts for these complications, while preserving the properties of kurtosis excess (Gaussian inputdata gives SK = 0, impulsive RFI yields SK > 0, and continuous sinusoids yield SK < 0). Thekurtosis calculation implemented in DiFX is somewhat simplified compared to the newly developedSK concept, but illustrates the principles of RFI detectionand the ease with which the algorithmcan be incorporated into a software correlator.3Software correlators as testbeds Adam Deller3. Implementation of kurtosis detection in DiFXAs an FX style correlator, DiFX already produced channelised data for each antenna, which isavailable in the code before the cross-multiplication to form visibilities. Over the course of a sub–integration, the 1st, 2nd, 3rd and 4th raw moments (m1, m2, m3 andm4) for each (complex) spectralchannel were maintained. At the end of each sub–integration, the 2nd and 4th central momentsu2 andu4 were calculated from the raw moments, and the kurtosis for each channel calculated asu4/u22. The calculated kurtosis values were written to a text file for later analysis. The durationof the sub–integration can be (and was) varied to times less than a millisecond, using the existingsoftware correlator infrastructure.This simple approach yields a biased estimation of the kurtosis, for the reasons given in Sec-tion 2 above. At the time of implementation, the author was not aware of the SK concept, which iswhy this simple implementation was made. Converting the existing simple implementation to thefull SK calculation will be trivial, however, and is coveredin Section 5. Nevertheless, Section 4shows that even a biased estimation of the kurtosis can be used to show the presence and spectralcharacteristics of RFI.The greatest value of the kurtosis implementation in DiFX, however, is as a proof of conceptfor the rapid prototyping of RFI algorithms. In total,∼100 lines of code were necessary for thekurtosis calculation, bookkeeping and storage. This is much less than would be required on a sys-tem based on DSP boards, due to the abstraction made possibleby th use of high level languages.Implementing and testing this additional code required only a few hours of coding. Compared tothe development which would be required on a DSP board correlator, this is extremely rapid.4. ResultsTo test the kurtosis implementation, a VLBA test experimentfrom 22 June 2009 at 1650 MHzwas correlated using DiFX with kurtosis calculation enabled. The spectrum around 1650 MHz ismoderately contaminated with RFI at many VLBA sites. The results below show a single 8 MHzband of right circular polarisation, spanning 1650.49 MHz to 1658.49 MHz. Auto– and cross–correlation spectra were integrated for a standard value of2 seconds, while the kurtosis calculationswere integrated for 100 ms, providing a higher time resolutin view of the RFI.Figure 1 shows a sample of the results from two stations. Three adjacent 100 ms kurtosis inte-grations are shown for the Fort Davis (left panel) and Hancock (right panel) stations, with the auto-correlations shown for comparison. The autocorrelation spectra are relatively clean for both stations– the spikes at 1 MHz intervals are the injected pulse calibration tones, used for removing instru-mental delays. Small deviations from the expected smooth bandpass are seen for the Hancock sta-tion, but the location and magnitude of the RFI is not obvious. The kurtosis results, however, showseveral strong, rapidly varying RFI sources at Hancock. Movies made from these plots, which showthe varying RFI more clearly, are available atht p://www.aoc.nrao.edu/~adeller/rfi/.The source of the RFI at the Hancock station is not known. It isclearly variable on mstimescales and is somewhat spectrally confined, although ithas a substantial bandwidth of around0.5 MHz. Other Hancock bands exhibit similar characteristics, ncluding in the protected 1660 –1670 MHz range. When kurtosis–based blanking (see Section 5) is implemented, the data which isclearly RFI–dominated could easily be excised, recoveringmost of the Hancock data.4Software correlators as testbeds Adam DellerThe measured receiver noise temperature at Hancock has beeno s rved to be erratic at theband encompassing these observations, and the knowledge gained from these tests has been passedon to NRAO staff to aid in the search for potential self–generated RFI, which might possibly orig-inate in the noise injection circuit. Obtaining high signalto noise results on short timescales usingthe kurtosis output of DiFX, which would not possible with the normal cross and/or autocorrela-tions, has provided useful information for diagnosing thisproblem. 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelation 0 2 4 6 8 10 12 1651 1652 1653 1654 1655 1656 1657 1658Arbitrary rescaled amplitudesFrequency (MHz)Subintegration kurtosisFull integration autocorrelationFigure 1: Autocorrelation (green) and kurtosis (red) for several successive 100 ms snapshots during theobservation, time running from top to bottom. The Fort Davisstation is shown in the left hand column, withno discernible RFI. The Hancock station is shown in the righthand column, and non-Gaussian signals areclearly present based on the kurtosis results. The autocorrelation amplitudes in green have been arbitrarilyscaled for convenient plotting on the same scale as the kurtosis measurements. The autocorrelation ampli-tudes remain constant, as they are taken from a 2 second integration covering this 300 ms period. Kurtosis(not kurtosis excess) is shown. It is clear that the RFI varies dramatically on timescales shorter than 100 ms.5Software correlators as testbeds Adam Deller5. Conclusions and future workA simple kurtosis estimator has been successfully implemented in the DiFX software cor-relator. This initial foray into high time resolution in–correlator RFI detection was made withminimal effort, requiring only an afternoon of code development and testing, and provided usefulinformation on previously unknown RFI at the Hancock VLBA station. The present simplisticimplementation should soon be updated to the unbiased SK estimator, and a thorough analysis ofthe effects of the coarse front–end quantisation should also be undertaken. Ultimately, it may bepossible to generalise the SK concept to the crosscorrelation outputs of the correlator.With or without these improvements, however, the calculated kurtosis could be used to flagdata on the fly (on timescales shorter than a full integration) or to produce a flag table which couldbe inspected and applied later in offline analysis. The latter case, whilst allowing the flexibility todeselect some flags, loses the ability to save data which has been affected for only a short subsetof a given integration. In either case, this approach is still an excision procedure which ultimatelyresults in data loss. "Kurtosis blanking" and "kurtosis flaggin " will be added as DiFX features inthe near feature, and the usability and presentation of the sav d kurtosis results will be improved.Development of other forms of in–correlator RFI mitigationare planned for the DiFX corre-lator. One example is the rejection of bright sources outside the desired array field of view, whichcan be achieved through field of view shaping using tapered time integrations (see [8]). This willallow a determination of the relative trade-off between accura y of the tapering function and theperformance overhead incurred. The success of the kurtosisdevelopment detailed in this workindicates that these approaches can be rapidly and effectively implemented.References[1] A. R. Thompson, J. M. Moran, & G. W. Swenson, Jn.,I terferometry and synthesis in radioastronomy, 2nd ed., Wiley, New York 1994[2] J. D. Romney, 1999,Cross–correlators, in Synthesis Imaging in Radio Astronomy II, ASP ConferenceSeries,180, 57[3] W. A. Baan, P. A. Fridman & R. P. Millenaar, 2004,Radio Frequency Interference Mitigation at theWesterbork Synthesis Radio Telescope: Algorithms, Test Observations, and System Implementation,AJ, 128, 933[4] R. Athreya, 2009,A New Approach to Mitigation of Radio Frequency Interferencin InterferometricData, ApJ, 696, 885[5] A. Deller, S. J. Tingay, M. Bailes, & C. West, 2007,DiFX: A Software Correlator for Very LongBaseline Interferometry Using Multiprocessor Computing Environments, PASP, 119, 318[6] R. D. de Roo, S. Misra, & C. S. Ruf, 2007,Sensitivity of the Kurtosis Statistic as a Detector of PulsedSinusoidal RFI, IEEE Transactions on Geoscience and Remote Sensing, 45, 1938[7] G. M. Nita, D. E. Gary, Z. Liu, G. J. Hurford, & S. M. White, 2007,Radio Frequency InterferenceExcision Using Spectral-Domain Statistics, PASP, 119, 805[8] C. J. Lonsdale, S. S. Doeleman, & D. Oberoi, 2004,Efficient Imaging Strategies For Next-GenerationRadio Array, Experimental Astronomy, 17, 3456

Adam Deller

Crossref

University of Groningen Research Database

  
 University of Groningen
Software correlators as testbeds for RFI algorithms
Deller, Adam
Published in:
Proceedings of Science
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2010
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Deller, A. (2010). Software correlators as testbeds for RFI algorithms. Proceedings of Science.
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the
author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the
number of authors shown on this cover page is limited to 10 maximum.
Download date: 11-02-2018
ar
X
iv
:1
01
2.
03
25
v1
  [
as
tro
-p
h.I
M
]  
1 D
ec
 20
10 Software correlators as testbeds for RFI algorithms
Adam Deller∗
National Radio Astronomy Observatory
E-mail: adeller@nrao.edu
In–correlator techniques offer the possibility of identifying and/or excising radio frequency inter-
ference (RFI) from interferometric observations at much higher time and/or frequency resolution
than is generally possible with the final visibility dataset. Due to the considerable computational
requirements of the correlation procedure, cross–correlators have most commonly been imple-
mented using high–speed digital signal processing boards, which typically require long develop-
ment times and are difficult to alter once complete. “Software" correlators, on the other hand,
make use of commodity server machines and a correlation algorithm coded in a high–level lan-
guage. They are inherently much more flexible and can be developed – and modified – much
more rapidly than purpose–built “hardware" correlators. Software correlators are thus a natural
choice for testing new RFI detection and mitigation techniques for interferometers. The ease with
which software correlators can be adapted to test RFI detection algorithms is demonstrated by
the addition of kurtosis detection and plotting to the widely used DiFX software correlator, which
highlights previously unknown short–duration RFI at the Hancock VLBA station.
RFI mitigation workshop - RFI2010,
March 29-31, 2010
Groningen, the Netherlands
∗Speaker.
c© Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. http://pos.sissa.it/
Software correlators as testbeds Adam Deller
1. Interferometry, cross–correlators and RFI
Radiotelescope arrays are inherently more robust against RFI than single dishes, due to the
spatial separation between the array elements. However, sources of RFI which are visible to more
than one element will not be rejected during the correlation process and will corrupt the measured
visibilities between those pairs of elements (baselines) and then the final interferometric image.
This problem is particularly severe for short baselines, where interfering sources are least likely to
have been attenuated significantly before being received at neighbouring stations.
The cross–correlation algorithm used by radio interferometers to estimate the visibility func-
tion has been extensively reviewed elsewhere (see e.g. [1] and [2]) and is not re–derived here. The
most important point of note is that cross–correlators can be built in an FX style (where the sampled
baseband data is segmented and channelised on a per–station basis, and then cross-multiplied) or
an XF style (in which the data streams for every baseline are cross–correlated with a range of lags,
and the resultant lag spectrum is periodically Fourier transformed to generated the final visibility
spectrum). From the point of view of RFI detection and mitigation, the FX algorithm offers greater
flexibility, since it provides access to the final channelised visibilities at higher time resolution.
Previous efforts to mitigate RFI in interferometric arrays have focused on identification and
excision/correction of the RFI either before [3] or after [4] the correlator. Integrating RFI mitigation
algorithms inside the correlator, however, exploits the advantages of both approaches:
1. Unlike post–correlation techniques, inspection of the data at high time resolution is possible,
providing better signal–to–noise for the detection of faint, short–duration RFI;
2. Likewise, if the duty cycle of the RFI is low, excision of the RFI at high time resolution is
possible and will result in the loss of less good data;
3. Unlike pre-correlation techniques, no new steps are inserted into the serial signal processing
chain, minimising the chance of introducing unwanted effects; and
4. Utilising the processing which is already undertaken in the correlator allows potentially more
powerful techniques to be applied.
However, incorporating RFI detection and mitigation inside the cross–correlator is more chal-
lenging than traditional approaches, since:
1. The data can only be inspected on–the–fly;
2. The implementation of the RFI algorithms cannot consume resources or otherwise impose
constraints that would disturb the calculation of the visibility function; and
3. In most correlators, access to intermediate data products such as partially integrated visibility
spectra is limited.
Since software correlators can expose intermediate data products in a convenient (software–
accessible) form, they are ideal for testing new RFI detection and mitigation techniques, with a
much shorter development time than would be required on a hardware correlator. The DiFX soft-
ware correlator [5], which is the subject of this investigation, is briefly described below.
2
Software correlators as testbeds Adam Deller
1.1 The DiFX correlator
The Distributed FX (DiFX) software correlator is described in detail in [5], and its relevant
qualities for high time resoution RFI studies are briefly summarised here. As the name suggests,
it is an FX style correlator, which distributes data across multiple computing nodes using time–
division multiplexing. The length of the time slices (termed “sub–integrations" below) is com-
pletely configurable, down to the inverse of the final spectral resolution, and the spectral resolution
itself is completely configurable. DiFX thus offers a very flexible platform for inspecting the RFI
characteristics of interferometric data. DiFX is used for production correlation of both the Very
Long Baseline Array (VLBA) in the USA and the Long Baseline Array (LBA) in Australia, pro-
viding ample opportunities to test RFI algorithms on real data.
DiFX is written in C++, but uses optimised C vector libraries to carry out the bulk of corre-
lation operations. This allows both a convenient and comprehensible high–level design, and good
performance. Access to the vectors which hold (for example) the sampled baseband data and the
visibility results is easily provided through pointers, and since the correlator runs in an unclocked
fashion (in contrast to purpose–built hardware correlators using Field Programmable Gate Arrays
or custom logic, which will generically be referred to as Digital Signal Processing – DSP – boards
below) there are no timing constraints on the addition of extra RFI–related processing.
2. Kurtosis–based RFI detection
Kurtosis measures the “peakedness" of the probability distribution of a real–valued random
variable. It is defined as the fourth central moment of the variable divided by the square of the
second central moment (where the second central moment is also the variance). Closely related is
the “kurtosis excess", which is equal to the kurtosis minus 3. The kurtosis of a normally distributed
variable is equal to 3, and hence the kurtosis excess of a normally distributed variable is 0.
Since the explicit assumption of radio interferometry is that bona fide radio astronomical sig-
nals are normally distributed, the calculation of kurtosis provides a useful diagnostic of the presence
of RFI. A continuously present sinusoid has a kurtosis excess < 0 (its distribution is platykurtic),
while impulsive bursts lead to a kurtosis excess > 0 (a leptokurtic distribution). The effects of RFI
with varying duty cycles on kurtosis in a radio astronomy context are investigated in [6].
In practice, the central moments of the signal are not known and must be estimated from a
sample of data. This is commonly achieved using unbiased estimators of the second and fourth
cumulants (“k statistics") in place of the second and fourth central moments, respectively.
In the ideal case of a perfect filterbank, each output spectral channel can be treated as an
independent time series and standard time–domain kurtosis analysis can be applied. However,
this straightforward approach is complicated by the fact that cross-correlators window data in the
time domain, produce complex channelised data, and result in non-negligible leakage between the
spectral channels. A more general concept of spectral kurtosis (SK; [7]) has been developed which
accounts for these complications, while preserving the properties of kurtosis excess (Gaussian input
data gives SK = 0, impulsive RFI yields SK > 0, and continuous sinusoids yield SK < 0). The
kurtosis calculation implemented in DiFX is somewhat simplified compared to the newly developed
SK concept, but illustrates the principles of RFI detection and the ease with which the algorithm
can be incorporated into a software correlator.
3
Software correlators as testbeds Adam Deller
3. Implementation of kurtosis detection in DiFX
As an FX style correlator, DiFX already produced channelised data for each antenna, which is
available in the code before the cross-multiplication to form visibilities. Over the course of a sub–
integration, the 1st, 2nd, 3rd and 4th raw moments (m1, m2, m3 and m4) for each (complex) spectral
channel were maintained. At the end of each sub–integration, the 2nd and 4th central moments
u2 and u4 were calculated from the raw moments, and the kurtosis for each channel calculated as
u4/u
2
2. The calculated kurtosis values were written to a text file for later analysis. The duration
of the sub–integration can be (and was) varied to times less than a millisecond, using the existing
software correlator infrastructure.
This simple approach yields a biased estimation of the kurtosis, for the reasons given in Sec-
tion 2 above. At the time of implementation, the author was not aware of the SK concept, which is
why this simple implementation was made. Converting the existing simple implementation to the
full SK calculation will be trivial, however, and is covered in Section 5. Nevertheless, Section 4
shows that even a biased estimation of the kurtosis can be used to show the presence and spectral
characteristics of RFI.
The greatest value of the kurtosis implementation in DiFX, however, is as a proof of concept
for the rapid prototyping of RFI algorithms. In total, ∼100 lines of code were necessary for the
kurtosis calculation, bookkeeping and storage. This is much less than would be required on a sys-
tem based on DSP boards, due to the abstraction made possible by the use of high level languages.
Implementing and testing this additional code required only a few hours of coding. Compared to
the development which would be required on a DSP board correlator, this is extremely rapid.
4. Results
To test the kurtosis implementation, a VLBA test experiment from 22 June 2009 at 1650 MHz
was correlated using DiFX with kurtosis calculation enabled. The spectrum around 1650 MHz is
moderately contaminated with RFI at many VLBA sites. The results below show a single 8 MHz
band of right circular polarisation, spanning 1650.49 MHz to 1658.49 MHz. Auto– and cross–
correlation spectra were integrated for a standard value of 2 seconds, while the kurtosis calculations
were integrated for 100 ms, providing a higher time resolution view of the RFI.
Figure 1 shows a sample of the results from two stations. Three adjacent 100 ms kurtosis inte-
grations are shown for the Fort Davis (left panel) and Hancock (right panel) stations, with the auto-
correlations shown for comparison. The autocorrelation spectra are relatively clean for both stations
– the spikes at 1 MHz intervals are the injected pulse calibration tones, used for removing instru-
mental delays. Small deviations from the expected smooth bandpass are seen for the Hancock sta-
tion, but the location and magnitude of the RFI is not obvious. The kurtosis results, however, show
several strong, rapidly varying RFI sources at Hancock. Movies made from these plots, which show
the varying RFI more clearly, are available at http://www.aoc.nrao.edu/~adeller/rfi/.
The source of the RFI at the Hancock station is not known. It is clearly variable on ms
timescales and is somewhat spectrally confined, although it has a substantial bandwidth of around
0.5 MHz. Other Hancock bands exhibit similar characteristics, including in the protected 1660 –
1670 MHz range. When kurtosis–based blanking (see Section 5) is implemented, the data which is
clearly RFI–dominated could easily be excised, recovering most of the Hancock data.
4
Software correlators as testbeds Adam Deller
The measured receiver noise temperature at Hancock has been observed to be erratic at the
band encompassing these observations, and the knowledge gained from these tests has been passed
on to NRAO staff to aid in the search for potential self–generated RFI, which might possibly orig-
inate in the noise injection circuit. Obtaining high signal to noise results on short timescales using
the kurtosis output of DiFX, which would not possible with the normal cross and/or autocorrela-
tions, has provided useful information for diagnosing this problem.
 0
 2
 4
 6
 8
 10
 12
 1651 1652 1653 1654 1655 1656 1657 1658
Ar
bi
tra
ry
 re
sc
al
ed
 a
m
pl
itu
de
s
Frequency (MHz)
Subintegration kurtosis
Full integration autocorrelation
 0
 2
 4
 6
 8
 10
 12
 1651 1652 1653 1654 1655 1656 1657 1658
Ar
bi
tra
ry
 re
sc
al
ed
 a
m
pl
itu
de
s
Frequency (MHz)
Subintegration kurtosis
Full integration autocorrelation
 0
 2
 4
 6
 8
 10
 12
 1651 1652 1653 1654 1655 1656 1657 1658
Ar
bi
tra
ry
 re
sc
al
ed
 a
m
pl
itu
de
s
Frequency (MHz)
Subintegration kurtosis
Full integration autocorrelation
 0
 2
 4
 6
 8
 10
 12
 1651 1652 1653 1654 1655 1656 1657 1658
Ar
bi
tra
ry
 re
sc
al
ed
 a
m
pl
itu
de
s
Frequency (MHz)
Subintegration kurtosis
Full integration autocorrelation
 0
 2
 4
 6
 8
 10
 12
 1651 1652 1653 1654 1655 1656 1657 1658
Ar
bi
tra
ry
 re
sc
al
ed
 a
m
pl
itu
de
s
Frequency (MHz)
Subintegration kurtosis
Full integration autocorrelation
 0
 2
 4
 6
 8
 10
 12
 1651 1652 1653 1654 1655 1656 1657 1658
Ar
bi
tra
ry
 re
sc
al
ed
 a
m
pl
itu
de
s
Frequency (MHz)
Subintegration kurtosis
Full integration autocorrelation
Figure 1: Autocorrelation (green) and kurtosis (red) for several successive 100 ms snapshots during the
observation, time running from top to bottom. The Fort Davis station is shown in the left hand column, with
no discernible RFI. The Hancock station is shown in the right hand column, and non-Gaussian signals are
clearly present based on the kurtosis results. The autocorrelation amplitudes in green have been arbitrarily
scaled for convenient plotting on the same scale as the kurtosis measurements. The autocorrelation ampli-
tudes remain constant, as they are taken from a 2 second integration covering this 300 ms period. Kurtosis
(not kurtosis excess) is shown. It is clear that the RFI varies dramatically on timescales shorter than 100 ms.
5
Software correlators as testbeds Adam Deller
5. Conclusions and future work
A simple kurtosis estimator has been successfully implemented in the DiFX software cor-
relator. This initial foray into high time resolution in–correlator RFI detection was made with
minimal effort, requiring only an afternoon of code development and testing, and provided useful
information on previously unknown RFI at the Hancock VLBA station. The present simplistic
implementation should soon be updated to the unbiased SK estimator, and a thorough analysis of
the effects of the coarse front–end quantisation should also be undertaken. Ultimately, it may be
possible to generalise the SK concept to the crosscorrelation outputs of the correlator.
With or without these improvements, however, the calculated kurtosis could be used to flag
data on the fly (on timescales shorter than a full integration) or to produce a flag table which could
be inspected and applied later in offline analysis. The latter case, whilst allowing the flexibility to
deselect some flags, loses the ability to save data which has been affected for only a short subset
of a given integration. In either case, this approach is still an excision procedure which ultimately
results in data loss. "Kurtosis blanking" and "kurtosis flagging" will be added as DiFX features in
the near feature, and the usability and presentation of the saved kurtosis results will be improved.
Development of other forms of in–correlator RFI mitigation are planned for the DiFX corre-
lator. One example is the rejection of bright sources outside the desired array field of view, which
can be achieved through field of view shaping using tapered time integrations (see [8]). This will
allow a determination of the relative trade-off between accuracy of the tapering function and the
performance overhead incurred. The success of the kurtosis development detailed in this work
indicates that these approaches can be rapidly and effectively implemented.
References
[1] A. R. Thompson, J. M. Moran, & G. W. Swenson, Jn., Interferometry and synthesis in radio
astronomy, 2nd ed., Wiley, New York 1994
[2] J. D. Romney, 1999, Cross–correlators, in Synthesis Imaging in Radio Astronomy II, ASP Conference
Series, 180, 57
[3] W. A. Baan, P. A. Fridman & R. P. Millenaar, 2004, Radio Frequency Interference Mitigation at the
Westerbork Synthesis Radio Telescope: Algorithms, Test Observations, and System Implementation,
AJ, 128, 933
[4] R. Athreya, 2009, A New Approach to Mitigation of Radio Frequency Interference in Interferometric
Data, ApJ, 696, 885
[5] A. Deller, S. J. Tingay, M. Bailes, & C. West, 2007, DiFX: A Software Correlator for Very Long
Baseline Interferometry Using Multiprocessor Computing Environments, PASP, 119, 318
[6] R. D. de Roo, S. Misra, & C. S. Ruf, 2007, Sensitivity of the Kurtosis Statistic as a Detector of Pulsed
Sinusoidal RFI, IEEE Transactions on Geoscience and Remote Sensing, 45, 1938
[7] G. M. Nita, D. E. Gary, Z. Liu, G. J. Hurford, & S. M. White, 2007, Radio Frequency Interference
Excision Using Spectral-Domain Statistics, PASP, 119, 805
[8] C. J. Lonsdale, S. S. Doeleman, & D. Oberoi, 2004, Efficient Imaging Strategies For Next-Generation
Radio Array, Experimental Astronomy, 17, 345
6


Software correlators as testbeds for RFI algorithms

Abstract

Similar works

Full text

Available Versions

University of Groningen

ARTS repository - University of Groningen

ARTS repository - University of Groningen

Crossref

University of Groningen Research Database

NARCIS