Alexov, A.

Anderson, K.R.

Båhren, L.

Grießmeier, J.-M.

Renting, G.A.

Wise, M.

English

For decades now, scientific data volumes have experienced relentless, exponential growth. As a result, legacy astronomical data formats are straining under a burden not conceived when these formats were first introduced. With future astronomical projects ensuring this trend, ASTRON and the LOFAR project are exploring the use of the Hierarchical Data Format, version 5 (HDF5), for LOFAR radio data encapsulation. Most of LOFAR’s standard data products will be stored using the HDF5 format. In addition, HDF5 analogues for traditional radio data structures such as visibility data and spectral image cubes are also being developed. The HDF5 libraries allow for the construction of distributed, entirely unbounded files. The nature of the HDF5 format further provides the ability to custom design a data encapsulation format, specifying hierarchies, content and attributes. The LOFAR project has designed several data formats that will accommodate and house all LOFAR data products, the primary styles and kinds of which are presented in this paper. With proper development and support, it is hoped that these data formats will be adopted by other astronomical projects as they, too, attempt to grapple with a future filled with mountains of data

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)LOFAR and HDF5: Toward a new radio data standardAnderson, K.R.; Alexov, A.; Båhren, L.; Grießmeier, J.-M.; Wise, M.; Renting, G.A.Publication date2010Document VersionFinal published versionPublished inProceedings of ScienceLink to publicationCitation for published version (APA):Anderson, K. R., Alexov, A., Båhren, L., Grießmeier, J-M., Wise, M., & Renting, G. A. (2010).LOFAR and HDF5: Toward a new radio data standard. Proceedings of Science, 112, 062.http://pos.sissa.it/archive/conferences/112/062/ISKAF2010_062.pdfGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:09 Mar 2023PoS(ISKAF2010)062LOFAR and HDF5: Toward a New Radio DataStandardKenneth R. Anderson∗ †University of Amsterdam (UvA), Amsterdam, The NetherlandsE-mail: k.r.anderson@uva.nlAnastasia AlexovUniversity of Amsterdam (UvA), Amsterdam, The NetherlandsE-mail: a.alexov@uva.nlLars BåhrenRadboud University of Nijmegen, Nijmegen, The NetherlandsE-mail: lbaehren@gmail.comJean-Mathias Grießmeier,Netherlands Institute for Radio Astronomy (ASTRON), Dwingeloo, The NetherlandsE-mail: griessmeier@astron.nlMichael WiseNetherlands Institute for Radio Astronomy (ASTRON), Dwingeloo, The NetherlandsGerard Adriaan RentingNetherlands Institute for Radio Astronomy (ASTRON), Dwingeloo, The NetherlandsFor decades now, scientific data volumes have experienced relentless, exponential growth. As aresult, legacy astronomical data formats are straining under a burden not conceived when theseformats were first introduced. With future astronomical projects ensuring this trend, ASTRONand the LOFAR project are exploring the use of the Hierarchical Data Format, version 5 (HDF5),for LOFAR radio data encapsulation. Most of LOFAR’s standard data products will be storedusing the HDF5 format. In addition, HDF5 analogues for traditional radio data structures such asvisibility data and spectral image cubes are also being developed. The HDF5 libraries allow forthe construction of distributed, entirely unbounded files. The nature of the HDF5 format furtherprovides the ability to custom design a data encapsulation format, specifying hierarchies, contentand attributes. The LOFAR project has designed several data formats that will accommodate andhouse all LOFAR data products, the primary styles and kinds of which are presented in this paper.With proper development and support, it is hoped that these data formats will be adopted by otherastronomical projects as they, too, attempt to grapple with a future filled with mountains of data.ISKAF2010 Science Meeting - ISKAF2010June 10-14, 2010Assen, the Netherlands∗Speaker.†A big thanks to all those who offered critique and commentary during this process.c© Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. http://pos.sissa.it/PoS(ISKAF2010)062LOFAR and HDF5 Kenneth R. Anderson1. IntroductionThe promising advent of the LOFAR telescope’s operational epoch holds forth both greatscientifiic potential and challenges to current and legacy information technologies: volume andcomplexity of the data will continue to push the envelope of commonly used data protocols.Recognizing that this envelope is already strained, the LOFAR project has embarked on anambitious project to design and define a set of radio data standard formats that are capable ofencapsulating the full spectrum of, not just LOFAR data products, but astronomical radio data ingeneral.It is with this ambition in mind that the LOFAR data formats group have been developingthese format specifications and associated software infrastructure, an effort now ongoing for overtwo years. It was determined that HDF5 would be a robust, viable data framework that can handlethe size, scope, diversity, distributed nature and parallel I/O processing requirements of LOFARdata. This work also has potential use beyond the radio community. New large scale opticaltelescopes such as the LSST are also investigating the viability of using HDF5. Furthermore, the20 year history of HDF and its continuing use by NASA’s earth orbiting/observing missions ensurebroad, ongoing use and support.In addition to the format descriptions themselves, the LOFAR project is currently developing aset of software tools for creating and working with these formats. The Data Access Library (DAL)in C++, along with an associated Python interface (pyDAL), are designed to allow for the easyconstruction and manipulation of these data formats. There are also a number of tools alreadyavailable to read and visualize HDF5 files, such as HDFView, ViSiT, PyTables, h5py and IDL.2. Data Files in the Modern AgeWith the invention and subsequent commercialization of the charge-coupled device (CCD),data volumes in the field of astronomy have grown exponentially since the early 1970s (Borne,2009), and, as Table 1 conveys, have now attained volumes that early file and file system protocolsare finding difficult to handle. For future radio astronomy projects, the looming plight is especiallytrue.EPOCH NOMINAL FILE DATA VOLUME1970 210 bytes1980 220 bytes1990 230 bytes2000 240 bytesTable 1: Typical file sizes of scientific datasets have exponentially increased since the 1970’s.Indeed, the path of diminishing utility of legacy protocols is clearly delineated. This loomingpredicament is especially germane to the SKA pathfinder LOFAR project, wherein certain opera-tional modes will be capable of generating datasets comprising hundreds of gigabytes to tens of ter-abytes. Therefore, the LOFAR project has been driven to consider viable alternatives to “standard”astronomical data formats, such as FITS. Though FITS development has sought to add capability2PoS(ISKAF2010)062LOFAR and HDF5 Kenneth R. Andersonto the protocol as needed (ESO-FWG, 2008), such ex post facto approaches should not be expectedto remain viable in the future.EXPOSURE NUMBER OF NUMBER OF FILE SIZE FILE SIZETIME SUBBANDS STATIONS KNOWN MODE SEARCH MODE1 min 248 5 11.2GB 56GB1 min 248 20 11.2GB 244GB10 min 248 5 112GB 560GB10 min 248 10 112GB 1.1TB10 min 248 20 112GB 2.2TB10 min 248 30 112GB 3.3TB20 min 248 5 224GB 1.1TB30 min 248 5 336GB 1.7TB1 hr 248 5 672GB 3.4TB1 hr 248 10 672GB 6.7TB1 hr 248 20 672GB 13.4TB1 hr 248 30 672GB 26.8TB2 hr 248 5 1.3TB 6.7TB12 hr 248 5 8.0TB 40.3TB12 hr 248 15 24.0TB 120.1TBTable 2: Sample, LOFAR Beam-formed dataset sizesRecognizing that legacy file technologies will eventually fail under the growing demands ofnewer and larger scientific datasets, while simultaneously noting that if observational modes ofSKA pathfinder projects like LOFAR suggest anything, it is that legacy technologies would haveto be abandoned, not just because of the volumes of data involved, but also because of the complexnature of the data per se.By way of LOFAR example, Table 2 presents expected data volumes for configurations ofBeam-Formed data observational mode. A once useful unit, the megabyte has been left in the wakeof next generation science.3. LOFAR Data Formats in HDF53.1 LOFAR data and the use HDF5 (Hierarchical Data Format 5)Datasets produced by LOFAR observations will vary tremendously in size. Images, Beam-formed data, Transient Buffer board (TBB) time-series data are expected to produce large files,with the beam-formed and TBB potentially forming files of several tens of terabytes. This, com-bined with the complex nature of the data from certain modes of observation, led the LOFARproject to examine and consider the Hierarchical Data Format (version 5) as a robust and viablesolution to the volume-complexity problem:– HDF5 presents a robust data model, featuring distributed files.3PoS(ISKAF2010)062LOFAR and HDF5 Kenneth R. Anderson– HDF5 data model: “a file system for your data.” which operates through POSIX-style file navi-gation, e.g., path names to objects.OBSERVATION/Image001/Data– Versatile data model accommodates complex data objects and associated metadata.– User-defined file design.– Portable, unbounded file format.– Multi-platform software library for single to massively parallel systems.The following schematic diagrams present two LOFAR file designs using the HDF5 frame-work, while readers should bear in mind that all LOFAR data products have been designed withmaximal parallel form.Figure 1: LOFAR Beam Formed High Level Data Structure; tables/arrays are not shown, but are implied atthe Stokes level. (Alexov et al, 2010)Figure 2: LOFAR Radio Sky Image Cubes, High Level Data Structure; tables/arrays are not shown, but areimplied at the Data group level. (Anderson et al, 2010)4. LOFAR Data Format SpecificationsHDF5 provides a framework allowing users to essentially design their own files to appropri-ately encapsulate and characterize a variety of data.4PoS(ISKAF2010)062LOFAR and HDF5 Kenneth R. AndersonFor ongoing years now, the LOFAR project has been engaged in developing and designinga complete set of specifications for all LOFAR observational data. This has necessarily requireddiffering file designs for differing data, with a certain structural parallelism maintained across allfile designs. These Interface Control Documents (ICD) (shown with their document indentifiers)provide detailed descriptions of all expected LOFAR Data Products.1LOFAR DATA FILE TYPE LOFAR DOCUMENT IDTBB Time Series Data LOFAR-USG-ICD-001Beam-Formed Data LOFAR-USG-ICD-003Radio Sky Image Cubes LOFAR-USG-ICD-004Dynamic Spectrum Data LOFAR-USG-ICD-006Visibility Data LOFAR-USG-ICD-007RM Synthesis Cubes LOFAR-USG-ICD-008Table 3: LOFAR file types and the corresponding Interface Control DocumentsAll LOFAR ICD documents are publicly available and can be downloaded fromhttp://usg.lofar.org/wiki/doku.php?id=documents:lofar_data_products5. Toolsets, Libraries, PackagesSince inception in 1988, the body of libraries and tools available to work with HDF5 files hasgrown substantially. Continued use, maintainance and development is assured with the adoptionof HDF by Boeing, NASA – for the Earth Observing System (EOS) – while the National Oceano-graphic and Atmoshperic Administration adopted HDF for the National Polar-orbiting OperationalEnvironmental Satellite System, NPOESS.In addition to the HDF5 library itself, these packages and interfaces are also available:PACKAGE LOCATIONIDL http://www.ittvis.com/ViSit https://wci.llnl.gov/HDFView http://www.hdfgroup.org/DAL http://usg.lofar.org/pyDAL http://usg.lofar.org/h5py http://h5py.alfven.org/PyTables http://www.pytables.org/Table 4: Some available HDF5 packagesThe LOFAR project is developing the C++ Data Access Library (DAL) and an associatedpython wrapper, pyDAL, which will provide full scope constructors for creating and accessingLOFAR data products. Further tools and packages are catalogued at the HDF Group website,http://www.hdfgroup.org/tools5desc.html1Interface control documents LOFAR-USG-ICD-002, and LOFAR-USG-ICD-005 provide supplemental specifica-tions for LOFAR naming conventions, and therefore are not included in the listing.5PoS(ISKAF2010)062LOFAR and HDF5 Kenneth R. Anderson6. Summary and Future ConsiderationsIn order that adoption of HDF5 in astronomy prove useful in the real world, the LOFARproject is committing resources to help develop the next generation of astronomical tools for, notonly LOFAR data, but other SKA pathfinder projects and, more broadly, astronomical radio datain general. Though our immediate goal is, of course, to meet LOFAR’s many and varied scientificrequirements, we hope these HDF5-based formats will be more broadly useful to our colleagues inthe radio community for their data products as well.The large effort is the development of the Data Access Library (DAL), which will ultimatelyprovide interfaces through FITS, the Casa/AIPS++ Measurement Set and LOFAR HDF5 formats.A python interface to the DAL, pyDAL, is pending development of the DAL. All LOFAR productswill be accessible through DAL and pyDAL tools.Future work involves developing an interface for DS9 and HDF5 LOFAR data files — thiswill allow users to open and examine LOFAR data with the de facto standard astronomical imageviewer.Ultimately we would like to see these formats grow into a true set of standards for radiodata that can meet the demands of the next generation of radio observatories. Such standards aresomething sorely lacking in the radio community at present and something we will certainly needas we move into the SKA era.The large effort by LOFAR to design an HDF5 radio data standard is driven in great part byconsideration that there are no effective standards for astronomical radio data. And, as indicatedearlier, the expected data volumes produced by LOFAR will, in many cases, swamp currentlyemployed file technologies. It is this reality that has led LOFAR on this work, and it is hoped thatother institutes and telescope projects will join this effort toward building a radio data standard,one essential to a cooperative future for radio astronomy.References[1] K. Borne, Data Science Challenges from Distributed Petabyte Astronomical Data Collections:Preparing for the Data Avalanche through Persistence, Parallelization, and Provenance, GMU, 2009.[2] A. Alexov et al, LOFAR Data Format ICD: Beam-formed Data, Document IDLOFAR-USG-ICD-003, Version 2.00.00, ASTRON, 2010.[3] K. Anderson et al, LOFAR Data Format ICD: Radio Sky Image Cubes, Document ID,LOFAR-USG-ICD-004, Version 2.00.00, ASTRON, 2010.[4] A. P. Schoenmakers, MS2 description for LOFAR-CS1, rev. 0.5, ASTRON, 2008.[5] ESO-FWG, ESO Data Interface Control Document, Doc. No. GEN-SPE-ESO-19400-0794, Issue 4,2008.[6] The HDF Group HDF5 File Organization, 2008,http://www.hdfgroup.org/HDF5/Tutor/fileorg.html[7] The HDF Group, Introduction to HDF5, 2006-2008,http://www.hdfgroup.org/HDF5/doc/H5.intro.html[8] LOFAR ICD working group, LOFAR data product descriptions, v1.0, ASTRON, 2010http://usg.lofar.org/wiki/doku.php?id=documents:lofar_data_products6

LOFAR and HDF5: Toward a new radio data standard

https://pure.uva.nl/ws/files/1430586/89441_336358.pdf

LOFAR and HDF5: Toward a new radio data standard

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications