Data Provenance and Management in Radio Astronomy: A Stream Computing Approach

New approaches for data provenance and data management (DPDM) are required for mega science projects like the Square Kilometer Array, characterized by extremely large data volume and intense data rates, therefore demanding innovative and highly efficient computational paradigms. In this context, we explore a stream-computing approach with the emphasis on the use of accelerators. In particular, we make use of a new generation of high performance stream-based parallelization middleware known as InfoSphere Streams. Its viability for managing and ensuring interoperability and integrity of signal processing data pipelines is demonstrated in radio astronomy. IBM InfoSphere Streams embraces the stream-computing paradigm. It is a shift from conventional data mining techniques (involving analysis of existing data from databases) towards real-time analytic processing. We discuss using InfoSphere Streams for effective DPDM in radio astronomy and propose a way in which InfoSphere Streams can be utilized for large antennae arrays. We present a case-study: the InfoSphere Streams implementation of an autocorrelating spectrometer, and using this example we discuss the advantages of the stream-computing approach and the utilization of hardware accelerators

Realfast: Real-Time, Commensal Fast Transient Surveys with the Very Large Array

Radio interferometers have the ability to precisely localize and better characterize the properties of sources. This ability is having a powerful impact on the study of fast radio transients, where a few milliseconds of data is enough to pinpoint a source at cosmological distances. However, recording interferometric data at millisecond cadence produces a terabyte-per-hour data stream that strains networks, computing systems, and archives. This challenge mirrors that of other domains of science, where the science scope is limited by the computational architecture as much as the physical processes at play. Here, we present a solution to this problem in the context of radio transients: realfast, a commensal, fast transient search system at the Jansky Very Large Array. Realfast uses a novel architecture to distribute fast-sampled interferometric data to a 32-node, 64-GPU cluster for real-time imaging and transient detection. By detecting transients in situ, we can trigger the recording of data for those rare, brief instants when the event occurs and reduce the recorded data volume by a factor of 1000. This makes it possible to commensally search a data stream that would otherwise be impossible to record. This system will search for millisecond transients in more than 1000 hours of data per year, potentially localizing several Fast Radio Bursts, pulsars, and other sources of impulsive radio emission. We describe the science scope for realfast, the system design, expected outcomes, and ways real-time analysis can help in other fields of astrophysics.Comment: Accepted to ApJS Special Issue on Data; 11 pages, 4 figure

Workshop proceedings: Information Systems for Space Astrophysics in the 21st Century, volume 1

The Astrophysical Information Systems Workshop was one of the three Integrated Technology Planning workshops. Its objectives were to develop an understanding of future mission requirements for information systems, the potential role of technology in meeting these requirements, and the areas in which NASA investment might have the greatest impact. Workshop participants were briefed on the astrophysical mission set with an emphasis on those missions that drive information systems technology, the existing NASA space-science operations infrastructure, and the ongoing and planned NASA information systems technology programs. Program plans and recommendations were prepared in five technical areas: Mission Planning and Operations; Space-Borne Data Processing; Space-to-Earth Communications; Science Data Systems; and Data Analysis, Integration, and Visualization

Computing and data processing

The applications of computers and data processing to astronomy are discussed. Among the topics covered are the emerging national information infrastructure, workstations and supercomputers, supertelescopes, digital astronomy, astrophysics in a numerical laboratory, community software, archiving of ground-based observations, dynamical simulations of complex systems, plasma astrophysics, and the remote control of fourth dimension supercomputers

Data Mining and Machine Learning in Astronomy

We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining process, from data collection through to the interpretation of results. We cover common machine learning algorithms, such as artificial neural networks and support vector machines, applications from a broad range of astronomy, emphasizing those where data mining techniques directly resulted in improved science, and important current and future directions, including probability density functions, parallel algorithms, petascale computing, and the time domain. We conclude that, so long as one carefully selects an appropriate algorithm, and is guided by the astronomical problem at hand, data mining can be very much the powerful tool, and not the questionable black box.Comment: Published in IJMPD. 61 pages, uses ws-ijmpd.cls. Several extra figures, some minor additions to the tex

Working Papers: Astronomy and Astrophysics Panel Reports

The papers of the panels appointed by the Astronomy and Astrophysics survey Committee are compiled. These papers were advisory to the survey committee and represent the opinions of the members of each panel in the context of their individual charges. The following subject areas are covered: radio astronomy, infrared astronomy, optical/IR from ground, UV-optical from space, interferometry, high energy from space, particle astrophysics, theory and laboratory astrophysics, solar astronomy, planetary astronomy, computing and data processing, policy opportunities, benefits to the nation from astronomy and astrophysics, status of the profession, and science opportunities

Pathway to the Square Kilometre Array - The German White Paper -

The Square Kilometre Array (SKA) is the most ambitious radio telescope ever planned. With a collecting area of about a square kilometre, the SKA will be far superior in sensitivity and observing speed to all current radio facilities. The scientific capability promised by the SKA and its technological challenges provide an ideal base for interdisciplinary research, technology transfer, and collaboration between universities, research centres and industry. The SKA in the radio regime and the European Extreme Large Telescope (E-ELT) in the optical band are on the roadmap of the European Strategy Forum for Research Infrastructures (ESFRI) and have been recognised as the essential facilities for European research in astronomy. This "White Paper" outlines the German science and R&D interests in the SKA project and will provide the basis for future funding applications to secure German involvement in the Square Kilometre Array.Comment: Editors: H. R. Kl\"ockner, M. Kramer, H. Falcke, D.J. Schwarz, A. Eckart, G. Kauffmann, A. Zensus; 150 pages (low resolution- and colour-scale images), published in July 2012, language English (including a foreword and an executive summary in German), the original file is available via the MPIfR homepag