WAX: A High Performance Spatial Auto-Correlation Application

We describe the algorithms employed by WAX, a spatial autocorrelation application written in C and C++ which allows for both rapid grouping of multi-epoch apparitions as well as customizable statistical analysis of generated groups. The grouping algorithm, dubbed the swiss cheese algorithm, is designed to handle diverse input databases ranging from the 2MASS working point source database (an all sky database with relatively little coverage depth) to the 2MASS working calibration source database (a database with sparse but very deep coverage). WAX retrieves apparitions and stores groups directly from and to a DBMS, generating optimized C structures and ESQL/C code based on user defined retrieval and output columns. Furthermore, WAX allows generated groups to be spatially indexed via the HTM scheme and provides fast coverage queries for points and small circular areas on the sky. Finally, WAX operates on a declination based sky subdivision, allowing multiple instances to be run simultaneously and independently, further speeding the process of merging apparitions from very large databases. The Two Micron All Sky Survey will use WAX to create merged apparition catalogs from their working point and calibration source databases, linking generated groups to sources in the already publicly available all-sky catalogs. For a given 2MASS source, this will allow astronomers to examine the properties of many related (and as yet unpublished) 2MASS extractions, and further extends the scientific value of the 2MASS data sets

Qserv: a distributed shared-nothing database for the LSST catalog

The LSST project will provide public access to a database catalog that, in its final year, is estimated to include 26 billion stars and galaxies in dozens of trillion detections in multiple petabytes. Because we are not aware of an existing open-source database implementation that has been demonstrated to efficiently satisfy astronomers' spatial self-joining and cross-matching queries at this scale, we have implemented Qserv, a distributed shared-nothing SQL database query system. To speed development, Qserv relies on two successful open-source software packages: the MySQL RDBMS and the Xrootd distributed file system. We describe Qserv's design, architecture, and ability to scale to LSST's data requirements. We illustrate its potential with test results on a 150-node cluster using 55 billion rows and 30 terabytes of simulated data. These results demonstrate the soundness of Qserv's approach and the scale it achieves on today's hardware

RADAR: A Fast, Scalable, and Distributable Archive Inventory Service

The NASA/IPAC Infrared Science Archive (IRSA) has recently deployed the Recursive Archive Digest and Reference (RADAR) service, which returns an inventory of IRSA's holdings in response to a spatial query, and offers one-click download of data and links to IRSA's data access services. RADAR also supports inventories and data access from remote archives; the current implementation supports access to the Multi-mission Archive at STScI (MAST) Spectral and Image Scrapbook and NEDBasic Data. When complete, RADAR will maintain the results of multiple queries in "data collections" and will provide tools that will allow users to augment collections, remove data from them, modify search criteria, resubmit jobs, and check job status. RADAR is supported by an evolution of IRSA's component based architecture. It utilizes a fast estimation service and runs under the Request Management Environment (ROME) funded by NVO

The NASA/IPAC Infrared Science Archive (IRSA): The Demo

This paper describes the services available at the NASA/IPAC Infrared Science Archive (IRSA). Currently there are nearly 250,000 data requests a month, taking advantage of IRSA's data repository which includes 660 million sources (60 catalogs), 10 million images (22 image sets; 10.4 TB) and over 30,000 spectra (7 spectroscopic datasets). These data are the science products of: The Two Micron All Sky Survey (2MASS), The Infrared Astronomical Satellite (IRAS), The Midcourse Space Experiment (MSX), The Submillimeter Wave Astronomy Satellite (SWAS), The Infrared Space Observatory (ISO), The Infrared Telescope in Space (IRTS), The Spitzer First Look Survey (FLS), Spitzer Legacy & Ancillary data, Spitzer Reserved Observations (ROC) and the Spitzer Space Telescope data. IRSA is also seamlessly interoperable with ten remote archives and services: GOODS, ISO, MAST, VizieR, DSS, NVSS, FIRST, HEASARC, NED and JPL, which help expand the available dataset wavelength range from X-ray to radio. The majority of IRSA's image collections are Simple Image Access (SIA) compliant and are available through the Virtual Observatory (VO) data mining tools. The IRSA demo includes IRSA's ¯ve main services: inventory service RADAR, catalog query service Gator, data fusion service OASIS, general search service for complex data collections Atlas, and IRSA's 2MASS Image data access services. IRSA's website is http://irsa.ipac.caltech.edu

Architecture for access to a compute-intensive image mosaic service in the NVO

The National Virtual Observatory (NVO) will provide on-demand access to data collections, data fusion services and compute intensive applications. The paper describes the development of a framework that will support two key aspects of these objectives: a compute engine that will deliver custom image mosaics, and a "request management system," based on an e-business applications server, for job processing, including monitoring, failover and status reporting. We will develop this request management system to support a diverse range of astronomical requests, including services scaled to operate on the emerging computational grid infrastructure. Data requests will be made through existing portals to demonstrate the system: the NASA/IPAC Extragalactic Database (NED), the On-Line Archive Science Information Services (OASIS) at the NASA/IPAC Infrared Science Archive (IRSA); the Virtual Sky service at Caltech's Center for Advanced Computing Research (CACR), and the yourSky mosaic server at the Jet Propulsion Laboratory (JPL)

ZChecker: Finding Cometary Outbursts with the Zwicky Transient Facility

ZChecker is new, automated software for finding, measuring, and visualizing known comets in the Zwicky Transient Facility time-domain survey. ZChecker uses on-line ephemeris generation and survey metadata to identify images of targets of interest in the archive. Photometry of each target is measured, and the images processed with temporal filtering to highlight morphological variations in time. Example outputs show outbursts of comets 29P/Schwassmann-Wachmann 1 and 64P/Swift-Gehrels, and an asymmetric coma at C/2017 M4 (ATLAS)

LSST: From Science Drivers to Reference Design and Anticipated Data Products

We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the solar system, exploring the transient optical sky, and mapping the Milky Way. LSST will be a large, wide-field ground-based system designed to obtain repeated images covering the sky visible from Cerro Pachón in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg^2 field of view, a 3.2-gigapixel camera, and six filters (ugrizy) covering the wavelength range 320–1050 nm. The project is in the construction phase and will begin regular survey operations by 2022. About 90% of the observing time will be devoted to a deep-wide-fast survey mode that will uniformly observe a 18,000 deg^2 region about 800 times (summed over all six bands) during the anticipated 10 yr of operations and will yield a co-added map to r ~ 27.5. These data will result in databases including about 32 trillion observations of 20 billion galaxies and a similar number of stars, and they will serve the majority of the primary science programs. The remaining 10% of the observing time will be allocated to special projects such as Very Deep and Very Fast time domain surveys, whose details are currently under discussion. We illustrate how the LSST science drivers led to these choices of system parameters, and we describe the expected data products and their characteristics

LSST: From Science Drivers to Reference Design and Anticipated Data Products

We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the solar system, exploring the transient optical sky, and mapping the Milky Way. LSST will be a large, wide-field ground-based system designed to obtain repeated images covering the sky visible from Cerro Pachón in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg^2 field of view, a 3.2-gigapixel camera, and six filters (ugrizy) covering the wavelength range 320–1050 nm. The project is in the construction phase and will begin regular survey operations by 2022. About 90% of the observing time will be devoted to a deep-wide-fast survey mode that will uniformly observe a 18,000 deg^2 region about 800 times (summed over all six bands) during the anticipated 10 yr of operations and will yield a co-added map to r ~ 27.5. These data will result in databases including about 32 trillion observations of 20 billion galaxies and a similar number of stars, and they will serve the majority of the primary science programs. The remaining 10% of the observing time will be allocated to special projects such as Very Deep and Very Fast time domain surveys, whose details are currently under discussion. We illustrate how the LSST science drivers led to these choices of system parameters, and we describe the expected data products and their characteristics

LSST: from Science Drivers to Reference Design and Anticipated Data Products

(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the Solar System, exploring the transient optical sky, and mapping the Milky Way. LSST will be a wide-field ground-based system sited at Cerro Pach\'{o}n in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg$^2$ field of view, and a 3.2 Gigapixel camera. The standard observing sequence will consist of pairs of 15-second exposures in a given field, with two such visits in each pointing in a given night. With these repeats, the LSST system is capable of imaging about 10,000 square degrees of sky in a single filter in three nights. The typical 5$\sigma$ point-source depth in a single visit in $r$ will be $\sim 24.5$ (AB). The project is in the construction phase and will begin regular survey operations by 2022. The survey area will be contained within 30,000 deg$^2$ with $\delta<+34.5^\circ$, and will be imaged multiple times in six bands, $ugrizy$, covering the wavelength range 320--1050 nm. About 90\% of the observing time will be devoted to a deep-wide-fast survey mode which will uniformly observe a 18,000 deg$^2$ region about 800 times (summed over all six bands) during the anticipated 10 years of operations, and yield a coadded map to $r\sim27.5$. The remaining 10\% of the observing time will be allocated to projects such as a Very Deep and Fast time domain survey. The goal is to make LSST data products, including a relational database of about 32 trillion observations of 40 billion objects, available to the public and scientists around the world.Comment: 57 pages, 32 color figures, version with high-resolution figures available from https://www.lsst.org/overvie

The Zwicky Transient Facility: Data Processing, Products, and Archive

The Zwicky Transient Facility (ZTF) is a new robotic time-domain survey currently in progress using the Palomar 48-inch Schmidt Telescope. ZTF uses a 47 square degree field with a 600 megapixel camera to scan the entire northern visible sky at rates of ~3760 square degrees/hour to median depths of g ~ 20.8 and r ~ 20.6 mag (AB, 5sigma in 30 sec). We describe the Science Data System that is housed at IPAC, Caltech. This comprises the data-processing pipelines, alert production system, data archive, and user interfaces for accessing and analyzing the products. The realtime pipeline employs a novel image-differencing algorithm, optimized for the detection of point source transient events. These events are vetted for reliability using a machine-learned classifier and combined with contextual information to generate data-rich alert packets. The packets become available for distribution typically within 13 minutes (95th percentile) of observation. Detected events are also linked to generate candidate moving-object tracks using a novel algorithm. Objects that move fast enough to streak in the individual exposures are also extracted and vetted. The reconstructed astrometric accuracy per science image with respect to Gaia is typically 45 to 85 milliarcsec. This is the RMS per axis on the sky for sources extracted with photometric S/N >= 10. The derived photometric precision (repeatability) at bright unsaturated fluxes varies between 8 and 25 millimag. Photometric calibration accuracy with respect to Pan-STARRS1 is generally better than 2%. The products support a broad range of scientific applications: fast and young supernovae, rare flux transients, variable stars, eclipsing binaries, variability from active galactic nuclei, counterparts to gravitational wave sources, a more complete census of Type Ia supernovae, and Solar System objects.Comment: 30 pages, 16 figures, Published in PASP Focus Issue on the Zwicky Transient Facility (doi: 10.1088/1538-3873/aae8ac