Data reduction pipelines for the Keck Observatory Archive

The Keck Observatory Archive (KOA) currently serves ~ 42 TB of data spanning over 20 years from all ten past and current facility instruments at Keck. Although most of the available data are in the raw form, for four instruments (HIRES, NIRC2, OSIRIS, LWS), quick-look, browse products generated by automated pipelines are also offered to facilitate assessment of the scientific content and quality of the data. KOA underwrote the update of the MAKEE package to support reduction of the CCD upgrade to HIRES, developed scripts for reduction of NIRC2 data and automated the existing OSIRIS and LWS data reduction packages. We describe in some detail the recently completed automated pipeline for NIRSPEC, which will be used to create browse products in KOA and made available for quicklook of the data by the observers at the telescope. We review the currently available data reduction tools for Keck data, and present our plans and anticipated priorities for the development of automated pipelines and release of reduced data products for the rest of the current and future instruments. We also anticipate that Keck's newest instrument, NIRES, which will be delivered with a fully automated pipeline, will be the first to have both raw and level-1 data ingested at commissioning

The Design of the W. M. Keck Observatory Archive

The Michelson Science Center (MSC) and the W. M. Keck Observatory are building an archive that will serve data obtained at the Keck Observatory. The archive has begun operations and is ingesting Level 0 (uncalibrated) observations made with the recently upgraded High Resolution Echelle Spectrometer (HIRES); these observations will be publicly accessible after expiration of a proprietary period. Observatory staff have begun using the archived data to determine the long-term performance of the HIRES instrument. The archive is housed at the Michelson Science Center (MSC) and employs a modular design with the following components: (1) Data Evaluation and Preparation: images from the telescope are evaluated and native FITS headers are converted to metadata that will support archiving; (2) Trans Pacific Data Transfer: metadata are sent daily by e-mail and ingested into the archive in a highly fault tolerant fashion, and FITS images are written to DVDs and sent to MSC each week; (3) Science Information System: inherited from the NASA/IPAC Infrared Science Archive, it provides all the functionality needed to support database inquiries and processing of requests; and a Web-based (4) User Interface, a thin layer above the information system that accepts user requests and returns results. The design offers two major cost-saving benefits: it overcomes the geographical separation between the telescope and the archive and enables development at Keck and at MSC to proceed independently; and it permits direct inheritance of the IRSA architecture

Unconventional PDV applications: detecting plasma and radiation

Author Institution: Sandia National LaboratoriesSlides presented at the 2018 Photonic Doppler Velocimetry (PDV) Users Workshop, Drury Plaza Hotel, Santa Fe, New Mexico, May 16-18, 2018

Data and Metadata Management at the Keck Observatory Archive

A collaboration between the W. M. Keck Observatory (WMKO) in Hawaii and the NASA Exoplanet Science Institute (NExScI) in California, the Keck Observatory Archive (KOA) was commissioned in 2004 to archive data from WMKO, which operates two classically scheduled 10 m ground-based telescopes. The data from Keck are not suitable for direct ingestion into the archive since the metadata contained in the original FITS headers lack the information necessary for proper archiving. The data pose a number of challenges for KOA: different instrument builders used different standards, and the nature of classical observing, where observers have complete control of the instruments and their observations, lead to heterogeneous data sets. For example, it is often difficult to determine if an observation is a science target, a sky frame, or a sky flat. It is also necessary to assign the data to the correct owners and observing programs, which can be a challenge for time-domain and target-of-opportunity observations, or on split nights, during which two or more principle investigators share a given night. In addition, having uniform and adequate calibrations is important for the proper reduction of data. Therefore, KOA needs to distinguish science files from calibration files, identify the type of calibrations available, and associate the appropriate calibration files with each science frame. We describe the methodologies and tools that we have developed to successfully address these difficulties, adding content to the FITS headers and “retrofitting" the metadata in order to support archiving Keck data, especially those obtained before the archive was designed. With the expertise gained from having successfully archived observations taken with all eight currently active instruments at WMKO, we have developed lessons learned from handling this complex array of heterogeneous metadata. These lessons help ensure a smooth ingestion of data not only for current but also future instruments, as well as a better experience for the archive user

The Design and Development of the NIRSPEC Data Reduction Pipeline for the Keck Observatory Archive

The Keck Observatory Archive, a collaboration between the NASA Exoplanet Science Institute and the W. M. Keck Observatory, serves science and calibration data for all current and retired instruments from the twin Keck Telescopes. In addition to the raw data, we publicly serve quick-look, reduced data products for four instruments (HIRES, LWS, NIRC2 and OSIRIS), so that KOA users can easily assess the quality and scientific content of the data. In this paper we present the design and implementation of the NIRSPEC data reduction pipeline (DRP) for KOA. We will discuss the publicly available reduction packages for NIRSPEC, the challenges encountered when designing this fully automated DRP and the algorithm used to determine wavelength calibration from sky lines. The reduced data products from the NIRSPEC DRP are expected to be available in KOA by mid-2016

An All-Sky 2MASS Mosaic Constructed on the TeraGrid

The Montage mosaic engine supplies on-request image mosaic services for the NVO astronomical community. A companion paper describes scientific applications of Montage. This paper describes one application in detail: the generation at SDSC of a mosaic of the 2MASS All-sky Image Atlas on the NSF TeraGrid. The goals of the project are: to provide a value-added 2MASS product that combines overlapping images to improve sensitivity; to demonstrate applicability of computing at-scale to astronomical missions and surveys, especially projects such as LSST; and to demonstrate the utility of the NVO Hyperatlas format. The numerical processing of an 8 TB, 32-bit survey to produce a 64-bit, 20 TB output atlas presented multiple scalability and operational challenges. An MPI Python module, MYMPI, was used to manage the alternately sequential and parallel steps of the Montage process. This allowed us to parallelize all steps of the mosaic process: that of many, sequential steps executing simultaneously for independent mosaics and that of a single MPI parallel job executing on many CPUs for a single mosaic. The Storage Resource Broker (SRB) was used to archive the output results in the Hyperatlas. The 2MASS mosaics are now being assessed for scientific quality. Around 130,000 CPU-hours were used to complete the mosaics. The output consists of 1734 plates spanning 6◦ for each of 3 bands. Each of the 5202 mosaics is roughly 4 GB in size, and each has been tiled into a 12×12 array of 26 MB files for ease of handling. The total size is about 20 TB in 750,000 tiles

Data reduction pipelines for the Keck Observatory Archive

The Keck Observatory Archive (KOA) currently serves ~ 42 TB of data spanning over 20 years from all ten past and current facility instruments at Keck. Although most of the available data are in the raw form, for four instruments (HIRES, NIRC2, OSIRIS, LWS), quick-look, browse products generated by automated pipelines are also offered to facilitate assessment of the scientific content and quality of the data. KOA underwrote the update of the MAKEE package to support reduction of the CCD upgrade to HIRES, developed scripts for reduction of NIRC2 data and automated the existing OSIRIS and LWS data reduction packages. We describe in some detail the recently completed automated pipeline for NIRSPEC, which will be used to create browse products in KOA and made available for quicklook of the data by the observers at the telescope. We review the currently available data reduction tools for Keck data, and present our plans and anticipated priorities for the development of automated pipelines and release of reduced data products for the rest of the current and future instruments. We also anticipate that Keck's newest instrument, NIRES, which will be delivered with a fully automated pipeline, will be the first to have both raw and level-1 data ingested at commissioning

The Design and Development of the NIRSPEC Data Reduction Pipeline for the Keck Observatory Archive

The Keck Observatory Archive, a collaboration between the NASA Exoplanet Science Institute and the W. M. Keck Observatory, serves science and calibration data for all current and retired instruments from the twin Keck Telescopes. In addition to the raw data, we publicly serve quick-look, reduced data products for four instruments (HIRES, LWS, NIRC2 and OSIRIS), so that KOA users can easily assess the quality and scientific content of the data. In this paper we present the design and implementation of the NIRSPEC data reduction pipeline (DRP) for KOA. We will discuss the publicly available reduction packages for NIRSPEC, the challenges encountered when designing this fully automated DRP and the algorithm used to determine wavelength calibration from sky lines. The reduced data products from the NIRSPEC DRP are expected to be available in KOA by mid-2016

Design and Implementation of Data Reduction Pipelines for the Keck Observatory Archive

The Keck Observatory Archive (KOA), a collaboration between the NASA Exoplanet Science Institute and the W. M. Keck Observatory, serves science and calibration data for all active and inactive instruments from the twin Keck Telescopes located near the summit of Mauna Kea, Hawaii. In addition to the raw data, we produce and provide quick look reduced data for four instruments (HIRES, LWS, NIRC2, and OSIRIS) so that KOA users can more easily assess the scientific content and the quality of the data, which can often be difficult with raw data. The reduced products derive from both publicly available data reduction packages (when available) and KOA-created reduction scripts. The automation of publicly available data reduction packages has the benefit of providing a good quality product without the additional time and expense of creating a new reduction package, and is easily applied to bulk processing needs. The downside is that the pipeline is not always able to create an ideal product, particularly for spectra, because the processing options for one type of target (eg., point sources) may not be appropriate for other types of targets (eg., extended galaxies and nebulae). In this poster we present the design and implementation for the current pipelines used at KOA and discuss our strategies for handling data for which the nature of the targets and the observers' scientific goals and data taking procedures are unknown. We also discuss our plans for implementing automated pipelines for the remaining six instruments