Data Access for LIGO on the OSG

During 2015 and 2016, the Laser Interferometer Gravitational-Wave Observatory (LIGO) conducted a three-month observing campaign. These observations delivered the first direct detection of gravitational waves from binary black hole mergers. To search for these signals, the LIGO Scientific Collaboration uses the PyCBC search pipeline. To deliver science results in a timely manner, LIGO collaborated with the Open Science Grid (OSG) to distribute the required computation across a series of dedicated, opportunistic, and allocated resources. To deliver the petabytes necessary for such a large-scale computation, our team deployed a distributed data access infrastructure based on the XRootD server suite and the CernVM File System (CVMFS). This data access strategy grew from simply accessing remote storage to a POSIX-based interface underpinned by distributed, secure caches across the OSG.Comment: 6 pages, 3 figures, submitted to PEARC1

Managing Research Data in Big Science

The project which led to this report was funded by JISC in 2010--2011 as part of its 'Managing Research Data' programme, to examine the way in which Big Science data is managed, and produce any recommendations which may be appropriate. Big science data is different: it comes in large volumes, and it is shared and exploited in ways which may differ from other disciplines. This project has explored these differences using as a case-study Gravitational Wave data generated by the LSC, and has produced recommendations intended to be useful variously to JISC, the funding council (STFC) and the LSC community. In Sect. 1 we define what we mean by 'big science', describe the overall data culture there, laying stress on how it necessarily or contingently differs from other disciplines. In Sect. 2 we discuss the benefits of a formal data-preservation strategy, and the cases for open data and for well-preserved data that follow from that. This leads to our recommendations that, in essence, funders should adopt rather light-touch prescriptions regarding data preservation planning: normal data management practice, in the areas under study, corresponds to notably good practice in most other areas, so that the only change we suggest is to make this planning more formal, which makes it more easily auditable, and more amenable to constructive criticism. In Sect. 3 we briefly discuss the LIGO data management plan, and pull together whatever information is available on the estimation of digital preservation costs. The report is informed, throughout, by the OAIS reference model for an open archive

Managing Research Data: Gravitational Waves

The project which led to this report was funded by JISC in 2010–2011 as part of its ‘Managing Research Data’ programme, to examine the way in which Big Science data is managed, and produce any recommendations which may be appropriate. Big science data is different: it comes in large volumes, and it is shared and exploited in ways which may differ from other disciplines. This project has explored these differences using as a case-study Gravitational Wave data generated by the LSC, and has produced recommendations intended to be useful variously to JISC, the funding council (STFC) and the LSC community. In Sect. 1 we define what we mean by ‘big science’, describe the overall data culture there, laying stress on how it necessarily or contingently differs from other disciplines. In Sect. 2 we discuss the benefits of a formal data-preservation strategy, and the cases for open data and for well-preserved data that follow from that. This leads to our recommendations that, in essence, funders should adopt rather light-touch prescriptions regarding data preservation planning: normal data management practice, in the areas under study, corresponds to notably good practice in most other areas, so that the only change we suggest is to make this planning more formal, which makes it more easily auditable, and more amenable to constructive criticism. In Sect. 3 we briefly discuss the LIGO data management plan, and pull together whatever information is available on the estimation of digital preservation costs. The report is informed, throughout, by the OAIS reference model for an open archive. Some of the report’s findings and conclusions were summarised in [1]. See the document history on page 37

BOSS-LDG: A Novel Computational Framework that Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery

We present a novel computational framework that connects Blue Waters, the NSF-supported, leadership-class supercomputer operated by NCSA, to the Laser Interferometer Gravitational-Wave Observatory (LIGO) Data Grid via Open Science Grid technology. To enable this computational infrastructure, we configured, for the first time, a LIGO Data Grid Tier-1 Center that can submit heterogeneous LIGO workflows using Open Science Grid facilities. In order to enable a seamless connection between the LIGO Data Grid and Blue Waters via Open Science Grid, we utilize Shifter to containerize LIGO's workflow software. This work represents the first time Open Science Grid, Shifter, and Blue Waters are unified to tackle a scientific problem and, in particular, it is the first time a framework of this nature is used in the context of large scale gravitational wave data analysis. This new framework has been used in the last several weeks of LIGO's second discovery campaign to run the most computationally demanding gravitational wave search workflows on Blue Waters, and accelerate discovery in the emergent field of gravitational wave astrophysics. We discuss the implications of this novel framework for a wider ecosystem of Higher Performance Computing users.Comment: 10 pages, 10 figures. Accepted as a Full Research Paper to the 13th IEEE International Conference on eScienc

Medium range structural order in amorphous tantala spatially resolved with changes to atomic structure by thermal annealing

Amorphous tantala (a-Ta2O5) is an important technological material that has wide ranging applications in electronics, optics and the biomedical industry. It is used as the high refractive index layers in the multi-layer dielectric mirror coatings in the latest generation of gravitational wave interferometers, as well as other precision interferometers. One of the current limitations in sensitivity of gravitational wave detectors is Brownian thermal noise that arises from the tantala mirror coatings. Measurements have shown differences in mechanical loss of the mirror coatings, which is directly related to Brownian thermal noise, in response to thermal annealing. We utilise scanning electron diffraction to perform Fluctuation Electron Microscopy (FEM) on Ion Beam Sputtered (IBS) amorphous tantala coatings, definitively showing an increase in the medium range order (MRO), as determined from the variance between the diffraction patterns in the scan, due to thermal annealing at increasing temperatures. Moreover, we employ Virtual Dark-Field Imaging (VDFi) to spatially resolve the FEM signal, enabling investigation of the persistence of the fragments responsible for the medium range order, as well as the extent of the ordering over nm length scales, and show ordered patches larger than 5 nm in the highest temperature annealed sample. These structural changes directly correlate with the observed changes in mechanical loss.Comment: 22 pages, 5 figure

Low scatter and ultra-low reflectivity measured in a fused silica window

We investigate the reflectivity and optical scattering characteristics at 1064\,nm of an antireflection coated fused silica window of the type being used in the Advanced LIGO gravitational-wave detectors. Reflectivity is measured in the ultra-low range of 5-10\,ppm (by vendor) and 14-30\,ppm (by us). Using an angle-resolved scatterometer we measure the sample's Bidirectional Scattering Distribution Function (BSDF) and use this to estimate its transmitted and reflected scatter at roughly 20-40\,ppm and 1\,ppm, respectively, over the range of angles measured. We further inspect the sample's low backscatter using an imaging scatterometer, measuring an angle resolved BSDF below $10^{-6}$ sr$^{-1}$ for large angles (10$^\circ$--80$^\circ$ from incidence in the plane of the beam). We use the associated images to (partially) isolate scatter from different regions of the sample and find that scattering from the bulk fused silica is on par with backscatter from the antireflection coated optical surfaces. To confirm that the bulk scattering is caused by Rayleigh scattering, we perform a separate experiment, measuring the scattering intensity versus input polarization angle. We estimate that 0.9--1.3\,ppm of the backscatter can be accounted for by Rayleigh scattering of the bulk fused silica. These results indicate that modern antireflection coatings have low enough scatter to not limit the total backscattering of thick fused silica optics.Comment: 9 pages, 10 figure

The Clarens Web Service Framework for Distributed Scientific Analysis in Grid Projects

Large scientific collaborations are moving towards service oriented architecutres for implementation and deployment of globally distributed systems. Clarens is a high performance, easy to deploy Web Service framework that supports the construction of such globally distributed systems. This paper discusses some of the core functionality of Clarens that the authors believe is important for building distributed systems based on Web Services that support scientific analysis