The Sloan Digitial Sky Surveys (SDSS) have been collecting imaging and spectoscopic data since 1998. These data as well as their derived data products are made publicly available through regular data releases, of which the 13th took place summer 2016. Although public data releases can be challenging to manage, they significantly increase the impact of a survey, both scientifically and educationally.Postprin

Blanton, Michael

Bolton, Adam S.

Brownstein, Joel

Raddick, M. Jordan

Thakar, Ani

Weijmans, Anne-Marie

arXiv

St Andrews Research Repository

arXiv:1612.05668v1  [astro-ph.IM]  16 Dec 2016The Challenges of a Public Data Release: behind the scenes of SDSSDR13Anne-Marie Weijmans,1 Michael Blanton,2 Adam S. Bolton,3,4 JoelBrownstein,4 M. Jordan Raddick,5 and Ani Thakar51University of St Andrews, St Andrews, Scotland, UK; amw23@st-andrews.ac.uk2New York University, New York, New York, US3NOAO, Tucson, Arizona, US4University of Utah, Salt Lake City, Utah, US5The Johns Hopkins University, Baltimore, Maryland, USAbstract. The Sloan Digitial Sky Surveys (SDSS) have been collecting imaging andspectoscopic data since 1998. These data as well as their derived data products are madepublicly available through regular data releases, of which the 13th took place summer2016. Although public data releases can be challenging to manage, they signficantlyincrease the impact of a survey, both scientifically and educationally.1. IntroductionThe Sloan Digital Sky Survey (SDSS) started observing in 1998, with the goal to im-age the Northern hemisphere in five optical wavelength bands (York et al. 2000), usingthe dedicated 2.5m Sloan Foundation Telescope (Gunn et al. 2006) at Apache PointObservatory (APO), New Mexico, US. Since then, SDSS has hosted several dedicatedsurveys, adding optical, infrared and integral-field spectroscopy to its orignal imagingsurvey (e.g. Eisenstein et al. 2011). The current SDSS-IV (Blanton et al. in prep) con-sists of three spectroscopic surveys: i) APOGEE-2 (Apache Point Observatory GalaxyEvolution Experiment), obtaining infrared spectra of stars to unravel the chemical anddynamical formation history of the Milky Way (Majewski et al. in prep), ii) MaNGA(Mapping Nearby Galaxies at APO), an integral-field spectroscopic survey to study theformation, growth and evolution of galaxies (Bundy et al. 2015), and iii) eBOSS (ex-tended Baryon Oscillation Spectroscopic Survey), measuring redshifts of ∼1.5 milliongalaxies and quasars from optical spectra to map the structure of the Universe and de-termine its expansion history (Dawson et al. 2016). In 2017, SDSS will for the first timeobserve from the Southern hemisphere at Las Campanas Observatory (LCO), Chile, aspart of APOGEE-2. The SDSS collaboration includes more than 50 member institutesspread over 4 continents, with close to 1000 scientists registered as members.SDSS has from its beginnings been dedicated to making its data publicly availablethrough data releases. The first early data release was made public in 2002 (Stoughtonet al. 2002), and in summer 2016 the Thirteenth SDSS data release (DR13) becamepublicly available (Albareti et al. 2016). These public data releases have signficantly12 Weijmans et al.widened the impact of the SDSS beyond its collaborations: more than 7000 papershave been published based on SDSS data. The 2015 National Research Council report“Optimizing the U.S. Ground-Based Optical and Infrared Astronomy System” notesthat within this US system the SDSS has a publication rate three times higher than anyother telescope. In addition, SDSS data has been incorporated in various educational(e.g., Voyages1) and public engagement activities (e.g., Galaxy Zoo; Lintott et al. 2008).In the field of data science, SDSS public data releases have been cited as the mostinfluential data source ahead of any other source in any field of science (Stalzer &Mentzel 2016). In this proceeding we discuss the mechanisms of an SDSS public datarelease, with a focus on the latest release - SDSS-IV’s DR13.2. Data and Data ProductsThe main components of a data release are the data products. The raw SDSS data istransferred daily from APO to the Center for High Performance Computing (CHPC) atthe University of Utah, where it is reduced by dedicated data reduction pipelines. Eachsurvey within SDSS-IV has a survey data team that is responsible for this part of theprocess. Once the data is reduced and the relevant data products have been vetted bythe survey data teams, they are copied to the Science Archive Server (SAS)2 where theyremain as proprietary data for the SDSS-IV collaboration, until they are incorporatedin a public data release (average time scale ∼1.5 year).The SAS offers a file-based system that allows for low-level bulk data access andoff line analysis. The SAS can also be accessed through a specialised web application,to visualise spectra. In addition, the Catalog Archive Server (CAS)3, hosted at the JohnsHopkins University, offers Web browser-based graphical and SQL query interfaces tothe catalog data that permit casual synchronous data retrieval (SkyServer; Szalay et al.2002) as well as batch-mode server-side analysis (CasJobs; Li & Thakar 2008) withadvanced capabilities like a server-side personal SQL database, data upload and datasharing via groups. There is also a command-line tool to submit CasJobs queries, andas of DR13, a jupyter notebook interface to retrieve catalog data from SkyServer andCasJobs within a docker container. Both the SAS and the CAS offer public data access,with the SAS hosting 267 TB of data products, as of DR13. All data releases arecumulative, in that they also contain data products from previous data releases. Byaccessing the latest SDSS data release, users will therefore always have access to allavailable SDSS public data.3. DocumentationMaking data publicly available is a necessary but not sufficient condition for generatingimpact: in addition, the data has to be accessible. Users need to know where to findthe data, and how to work with the data. Documentation is essential, and each SDSS1http://voyages.sdss.org2http://data.sdss.org/sas3http://skyserver.sdss.orgThe Challenges of a Public Data Release 3data product therefore has a detailed data model. Each data release is also described indetail in an accompaning data release paper (for DR13, see Albareti et al. 2016).However, a public data release serves many different audiences. Although for as-tronomers familiar with the SDSS data format the data model may be sufficient, novelusers and students require more documentation. The SDSS website4 therefore offersa portal with more information on how to access the data (including links to the SAS,CAS and web application), as well as background information, links to technical pa-pers, examples and tutorials. As teachers and instructors also use SDSS data in theirclassrooms, the website also offers links to education and public engagement activities.Finally, users can e-mail the SDSS helpdesk if they have any questions or encounterany problems when working with SDSS data.4. Data Release ManagementThe SDSS-IV data team is led by the Science Archive Scientist, the Catalog ArchiveScientist and the Data Release Coordinator. The latter is repsonsible for managing thedata release and keeping it on schedule. Planning for a public data release typicallystarts one year in advance. First the data products are generated by the survey datateams, and vetted for quality control by the survey science teams. Once the data prod-ucts are finalised, catalogs are generated and loaded into the CAS.Most of the documenation is written during a one-week documentation workshop(”DocuFeest”), ∼four months before the data release. Several members of the datateam, the survey teams and the education and public engagement team meet to generatemost of the website content, as well as a comprehensive draft of the data release paper.Descriptions and tutorials for new data products have to be added into the documenationhierarchy: for DR13 this meant a redesign of the website to include the new MaNGAsurvey. The website and paper are finalised in the weeks after the DocuFeest. Meetingin person has proved crucial for efficient writing of documentation, as both the websiteand paper require input from all teams across SDSS-IV, and communication is key.As the date for the data release approaches, a plan is drafted for the announcementof the data release. SDSS-IV has a strong social media presence, with a general andsurvey-specific Twitter accounts, a Facebook page, and a blog aimed at the generalpublic. For DR13, we issued a Facebook announcement and tweeted in almost alllanguages spoken within the collaboration (thanks to a team-wide effort to translate thetweets). A description of the content of DR13 was given in a multilingual blog post.Announcements were also issued over mailing lists aimed at professional astronomers.5. ConclusionThe impact of an astronomical survey is set by the reach of its data distribution system.If the data does not reach the astronomers for their research projects to make new dis-coveries, the teachers to teach their students how to work with astronomical data, andthe general public to increase their awareness of astronomy and science, then impactwill be limited to a small core survey team. A successful public data release, especially4http://www.sdss.org/4 Weijmans et al.one aimed at a variety of end users, therefore needs to ensure that the data is not onlyfreely available, but also clearly documented and in accessible formats.Managing a public data release is challenging: it involves keeping track of a myr-iad of different data products and their documenation, which includes data models,descriptions and tutorials, all aimed at different user audiences. Quality control of boththe data products and the documentation is crucial. Good communication between dif-ferent teams within the collaboration is key, given that the data release requires inputfrom everyone. But despite the challenges and work involved, public data releases areworthwhile: they increase the impact of the survey, allow for more science output be-yond the collaboration, and provide accessible data products not only for astronomicalresearch, but also for education and public engagement.Acknowledgments. AW acknowledges support of a Leverhulme Trust Early Ca-reer Fellowship. The authors would like to thank the Center for Data Science at NewYork University for their hospitality during DocuFeest 2016. Funding for the SloanDigital Sky Survey IV has been provided by the Alfred P. Sloan Foundation, the U.S.Department of Energy Office of Science, and the Participating Institutions. SDSS-IVacknowledges support and resources from the Center for High-Performance Comput-ing at the University of Utah. The SDSS web site is www.sdss.org. SDSS-IV is man-aged by the Astrophysical Research Consortium for the Participating Institutions of theSDSS Collaboration including the Brazilian Participation Group, the Carnegie Insti-tution for Science, Carnegie Mellon University, the Chilean Participation Group, theFrench Participation Group, Harvard-Smithsonian Center for Astrophysics, Instituto deAstrofísica de Canarias, The Johns Hopkins University, Kavli Institute for the Physicsand Mathematics of the Universe (IPMU) / University of Tokyo, Lawrence BerkeleyNational Laboratory, Leibniz Institut für Astrophysik Potsdam (AIP), Max-Planck-Institut für Astronomie (MPIA Heidelberg), Max-Planck-Institut für Astrophysik (MPAGarching), Max-Planck-Institut für Extraterrestrische Physik (MPE), National Astro-nomical Observatory of China, New Mexico State University, New York University,University of Notre Dame, Observatário Nacional / MCTI, The Ohio State University,Pennsylvania State University, Shanghai Astronomical Observatory, United KingdomParticipation Group, Universidad Nacional Autónoma de México, University of Ari-zona, University of Colorado Boulder, University of Oxford, University of Portsmouth,University of Utah, University of Virginia, University of Washington, University ofWisconsin, Vanderbilt University, and Yale University.ReferencesAlbareti, F. D., et al. 2016, ArXiv e-prints. 1608.02013Bundy, K., et al. 2015, ApJ, 798, 7Dawson, K. S., et al. 2016, AJ, 151, 44Eisenstein, D. J., et al. 2011, AJ, 142, 72Gunn, J. E., et al. 2006, AJ, 131, 2332Li, N., & Thakar, A. R. 2008, Computing in Science and Engineering, 10, 18Lintott, C. J., et al. 2008, MNRAS, 389, 1179Stalzer, M., & Mentzel, C. 2016, SpringerPlus, 5, 1266Stoughton, C., et al. 2002, AJ, 123, 485Szalay, A. S., et al. 2002, in Proceedings of the 2002 ACM SIGMOD International Conferenceon Management of Data (New York, NY, USA: ACM), SIGMOD ’02, 570York, D. G., et al. 2000, AJ, 120, 1579

The challenges of a public data release : behind the scenes of SDSS DR13

Abstract

Similar works

Full text

Available Versions

St Andrews Research Repository