Data Mining the SDSS SkyServer Database

An earlier paper (Szalay et. al. "Designing and Mining MultiTerabyte Astronomy Archives: The Sloan Digital Sky Survey," ACM SIGMOD 2000) described the Sloan Digital Sky Survey's (SDSS) data management needs by defining twenty database queries and twelve data visualization tasks that a good data management system should support. We built a database and interfaces to support both the query load and also a website for ad-hoc access. This paper reports on the database design, describes the data loading pipeline, and reports on the query implementation and performance. The queries typically translated to a single SQL statement. Most queries run in less than 20 seconds, allowing scientists to interactively explore the database. This paper is an in-depth tour of those queries. Readers should first have studied the companion overview paper Szalay et. al. "The SDSS SkyServer, Public Access to the Sloan Digital Sky Server Data" ACM SIGMOND 2002.Comment: 40 pages, Original source is at http://research.microsoft.com/~gray/Papers/MSR_TR_O2_01_20_queries.do

The Sloan Digital Sky Survey Science Archive: Migrating a Multi-Terabyte Astronomical Archive from Object to Relational DBMS

The Sloan Digital Sky Survey Science Archive is the first in a series of multi-Terabyte digital archives in Astronomy and other data-intensive sciences. To facilitate data mining in the SDSS archive, we adapted a commercial database engine and built specialized tools on top of it. Originally we chose an object-oriented database management system due to its data organization capabilities, platform independence, query performance and conceptual fit to the data. However, after using the object database for the first couple of years of the project, it soon began to fall short in terms of its query support and data mining performance. This was as much due to the inability of the database vendor to respond our demands for features and bug fixes as it was due to their failure to keep up with the rapid improvements in hardware performance, particularly faster RAID disk systems. In the end, we were forced to abandon the object database and migrate our data to a relational database. We describe below the technical issues that we faced with the object database and how and why we migrated to relational technology

GridCertLib: a Single Sign-on Solution for Grid Web Applications and Portals

This paper describes the design and implementation of GridCertLib, a Java library leveraging a Shibboleth-based authentication infrastructure and the SLCS online certificate signing service, to provide short-lived X.509 certificates and Grid proxies. The main use case envisioned for GridCertLib, is to provide seamless and secure access to Grid/X.509 certificates and proxies in web applications and portals: when a user logs in to the portal using Shibboleth authentication, GridCertLib can automatically obtain a Grid/X.509 certificate from the SLCS service and generate a VOMS proxy from it. We give an overview of the architecture of GridCertLib and briefly describe its programming model. Its application to some deployment scenarios is outlined, as well as a report on practical experience integrating GridCertLib into portals for Bioinformatics and Computational Chemistry applications, based on the popular P-GRADE and Django softwares.Comment: 18 pages, 1 figure; final manuscript accepted for publication by the "Journal of Grid Computing

The SDSS SkyServer, Public Access to the Sloan Digital Sky Server Data

The SkyServer provides Internet access to the public Sloan Digital Sky Survey (SDSS) data for both astronomers and for science education. This paper describes the SkyServer goals and architecture. It also describes our experience operating the SkyServer on the Internet. The SDSS data is public and well-documented so it makes a good test platform for research on database algorithms and performance.Comment: submitted for publication, original at http://research.microsoft.com/scripts/pubs/view.asp?TR_ID=MSR-TR-2001-10

High-Redshift Quasars Found in Sloan Digital Sky Survey Commissioning Data IV: Luminosity Function from the Fall Equatorial Stripe Sampl

This is the fourth paper in a series aimed at finding high-redshift quasars from five-color imaging data taken along the Celestial Equator by the SDSS. during its commissioning phase. In this paper, we use the color-selected sample of 39 luminous high-redshift quasars presented in Paper III to derive the evolution of the quasar luminosity function over the range of 3.6<z<5.0, and -27.5<M_1450<-25.5 (Omega=1, H_0=50 km s^-1 Mpc^-1). We use the selection function derived in Paper III to correct for sample incompleteness. The luminosity function is estimated using three different methods: (1) the 1/V_a estimator; (2) a maximum likelihood solution, assuming that the density of quasars depends exponentially on redshift and as a power law in luminosity and (3) Lynden-Bell's non-parametric C^- estimator. All three methods give consistent results. The luminous quasar density decreases by a factor of ~ 6 from z=3.5 to z=5.0, consistent with the decline seen from several previous optical surveys at z<4.5. The luminosity function follows psi(L) ~ L^{-2.5} for z~4 at the bright end, significantly flatter than the bright end luminosity function psi(L) \propto L^{-3.5} found in previous studies for z<3, suggesting that the shape of the quasar luminosity function evolves with redshift as well, and that the quasar evolution from z=2 to 5 cannot be described as pure luminosity evolution. Possible selection biases and the effect of dust extinction on the redshift evolution of the quasar density are also discussed.Comment: AJ accepted, with minor change

Photometric Redshifts of Quasars

We demonstrate that the design of the Sloan Digital Sky Survey (SDSS) filter system and the quality of the SDSS imaging data are sufficient for determining accurate and precise photometric redshifts (``photo-z''s) of quasars. Using a sample of 2625 quasars, we show that photo-z determination is even possible for z<=2.2 despite the lack of a strong continuum break that robust photo-z techniques normally require. We find that, using our empirical method on our sample of objects known to be quasars, approximately 70% of the photometric redshifts are correct to within delta z = 0.2; the fraction of correct photometric redshifts is even better for z>3. The accuracy of quasar photometric redshifts does not appear to be dependent upon magnitude to nearly 21st magnitude in i'. Careful calibration of the color-redshift relation to 21st magnitude may allow for the discovery of on the order of 10^6 quasars candidates in addition to the 10^5 quasars that the SDSS will confirm spectroscopically. We discuss the efficient selection of quasar candidates from imaging data for use with the photometric redshift technique and the potential scientific uses of a large sample of quasar candidates with photometric redshifts.Comment: 29 pages, 8 figures, submitted to A

KL Estimation of the Power Spectrum Parameters from the Angular Distribution of Galaxies in Early SDSS Data

We present measurements of parameters of the 3-dimensional power spectrum of galaxy clustering from 222 square degrees of early imaging data in the Sloan Digital Sky Survey. The projected galaxy distribution on the sky is expanded over a set of Karhunen-Loeve eigenfunctions, which optimize the signal-to-noise ratio in our analysis. A maximum likelihood analysis is used to estimate parameters that set the shape and amplitude of the 3-dimensional power spectrum. Our best estimates are Gamma=0.188 +/- 0.04 and sigma_8L = 0.915 +/- 0.06 (statistical errors only), for a flat Universe with a cosmological constant. We demonstrate that our measurements contain signal from scales at or beyond the peak of the 3D power spectrum. We discuss how the results scale with systematic uncertainties, like the radial selection function. We find that the central values satisfy the analytically estimated scaling relation. We have also explored the effects of evolutionary corrections, various truncations of the KL basis, seeing, sample size and limiting magnitude. We find that the impact of most of these uncertainties stay within the 2-sigma uncertainties of our fiducial result.Comment: Fig 1 postscript problem correcte

High-Redshift Quasars Found in Sloan Digital Sky Survey Commissioning Data VI. Sloan Digital Sky Survey Spectrograph Observations

We present results on over 100 high-redshift quasars found in the Sloan Digital Sky Survey (SDSS), using automated selection algorithms applied to SDSS imaging data and with spectroscopic confirmation obtained during routine spectroscopic operations of the Sloan 2.5-m telescope. The SDSS spectra cover the wavelength range 3900--9200 Angstroms at a spectral resolution of 1800, and have been obtained for 116 quasars with redshifts greater than 3.94; 92 of these objects were previously uncataloged, significantly increasing the current tally of published z>4 quasars. The paper also reports observations of five additional new z>4.6 quasars; all were found from the SDSS imaging survey and spectroscopically confirmed with data from the Apache Point Observatory's 3.5-m telescope. The i' magnitudes of the quasars range from 18.03 to 20.56. Of the 97 new objects in this paper, 13 are Broad Absorption Line quasars. Five quasars, including one object at a redshift of 5.11, have 20 cm peak flux densities greater than 1 mJy. Two of the quasars, both at z=4.5, have very weak emission lines; one of these objects is a radio source. Nineteen of the newly-discovered objects have redshifts above 4.6, and the maximum redshift is z=5.41; among objects reported to date, the latter is the third highest redshift AGN, and penultimate in redshift among luminous quasars.Comment: 34 pages, 7 figures, submitted to A

PYTHIA 6.4 Physics and Manual

The PYTHIA program can be used to generate high-energy-physics `events', i.e. sets of outgoing particles produced in the interactions between two incoming particles. The objective is to provide as accurate as possible a representation of event properties in a wide range of reactions, within and beyond the Standard Model, with emphasis on those where strong interactions play a role, directly or indirectly, and therefore multihadronic final states are produced. The physics is then not understood well enough to give an exact description; instead the program has to be based on a combination of analytical results and various QCD-based models. This physics input is summarized here, for areas such as hard subprocesses, initial- and final-state parton showers, underlying events and beam remnants, fragmentation and decays, and much more. Furthermore, extensive information is provided on all program elements: subroutines and functions, switches and parameters, and particle and process data. This should allow the user to tailor the generation task to the topics of interest.Comment: 576 pages, no figures, uses JHEP3.cls. The code and further information may be found on the PYTHIA web page: http://www.thep.lu.se/~torbjorn/Pythia.html Changes in version 2: Mistakenly deleted section heading for "Physics Processes" reinserted, affecting section numbering. Minor updates to take into account referee comments and new colour reconnection option