Data Mining the SDSS SkyServer Database

An earlier paper (Szalay et. al. "Designing and Mining MultiTerabyte Astronomy Archives: The Sloan Digital Sky Survey," ACM SIGMOD 2000) described the Sloan Digital Sky Survey's (SDSS) data management needs by defining twenty database queries and twelve data visualization tasks that a good data management system should support. We built a database and interfaces to support both the query load and also a website for ad-hoc access. This paper reports on the database design, describes the data loading pipeline, and reports on the query implementation and performance. The queries typically translated to a single SQL statement. Most queries run in less than 20 seconds, allowing scientists to interactively explore the database. This paper is an in-depth tour of those queries. Readers should first have studied the companion overview paper Szalay et. al. "The SDSS SkyServer, Public Access to the Sloan Digital Sky Server Data" ACM SIGMOND 2002.Comment: 40 pages, Original source is at http://research.microsoft.com/~gray/Papers/MSR_TR_O2_01_20_queries.do

The SDSS SkyServer, Public Access to the Sloan Digital Sky Server Data

The SkyServer provides Internet access to the public Sloan Digital Sky Survey (SDSS) data for both astronomers and for science education. This paper describes the SkyServer goals and architecture. It also describes our experience operating the SkyServer on the Internet. The SDSS data is public and well-documented so it makes a good test platform for research on database algorithms and performance.Comment: submitted for publication, original at http://research.microsoft.com/scripts/pubs/view.asp?TR_ID=MSR-TR-2001-10

AKARI-CAS --- Online Service for AKARI All-Sky Catalogues

The AKARI All-Sky Catalogues are an important infrared astronomical database for next-generation astronomy that take over the IRAS catalog. We have developed an online service, AKARI Catalogue Archive Server (AKARI-CAS), for astronomers. The service includes useful and attractive search tools and visual tools. One of the new features of AKARI-CAS is cached SIMBAD/NED entries, which can match AKARI catalogs with other catalogs stored in SIMBAD or NED. To allow advanced queries to the databases, direct input of SQL is also supported. In those queries, fast dynamic cross-identification between registered catalogs is a remarkable feature. In addition, multiwavelength quick-look images are displayed in the visualization tools, which will increase the value of the service. In the construction of our service, we considered a wide variety of astronomers' requirements. As a result of our discussion, we concluded that supporting users' SQL submissions is the best solution for the requirements. Therefore, we implemented an RDBMS layer so that it covered important facilities including the whole processing of tables. We found that PostgreSQL is the best open-source RDBMS products for such purpose, and we wrote codes for both simple and advanced searches into the SQL stored functions. To implement such stored functions for fast radial search and cross-identification with minimum cost, we applied a simple technique that is not based on dividing celestial sphere such as HTM or HEALPix. In contrast, the Web application layer became compact, and was written in simple procedural PHP codes. In total, our system realizes cost-effective maintenance and enhancements.Comment: Yamauchi, C. et al. 2011, PASP..123..852

The Sloan Digital Sky Survey Science Archive: Migrating a Multi-Terabyte Astronomical Archive from Object to Relational DBMS

The Sloan Digital Sky Survey Science Archive is the first in a series of multi-Terabyte digital archives in Astronomy and other data-intensive sciences. To facilitate data mining in the SDSS archive, we adapted a commercial database engine and built specialized tools on top of it. Originally we chose an object-oriented database management system due to its data organization capabilities, platform independence, query performance and conceptual fit to the data. However, after using the object database for the first couple of years of the project, it soon began to fall short in terms of its query support and data mining performance. This was as much due to the inability of the database vendor to respond our demands for features and bug fixes as it was due to their failure to keep up with the rapid improvements in hardware performance, particularly faster RAID disk systems. In the end, we were forced to abandon the object database and migrate our data to a relational database. We describe below the technical issues that we faced with the object database and how and why we migrated to relational technology

SkyDOT (Sky Database for Objects in the Time Domain): A Virtual Observatory for Variability Studies at LANL

The mining of Virtual Observatories (VOs) is becoming a powerful new method for discovery in astronomy. Here we report on the development of SkyDOT (Sky Database for Objects in the Time domain), a new Virtual Observatory, which is dedicated to the study of sky variability. The site will confederate a number of massive variability surveys and enable exploration of the time domain in astronomy. We discuss the architecture of the database and the functionality of the user interface. An important aspect of SkyDOT is that it is continuously updated in near real time so that users can access new observations in a timely manner. The site will also utilize high level machine learning tools that will allow sophisticated mining of the archive. Another key feature is the real time data stream provided by RAPTOR (RAPid Telescopes for Optical Response), a new sky monitoring experiment under construction at Los Alamos National Laboratory (LANL).Comment: to appear in SPIE proceedings vol. 4846, 11 pages, 5 figure