The Dark Energy Survey Data Management System

The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at NCSA and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.Comment: To be published in the proceedings of the SPIE conference on Astronomical Instrumentation (held in Marseille in June 2008). This preprint is made available with the permission of SPIE. Further information together with preprint containing full quality images is available at http://desweb.cosmology.uiuc.edu/wik

Collaboration Enabling Internet Resource Collection-Building Software and Technologies

Over the last decade the Library of the University of California, Riverside and its collaborators have developed a number of systems, service designs, and projects that utilize innovative technologies to foster better Internet finding tools in libraries and more cooperative and efficient effort in Internet link and metadata collection building. The open-source software and projects discussed represent appropriate technologies and sustainable strategies that we believe will help Internet portals, digital libraries, virtual libraries, library catalogs-with-portal-like-capabilities (IPDVLCs), and related collection-building efforts in academia to better scale and more accurately anticipate and meet the needs of scholarly and educational users.published or submitted for publicatio

Southeastern Librarian 71(3) Fall 2023 (Full Issue)

Complete issue of The Southeastern Librarian Volume 71 Number

Web-Scale Discovery and Federated Search

In stark contrast to the library card catalogs of old, today’s library search interfaces offer much more than one-dimensional, item-specific searching. Users are now engaged in a process of discovery in which they are empowered to control not only the sources of content being searched, but also the context into which information is delivered, and the platform onto which information is synthesized. By eliminating the barriers to information discovery, law libraries can position themselves as true partners in this process, defining their mission in new ways, and providing critical services in an ever-complex information ecosystem

Volume 35, Number 3, September 2015 OLAC Newsletter

Digitized September 2015 issue of the OLAC Newsletter

The rare books catalog and the scholarly database

The article is a researcher's eye view of the value of the library catalog not only as a database to be searched for surrogates of objects of study, but as a corpus of text that can be analyzed in its own right, or incorporated within the researcher's own research database. Barriers are identified in the ways in which catalog data can be output and the technical skills researchers currently need to download, ingest, and manipulate data. Research tools and datasets created by, or in collaboration with, the library community are identified