Preservation is important for documenting original observations, and existing data are an important resource which can be re-used. Observatories should set up electronic data archives and formulate archiving policies. VO (Virtual Observatory) compliance is desirable; even if this is not possible, at least some VO ideas should be applied. Data archives should be visible and their data kept on-line. Metadata should be plentiful, and as standard as possible, just like file formats. Literature and data should be cross-linked. Libraries can play an important role in this process. In this paper, we discuss data archiving for small projects and observatories. We review the questions of digitization, cost factors, manpower, organizational structure and more

Holl, András

English

Repository of the Academy's Library

LIBRARY AND INFORMATION SERVICES IN ASTRONOMY VIASP Conference Series, Vol. 433E. Isaksson, J. Lagerstrom, A. Holl, and N. Bawdekar, eds.c©2010 Astronomical Society of the PacificSmall Data Archives and LibrariesA. HollKonkoly Observatory, H-1525 P.O. Box 67, Budapest, HungaryAbstract. Preservation is important for documenting original observations, and ex-isting data are an important resource which can be re-used. Observatories should setup electronic data archives and formulate archiving policies. VO (Virtual Observatory)compliance is desirable; even if this is not possible, at least some VO ideas should beapplied. Data archives should be visible and their data kept on-line. Metadata should beplentiful, and as standard as possible, just like file formats. Literature and data shouldbe cross-linked. Libraries can play an important role in this process. In this paper, wediscuss data archiving for small projects and observatories. We review the questions ofdigitization, cost factors, manpower, organizational structure and more.This is a paper about data and archiving, about an electronic archive which is muchthought about but not yet built. I begin with a story.On the night of February 2, 1826, after a month of unusually favorable weather— meaning tiring observational work — Paulus Tittel (I use the name of the directorof Gelle´rthegy Observatory, a Catholic priest, in Latin form) fell asleep at his desk.His candle burned down, and the papers burst into flames. He woke up and managedto extinguish the fire, but several books and stacks of paper were reduced to cindersand ash. Among them was a fully written paper on the fourth comet of 1825, to besent to the Astronomische Nachrichten in Altona, together with all of the observationalmaterial recorded on this object. As we can learn from the description of the event byhis pupil at that time, Ferenc Albert, they had no time to make a copy of the originalobservations, so their work was lost, with no chance for reproduction (Vargha & Kanyo´1998).As a marginal note, a large part of the old book collection of Konkoly Observatoryoriginates from Tittel’s collection at the Gelle´rthegy Observatory. Most of the bookshave survived the conflagration, the siege of Buda in 1849 — thanks to Albert — andmany perils since then.Why do we need to archive observations? Naturally, we want to use them in pub-lications. Those publications need to be verifiable. The editors and peer-reviewers, aswell as the readers, must be able to go back to the original observations to check them.The eminent science journalNature requires that supporting data be made available. Wemust recognize that, although most scientific papers lose their importance after sometime, there is no time limit for this requirement — data should be kept indeterminately.Published data are by no means exhausted data. Observations could have secondary,serendipitous use. Or, to put it simply, information overlooked could later lead to adiscovery, as in the case of a supernova patrol plate taken in 1968 at the Piszke´steto˝Station of Konkoly Observatory in which a supernova was found. Later, a new one wasrevealed by a different astronomer.201202 HollObserving logbooks, glass plates — these media might last for centuries if thepaper is acid free, if the development of the plate was correct, if we manage to keepthem safe from fire, damp, mishandling, dust etc. But the greater perils lie elsewhere:we can not maintain the equipment and expertise necessary for measuring the plates.Moreover, non-digital data can be difficult to access.Digitization could be the answer. But are digital data preservable, are they keptpermanently accessible? The dismal fact is that digital data are often less accessibleand preservable than the old non-digital material.Digital data might get lost because they are physically damaged. Ink might fadeduring the centuries, but the information storing capability of magnetic and even opticalmedia is much shorter. The lifetime of a technological device could be even shorter; wemight have the data intact on a tape cartridge only to find ourselves without a workingtape drive. The third factor is data format: the data might be readable bit by bit, butwithout the proper software, still unintelligible to us.My experience shows that often the reasons for unavailability or loss of digitaldata lie not in the technology, but in ourselves. The data is there, on a hard drive ina drawer, on a DVD mislaid somewhere. Observatories might have well-functioninglibraries and plate vaults, but at the same time they lack a proper electronic archive.Data might even be on the web — but in peril. Data might still be lost, or might lingeron the web unmaintained. Often digital data lack unique identifiers.Large institutions and large projects have excellent archive facilities, while thesituation is often more problematic at small observatories. What could be done at suchplaces? Here is a to-do list I have compiled for establishing the electronic archive ofKonkoly Observatory.Electronic Archive as Part of the Organizational StructureThe electronic archive should be set up as an integral part of the observatory’s organi-zational structure. A person in charge should be appointed and staff allocated. In thesetimes, it is hard to create new jobs, so the head of the IT department or library could beentrusted with this task, and existing IT/library staff could be used for the operation —electronic archive responsibilities should become part of their job descriptions.Budgetary needs should be estimated and the observatory budget readjusted ac-cordingly. It does not necessarily mean a budget increase, we should note. For digiti-zation, existing photography and/or microfilm units could be converted. Archive statusshould be reported in the annual report.RegulationThe Director should regulate what should be archived, the rights of the original ob-server, and the length of the proprietary period. The ingest and operation should beregulated — librarians and IT specialists should create the rules of operation.HardwareThe physical archive should be part of the IT infrastructure. Nowadays spinning disksprovide the best media for storage as they are the cheapest and easiest to access andSmall Data Archives and Libraries 203replicate. (A comparison of the DVD and 1 TB HDD prices on a vendor website —at the time of the preparation of this article — shows that HDDs are half as expensivefor unit storage capacity. Linear tape cartridges are cheaper.) My suggestion is to usean expandable storage appliance. Migration of the whole data content of the electronicarchive should be possible with a simple cp command. Storage costs are manageableuntil the disk drive capacities for a given price grow exponentially.What to ArchiveIn general, the research process could be viewed from the data perspective as depictedin Fig. 1. PUBLICATION REPRESENTATION DATA OBSERVED DATA WORLD Figure 1. There are certain phases of research when archiving is easier. The boxat the bottom represents the object or phenomena to be observed. On top of it arethe raw observational data: they contain only selected information, plus noise, instru-mental signatures. The next box depicts the reduced, cleaned data— here everythingimportant is present, and this is the appropriate stage to archive the data. In the nextphase of research this data is visualized and combined with data from catalogs andthe literature and entered into complex databases. No new information is observedhere, only the existing information is being combined, increasing the volume. Thelast phase is the publication — the information is concise, offering another opportu-nity to archive.204 HollThe important point here is that there are phases in the research process when allthe important data could be captured and archived, at a minimal cost. Complex datastructures could improve functionality, but are hard to maintain in the long term. Thesecould be always rebuilt from the basic data set.Our solution will be to set up a two-tier archive: one part will store the raw datacoming from the telescopes. The camera pipeline will store everything in two copies:one copy will go to the archive, the other to the place chosen by the observer.The other part of the archive will contain clean, reduced data, processed by theobserver. This part will hold all additional types of data (e.g. observatory publications,documentation, digitized plates etc.), and will be publicly visible. It is an open question— it depends on our capabilities to expand the storage — whether the raw data of thematerial deposited in the public part would be deleted from the raw archive section ornot.Data FormatsFITS is an old and proven standard. The CCD images and scanned plates will be kept inFITS. For the documents, we will use PDF, or rather PDF/A as much as possible. Thethird choice is the simplest: plain ASCII text. Limiting the accepted data formats mightcause inconvenience for the depositing researchers, but helps to keep the long-termcosts and risks lower.MetadataPipelines should be fixed to provide as complete metadata as possible. We intend toemploy members of the IT/library staff for the maintenance of the FITS headers. Theway metadata are kept within the file is a strength of the FITS format. For documents— observatory publications and others — we prefer to put the most important meta-data into the text of the document — author’s name, title, date, publication name, etc.,should be visible on the printed document. Meanwhile, it should be present in the PDFmetadata too.We intend to keep our archive robust — most metainformation should be recon-structible from a heap of files. There will be databases facilitating search, but theircontents should be regenerated from the files themselves.Continuity — a VirtueLibrarians knowwell the benefits of publishing the LISA proceedings in a series. I thinkthat “sporadic” articles have less impact than those published in a well-known journal.My suggestion is to keep the observatory’s publication series running, and publish everyimportant piece of information there. Even if it is not printed any longer, the frameworkof the series will give strength to the published information. Each published articleshould get a unique identifier known outside the observatory: a bibcode or (preferablyand) a DOI.Small Data Archives and Libraries 205Cross-linkingDatasets and data files need to have unique identifiers assigned (Eichhorn et al. 2007).A local naming system should be devised, and a name resolution service for the localdata identifiers (PrivateId) should be set up. Research papers should contain uniquedataset identifiers. If it is impractical in the body of the article, they should be placedin supplementary files, which could be published only in the electronic version (Holl,Kalaglarsky, Tsvetkov et al. 2006). Persistent identifiers of relevant information shouldbe inserted in the FITS headers of the data files, like telescope and detector documen-tation published in some of the observatory publication series, or research papers basedon the given dataset. Obviously, bibcodes (DOIs) of the latter could only be inserted aposteriori. This requires manpower: the work of a librarian.Costs and ManpowerSmall observatories probably can not allocate substantial budgets to archiving. Long-term archiving costs should be kept minimal. Raw data archiving should be done au-tomatically and built in to the data acquisition pipeline. Science-ready data should bearchived by the person or team who acquired it, after the publication of the primarypaper(s), or before the proprietary period ends. In small observatories we could notexpect to have complex reduction pipelines maintained centrally. Data should be re-duced on a best-effort basis by the observer. Archival costs should be budgeted withinthe framework of the given research project and data should be archived in a standardformat, requiring minimal migration costs for the long term.The maintenance of the storage will be done by the IT staff. Hardware migra-tion, as we already mentioned, should be simple. Using standard formats, like FITS orPDF/A, no format migration needs are foreseen for a long time. Metadata maintenanceshould be done by the IT/library staff.VisibilityVisibility is beneficial for maintenance. Errors (either present or creeping in) are de-tected better if the data are used more. VO techniques could provide good visibility,and out-of-the-box free software is available to set up a VO archive. Textual documentsshould be published in established series and reported to ADS. Data identifiers shouldbe referred to in papers. A possible way to expose documents is to set up an institutionalrepository — there are excellent free software tools for this purpose as well.Incentives, Drivers, BenefitsWhat are the incentives for researchers to deposit their data? First of all, depositingshould be mandated by the director. (Or perhaps by publisher, or funder. Fundersshould require archiving plans and accommodate archiving costs.) Annual reportsshould refer to publications and data. The latter should be freely available, either inexternal archives, or in the IR or institutional archive. Archive policies should safe-guard the researcher’s and observatory’s interests with a suitable embargo policy. Thearchive could take the burden of storage off the individual researcher or the group, at206 Hollleast to some degree. If there is no money for digitization, digitization/archiving shouldat least be done on demand. If depositing could be automated, the archive could growfaster. It is easy with the raw data. On the publication side, protocols like SWORD1could help. Provenance of the data should be maintained. If the observers and theinstitute are acknowledged and papers describing the observations cited, researchers’willingness to deposit will increase.Maintenance and BackupThe electronic archive needs continuous care and maintenance, although that of lowintensity. Mechanisms well-known in the management of server computers and ser-vices could monitor the operation of the storage and data services. Hardware migrationshould be performed if storage technology changes, and migration of the registereddata formats present in the archive should be done when necessary — not too often ifwe choose the formats carefully. Printed documents were distributed in a large numberof copies, and that ensured the safety of information. Electronic archives need off-sitebackups in addition to local ones.The electronic data archive is one of the points where IT and Library meet. Dataand publications are inseparable. Librarians are skilled in working with metadata;moreover, the concepts of long-term preservation are known to them.Acknowledgments. The author is grateful for the support of the organizers and theFOL (Friends of LISA).ReferencesEichhorn, G., Accomazzi, A., Grant, C.S. et al. 2007, in ASP Conf. Ser. Vol. 377, Library andInformation Services in Astronomy V, ed. S. Ricketts, C. Birdie & E. Isaksson (SanFrancisco: ASP), 36Holl, A., Kalaglarsky, D.G., Tsvetkov, M.K. et al. 2006, in Virtual Observatory: Plate ContentDigitization, Archive Mining and Image Sequence Processing, ed. M.K. Tsvetkov, V.Golev, F. Murtagh & R. Molina (Sofia: Heron Press), 374Vargha, D., Kanyo´, S. 1998, ...csillagkorona´k e´jfe´li bara´tja: Tittel Pa´l e´lete e´s mu˝ko¨de´se, (Bu-dapest: Akade´miai Kiado´)1http://www.swordapp.org

Small Data Archives and Libraries

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library