Digital forensics formats: seeking a digital preservation storage format for web archiving

In this paper we discuss archival storage formats from the point of view of digital curation and preservation. Considering established approaches to data management as our jumping off point, we selected seven format attributes which are core to the long term accessibility of digital materials. These we have labeled core preservation attributes. These attributes are then used as evaluation criteria to compare file formats belonging to five common categories: formats for archiving selected content (e.g. tar, WARC), disk image formats that capture data for recovery or installation (partimage, dd raw image), these two types combined with a selected compression algorithm (e.g. tar+gzip), formats that combine packing and compression (e.g. 7-zip), and forensic file formats for data analysis in criminal investigations (e.g. aff, Advanced Forensic File format). We present a general discussion of the file format landscape in terms of the attributes we discuss, and make a direct comparison between the three most promising archival formats: tar, WARC, and aff. We conclude by suggesting the next steps to take the research forward and to validate the observations we have made

MODELING AND EVALUATION OF A HYBRID OPTICAL AND MAGNETIC DISK STORAGE ARCHITECTURE

A hybrid storage system combining optical disks and magnetic disks is proposed and evaluated via mathematical models. Unlike most current applications of optical disk technology, which consider static databases or deferred update, this research considers environments with a moderate level of near real-time updates. An example of such an environment is databases for administrative decision support systems (DSS). The proposed hybrid storage system uses a write-once, read-many optical disk device (ODD) for the database and a conventional magnetic disk (MD) for storage of a differential file. Periodically, the differential file is used to refresh the ODD file by writing updated records to free space on the ODD. When available free space on the ODD is exhausted, the file is written to new ODD media - - a reorganization operation. Models of storage cost are developed to determine the optimum refresh cycle time, t*, and optimum reorganization cycle time, T*. Parameters of the model include data file volatility, file size, device costs, and costs for refresh and reorganization. Numerical results indicate that the hybrid system is attractive for a broad range of database environments

Developing a Robust Migration Workflow for Preserving and Curating Hand-held Media

Many memory institutions hold large collections of hand-held media, which can comprise hundreds of terabytes of data spread over many thousands of data-carriers. Many of these carriers are at risk of significant physical degradation over time, depending on their composition. Unfortunately, handling them manually is enormously time consuming and so a full and frequent evaluation of their condition is extremely expensive. It is, therefore, important to develop scalable processes for stabilizing them onto backed-up online storage where they can be subject to highquality digital preservation management. This goes hand in hand with the need to establish efficient, standardized ways of recording metadata and to deal with defective data-carriers. This paper discusses processing approaches, workflows, technical set-up, software solutions and touches on staffing needs for the stabilization process. We have experimented with different disk copying robots, defined our metadata, and addressed storage issues to scale stabilization to the vast quantities of digital objects on hand-held data-carriers that need to be preserved. Working closely with the content curators, we have been able to build a robust data migration workflow and have stabilized over 16 terabytes of data in a scalable and economical manner.Comment: 11 pages, presented at iPres 2011. Also publishing in corresponding conference proceeding

IPAC Image Processing and Data Archiving for the Palomar Transient Factory

The Palomar Transient Factory (PTF) is a multiepochal robotic survey of the northern sky that acquires data for the scientific study of transient and variable astrophysical phenomena. The camera and telescope provide for wide-field imaging in optical bands. In the five years of operation since first light on 2008 December 13, images taken with Mould-R and SDSS-g′ camera filters have been routinely acquired on a nightly basis (weather permitting), and two different Hα filters were installed in 2011 May (656 and 663 nm). The PTF image-processing and data-archival program at the Infrared Processing and Analysis Center (IPAC) is tailored to receive and reduce the data, and, from it, generate and preserve astrometrically and photometrically calibrated images, extracted source catalogs, and co-added reference images. Relational databases have been deployed to track these products in operations and the data archive. The fully automated system has benefited by lessons learned from past IPAC projects and comprises advantageous features that are potentially incorporable into other ground-based observatories. Both off-the-shelf and in-house software have been utilized for economy and rapid development. The PTF data archive is curated by the NASA/IPAC Infrared Science Archive (IRSA). A state-of-the-art custom Web interface has been deployed for downloading the raw images, processed images, and source catalogs from IRSA. Access to PTF data products is currently limited to an initial public data release (M81, M44, M42, SDSS Stripe 82, and the Kepler Survey Field). It is the intent of the PTF collaboration to release the full PTF data archive when sufficient funding becomes available

Fourth NASA Goddard Conference on Mass Storage Systems and Technologies

This report contains copies of all those technical papers received in time for publication just prior to the Fourth Goddard Conference on Mass Storage and Technologies, held March 28-30, 1995, at the University of Maryland, University College Conference Center, in College Park, Maryland. This series of conferences continues to serve as a unique medium for the exchange of information on topics relating to the ingestion and management of substantial amounts of data and the attendant problems involved. This year's discussion topics include new storage technology, stability of recorded media, performance studies, storage system solutions, the National Information infrastructure (Infobahn), the future for storage technology, and lessons learned from various projects. There also will be an update on the IEEE Mass Storage System Reference Model Version 5, on which the final vote was taken in July 1994

A guide to the National Space Science Data Center

This is the second edition of a document that was published to acquaint space and Earth research scientists with an overview of the services offered by the NSSDC. As previously stated, the NSSDC was established by NASA to be the long term archive for data from its space missions. However, the NSSDC has evolved into an organization that provides a multitude of services for scientists throughout the world. Brief articles are presented which discuss these services. At the end of each article is the name, address, and telephone number of the person to contact for additional information. Online Information and Data Systems, Electronic Access, Offline Data Archive, Value Added Services, Mass Storage Activities, and Computer Science Research are all detailed

Digital imaging technology assessment: Digital document storage project

An ongoing technical assessment and requirements definition project is examining the potential role of digital imaging technology at NASA's STI facility. The focus is on the basic components of imaging technology in today's marketplace as well as the components anticipated in the near future. Presented is a requirement specification for a prototype project, an initial examination of current image processing at the STI facility, and an initial summary of image processing projects at other sites. Operational imaging systems incorporate scanners, optical storage, high resolution monitors, processing nodes, magnetic storage, jukeboxes, specialized boards, optical character recognition gear, pixel addressable printers, communications, and complex software processes

Volume 12, Number 3, September 1992 OLAC Newsletter

Digitized September 1992 issue of the OLAC Newsletter