Technical alignment

This essay discusses the importance of the areas of infrastructure and testing to help digital preservation services demonstrate reliability, transparency, and accountability. It encourages practitioners to build a strong culture in which transparency and collaborations between technical frameworks are valued highly. It also argues for devising and applying agreed-upon metrics that will enable the systematic analysis of preservation infrastructure. The essay begins by defining technical infrastructure and testing in the digital preservation context, provides case studies that exemplify both progress and challenges for technical alignment in both areas, and concludes with suggestions for achieving greater degrees of technical alignment going forward

Federating Heterogeneous Digital Libraries by Metadata Harvesting

This dissertation studies the challenges and issues faced in federating heterogeneous digital libraries (DLs) by metadata harvesting. The objective of federation is to provide high-level services (e.g. transparent search across all DLs) on the collective metadata from different digital libraries. There are two main approaches to federate DLs: distributed searching approach and harvesting approach. As the distributed searching approach replies on executing queries to digital libraries in real time, it has problems with scalability. The difficulty of creating a distributed searching service for a large federation is the motivation behind Open Archives Initiatives Protocols for Metadata Harvesting (OAI-PMH). OAI-PMH supports both data providers (repositories, archives) and service providers. Service providers develop value-added services based on the information collected from data providers. Data providers are simply collections of harvestable metadata. This dissertation examines the application of the metadata harvesting approach in DL federations. It addresses the following problems: (1) Whether or not metadata harvesting provides a realistic and scalable solution for DL federation. (2) What is the status of and problems with current data provider implementations, and how to solve these problems. (3) How to synchronize data providers and service providers. (4) How to build different types of federation services over harvested metadata. (5) How to create a scalable and reliable infrastructure to support federation services. The work done in this dissertation is based on OAI-PMH, and the results have influenced the evolution of OAI-PMH. However, the results are not limited to the scope of OAI-PMH. Our approach is to design and build key services for metadata harvesting and to deploy them on the Web. Implementing a publicly available service allows us to demonstrate how these approaches are practical. The problems posed above are evaluated by performing experiments over these services. To summarize the results of this thesis, we conclude that the metadata harvesting approach is a realistic and scalable approach to federate heterogeneous DLs. We present two models of building federation services: a centralized model and a replicated model. Our experiments also demonstrate that the repository synchronization problem can be addressed by push, pull, and hybrid push/pull models; each model has its strengths and weaknesses and fits a specific scenario. Finally, we present a scalable and reliable infrastructure to support the applications of metadata harvesting

FreeLib: A Self-Sustainable Peer-to-Peer Digital Library Framework for Evolving Communities

The need for efficient solutions to the problem of disseminating and sharing of data is growing. Digital libraries provide an efficient solution for disseminating and sharing large volumes of data to diverse sets of users. They enable the use of structured and well defined metadata to provide quality search services. Most of the digital libraries built so far follow a centralized model. The centralized model is an efficient model; however, it has some inherent problems. It is not suitable when content contribution is highly distributed over a very large number of participants. It also requires an organizational support to provide resources (hardware, software, and network bandwidth) and to manage processes for collecting, ingesting, curating, and maintaining the content. In this research, we develop an alternative digital library framework based on peer-to-peer. The framework utilizes resources contributed by participating nodes to achieve self-sustainability. A second key contribution of this research is a significant enhancement of search performance by utilizing the novel concept of community evolution. As demonstrated in this thesis, bringing users sharing similar interest together in a community significantly enhances the search performance. Evolving users into communities is based on a simple analysis of user access patterns in a completely distributed manner. This community evolution process is completely transparent to the user. In our framework, community membership of each node is continuously evolving. This allows users to move between communities as their interest shifts between topics, thus enhancing search performance for users all the time even when their interest changes. It also gives our framework great flexibility as it allows communities to dissolve and new communities to form and evolve over time to reflect the latest user interests. In addition to self-sustainability and performance enhancements, our framework has the potential of building extremely large collections although every node is only maintaining a small collection of digital objects

StreamOnTheFly: a network for radio content dissemination

A distributed digital library has been designed and implemented for the support of community radios. This framework, developed by the StreamOn- TheFly IST project of the EU, provides a common background for preparation, archival, exchange and reuse of radio programs and supports radio personalization. The architecture is based on a decentralized network of software components using automatic metadata replication in a peer-to-peer manner. This approach combines the principles and practice of OAI (Open Archives Initiative) with the peer-to-peer networking paradigm, and extends usual content dissemination with the aggregation of use statistics and user feedback. The network also promotes social self-organization of the community and a new common metadata schema and content exchange format

Open Peer to Peer Technologies

Peer-to-peer applications allow us to separate out the concepts of authoring information and publishing that same information. It allows for decentralized application design, something that is both an opportunity and a challenge. All the peer-to-peer applications, in various ways, return the content, choice, and control to ordinary users. Tiny end points on the Internet, sometimes even without knowing each other, exchange information and form communities. In these applications there are no more clients and servers, instead the communication takes place between cooperating peers. There are many applications nowadays which are being labeled as peer-to-peer. A way to examine the distinction of whether an application is peer-to-peer or not is to check on the owner of the hardware that the service runs on. Like Napster, if the huge part of the hardware that Napster runs on is owned by the Napster users on millions of desktops then it is peer-to-peer. Peer-to-peer is a way of decentralizing not only features, but costs and administration also. By decentralizing data and therefore redirecting users so they download data directly from other user's computers, Napster reduced the load on its servers to the point where it could cheaply support tens of millions of users. The same principle is used in many commercial peer-to-peer systems. In short peer-to-peer cannot only distribute files. It can also distribute the burden of supporting network connections. The overall bandwidth remains the same as in centralized systems, but bottlenecks are eliminated at central sites and equally importantly, at their ISPs. Search techniques are important to making peer-to-peer systems useful. But there is a higher level of system design and system use. Topics like trust, accountability and metadata have to be handled before searching is viable

A schema-based peer-to-peer infrastructure for digital library networks

[no abstract