Dudczak, Adam

English

PSNC Institutional Repository

Access IT Training Introduction Metadata reuse scenarios OAI-PMH  Object content reuse scenarios OAI-ORE Persistent identifiers Conclusions „It is highly probable that digital libraries are created for users”  We assume that someone will: read books analyze manuscripts watch old documentaries … Go beyond „walls” (pages) of our digital library Discuss how to attract users to our digital library Show why it is a good thing to encourage others(users/services) to reuse digital library objects Show why OAI-PMH is such a big thing Investigate how digital libraries can be used in education Talk a bit about persistent identifiersUsersServicesGeneral purposesearch enginesAggregators Other… Metadata records Various protocols and formats Object content Text, images, video etc. Introduction Metadata reuse scenarios OAI-PMH  Object content reuse scenarios OAI-ORE Persistent identifiers Conclusions Metadata can be reused through aggregation Definition: “An Aggregator is an organization that collects metadata from its group of content providers and transmits them to Europeana, helps content providers with guidance on conformance with Europeana norms and converts metadata if necessary. The aggregator also supports the content providers with administration, operations and training.” Europeana Content Strategy Aggregator gathers metadata from a number of repositories/digital libraries On top of those data new advanced network services can be built Metadata unification and cleaning Forwarding metadata to other services Unified access point to all resources„The model of aggregation of content is of crucial importance and will enable Europeana to reach its objectives. Aggregators, on a national, regional or vertical level, play a key role not only in aggregating content, but also in the organizational structure, standardization of content, services to end-users and future sustainability of Europeana and related projects and aggregators”Europeana Content Strategy Two types of aggregation Horizontal aggregation ▪ General purpose aggregation▪ Gathers metadata of various resources from different institutions Vertical aggregation▪ Thematic aggregator ▪ Gathers information about specific types of objects▪ e.g. European Film Gateway Metadata aggregation can be realized using different means one of them is OAI-PMH OAI-PMH – Open Archives Initiative Protocol for Metadata Harvesting Defines two actors ▪ data providers (repositories)▪ service providers (harvesters) Service providers provide services based on the metadata harvested from data providersOAI for Beginners - the Open Archives Forum online tutorial(http://www.oaforum.org/tutorial/english/intro.htm) Service providers provide services based on the metadata harvested from data providersOAI for Beginners - the Open Archives Forum online tutorial(http://www.oaforum.org/tutorial/english/intro.htm) OAI-PMH – Open Archives Initiative Protocol for Metadata Harvesting XML based Built on top of HTTP protocol Widely used ▪ Over 2 000 repositories worldwide containing  several millions of records  More info: http://www.openarchives.org/ OAI-PMH Repository data structure Repository contains items Item may be described in multiple metadata schemas Item must be described with Dublin Core schema (15 general attributes – title, author etc.) Metadata is exchanged via records ▪ entire description of an item in the particular schema▪ headers (basic description - id, modification date etc.) OAI-PMH Repository data structure Repository may define sets (groups of items) –possible usage:▪ Subject sets, MIME type sets, … Set may contain subsets Each item may belong to zero or more sets Sets are used for selective harvesting – they allow to harvest only a part of a repository Communication between OAI-PMH data provider and service provider is based on predefined verbs, which includes: Identify ▪ Basic information about repository and its OAI-PMH interface▪ http://www.wbc.poznan.pl/dlibra/oai-pmh-repository.xml?verb=Identify ListIdentifiers▪ Lists identifiers for a given set and metadata format▪ http://www.wbc.poznan.pl/dlibra/oai-pmh-repository.xml?verb=ListIdentifiers&set=DigitalLibraryOfWielkopolska&from=1998-01-15&metadataPrefix=oai_dc ListMetadataFormats▪ Lists supported metadata formats▪ http://www.wbc.poznan.pl/dlibra/oai-pmh-repository.xml?verb=ListMetadataFormats ListRecords▪ Lists records for a given set and metadata format▪ http://www.wbc.poznan.pl/dlibra/oai-pmh-repository.xml?verb=ListRecords&from=1998-01-15&set=DigitalLibraryOfWielkopolska&metadataPrefix=oai_dc GetRecord▪ Gets content of given record▪ http://www.wbc.poznan.pl/dlibra/oai-pmh-repository.xml?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:www.wbc.poznan.pl:1490 ListSets▪ Lists available sets▪ http://www.wbc.poznan.pl/dlibra/oai-pmh-repository.xml?verb=ListSets More examples can be found in :“The Open Archives Initiative Protocol for Metadata Harvesting” http://www.openarchives.org/OAI/openarchivesprotocol.html Pionier Digital Libraries Federation is a Polish national aggregator http://fbc.pionier.net.pl/   It was created to facilitate the use of resources from Polish digital libraries To increase the visibility of these resources in the Internet To create new, advanced network servicesboth for end-users and digital libraries creators on the base of these resources Basic assumptions No need nor requirement to move resources to the DLF No fees for the use of the DLF and for being a part of it Open standards are the basis for cooperation▪ Particular digital libraries can use different technological platformsRegional digital librariesInstitutional digital librariesOverall number of digitalobjects 340 thousandsNumber of active digitallibraries: 21 regional 28 institutionalNumber of cooperatinginstitutions: over 200 libraries, museumsand archivesDigital LibrariesFederationInstitutionalRegionalLibrariesArchivesMuseums….National(exclude??)OtherInstitutionsDigital librariesMetadataaggregator Basic functions Search in the available publications▪ Simple▪ Advanced Digitization plans▪ Searchable▪ Report▪ API for the prevention of duplicated digitization Location of digital objects on the basis of their OAI Identifiers Database of Polish digital libraries Statistics and reports Information in the DLF is updated on the daily (nightly) basis OpenSearch plug-in for web browsers available since the beginning of the DLF Must be installed manually by the user The user must know that there are digital libraries and the Federation Several months of activities towards the inclusion of DLF plug-in into official Polish Firefox releases Negotiations with Polish Mozilla team, and finally with Mozilla Foundation Technical changes related to high performance requirements▪ Autosuggest service response time less than 300 msDigital LibrariesFederationsearch plugin June 17th, 2008 – Firefox 3.0 released with DLF plug-in included in the Polish version July 3rd, 2008 – Firefox 2.0.15 released with DLF plug-in included in the Polish version Resources of Polish digital libraries available for Polish internet users together with Google, Wikipedia etc.  Currently Firefox 2 and 3 have together 41% of web browsers market in Poland Measurable result?05000100001500020000250003000035000400004500050000Jul 2007 Aug 2007Sep 2007Oct 2007Nov 2007Dec 2007Jan 2008Feb 2008Mar 2008Apr 2008May 2008Jun 2008Jul 2008 Aug 2008Sep 2008120625836403821911893 12350151001100618176 184122083726251426353676747200DLF included inFirefox 2 and 3 Pionier DLF has its own OAI-PMH 2.0 compliant interface All gathered data are available through this interface It offers the support for deleted records and incremental harvesting It also allows to create dynamic OAI sets on a search query basis Pionier DLF exposes unified metadata from Polish Digital Libraries in ESE format Since 11th December 2009 that interface is harvested by Europeana OAIster http://oaister.worldcat.org/ 23 million of records from 1100 institutions Resources also visible through WorlCat.org ScientificCommons.org http://scientificcommons.org „The major aim of the project is to develop the worldslargest communication medium for scientific knowledge products which is freely accessible to the public.” 13 million of scientific publications Europeana wants to aggregate all publicly available digital content relevant to the term “European cultural and scientific heritage” What about vertical services based on a large scale aggregation? There is a need to enable precise selective harvesting of aggregated metadata How to obtain the metadata? Solution 1: Harvest all records from repositories, decide what records are useful▪ A lot of useless data is harvested and processed  Solution 2: Harvest only specific sets of items matching the theme of the portal▪ Each harvested repository must define a set / sets matching the theme of the portal – practically impossible Solution 3: DIY variant of scenario 2 – define a set containing items matching the theme of the portal and harvest it▪ Not supported in the OAI-PMH protocol  Dynamic sets – OAI-PMH protocol compatibility Harvester side▪ If a harvester does not supports dynamic sets, it will be still able to harvest the repository supporting such sets Repository side▪ If a repository does not supports dynamic sets, it still may be harvested by a harvester supporting such sets▪ The repository extended with dynamic sets should be compatible with OAI-PMH validators Dynamic sets – implementation Harvester side▪ Prepare the support for OAI-PMH set harvesting▪ Analyze the nature of metadata in particular repository and prepare proper dynamic set definition to use during harvesting Repository side▪ Modify the harvesting requests processing to support the definition of dynamic sets ▪ This may be based on the search mechanism already implemented in the majority of repositories – in such case the support for CQL queries must be assured Dynamic sets – specification Sets defined by repository users Contain items that matched dynamic set definition sent by the user The definition is in fact a CQL query encoded into a set name▪ CQL – Contextual Query Language▪ Part of SRU protocol specification – used in integrated library systems as a replacement for the z39.50 protocol to obtain bibliographic descriptions▪ Allows to define simple and complex queries▪ Compatible with any metadata schema▪ Example: dc.creator = “Albert Einstein”QueryHarvested number of Harvested % of overall number ofrepositories records repositories recordsnone (all records) 16 93681 100,00% 100,00%dc.language eng 13 626 81,25% 0,67%dc.language ger 12 10357 75,00% 11,06%dc.type podręcznik (handbook) 4 104 25,00% 0,11%dc.type rozprawa (thesis) 5 199 31,25% 0,21%dc.type czasopismo (magazine) 16 28163 100,00% 30,06%dc.type gazeta (newspaper) 4 33793 25,00% 36,07%dc.subject pedagogika(pedagogy) 8 130 50,00% 0,14%dc.subject chemia (chemistry) 8 715 50,00% 0,76%dc.subject 8 2759 50,00% 2,95% eContentPlus ENRICH Project  Started in December 2007 The aim is to built a virtual European repository of manuscripts The metadata about the manuscripts is harvested from multiple European repositories Harvests metadata of manuscripts from several Polish digital libraries Criteria used to get objects for Manuscriptorium dc.date>=1000  dc.date<=1850  not dc.type=czasop*  not dc.type=prog*  dc.format=(app* or pd* or dj* or im*) Results: 883 objects (22.06.2009) Automatic conversion from DjVu to JPG Publication structure exposed through OAI-PMH encoded using METS Links to pages and files Info about original DjVu files and results of conversionPreviewThumbnails DART-Europe E-theses Portal  http://www.dart-europe.eu „DART-Europe is a partnership of research libraries and library consortia who are working together to improve global access to European research theses.” Resources from Polish digital libraries are exposed through OAI-PMH interface of Pionier DLF Social bookmarking http://del.icio.us, http://digg.com Bibliography organizers Zotero▪ Handles different metadata encodings : ▪ MODS, RDF, MARC, BibTeX▪ http://www.zotero.org/ Bibsonomy (http://www.bibsonomy.org/ )  … Introduction Metadata reuse scenarios OAI-PMH  Object content reuse scenarios OAI-ORE Persistent identifiers Conclusions Blogs Good way to get external links to digital librarywebsite (good for SEO) Readers know more about resources Accumulating user generated content Europeana Flickr: The Commons Panoramiohttp://www.flickr.com/photos/powerhouse_museum/2937577694http://www.flickr.com/photos/powerhouse_museum/2937577694 OAI-ORE defines standards for the description and exchange of aggregations of Web resources http://www.openarchives.org/ore/ The goal of these standards is to expose the rich content to applications that support : authoring, deposit, exchange, visualization, reuse, and preservation..Source: ORE User Guide – Primer(http://www.openarchives.org/ore/1.0/primer.html) Some solution exists in this space already (METS)  OAI ORE is built on top of successful OAI-PMH It’s a very flexible spec, promissing low adoption curve ORE can be used to aggregate whole objects (+metadata) More about ORE in the last part of tutorial Introduction Metadata reuse scenarios OAI-PMH  Object content reuse scenarios OAI-ORE Persistent identifiers Nowadays it is very easy to create new website, publish content on the web Overtime, more and more of these hyperlinks are „broken” Organization’s website was re-organized Persistent identifiers introduce a service which records  how precise location of document changes over time When end-user wishes to access a document, the identifier in his request is „resolved” The correct document is retrieved User don’t need to know the exact location of the document Support for persistent identification some administrative effort Various propositions:  Handles,  Digital Object Identifier (DOI) Archival Resource Keys (ARK) Persistent Uniform Resource Locators (PURL) Uniform Resource Names (URN) OAI Identifiers The syntax of URNs was fully specified in 1997 in another RFC, „URN Syntax” Global uniqueness: different resources cannot have the same URN Persistence: in the URN context, the name's lifespan is permanent, regardless of the lifespan of the named resource. Scalability: room to accommodate the number of names required in the nextcenturies. Syntax: urn:<NID>:<NSS> Every URN begins with the ‘urn:’ character string, followed by the Namespace Identifier (NID, e.g. ISBN) Namespace Specific String (NSS) syntax depends on the namespace identified by the NID Example: urn:isbn:3-938616-59-8 Syntax: oai:<NI>:<LI> Persistent identifier for repositories with OAI-PMH interface Identifier points to resource which holds metadata record NI – namespace identifier – repository domain LI – local identifier – points to resource in repository It does not require central resolution service No fees for using this solution Example:  oai:www.wbc.poznan.pl:8711 The more doors we have in our library there is a bigger chance that someone will come in Reuse is a key to enrichment Chance for User Generated Content Chance to attract new users Whole Web 2.0 is about reuse and exchange Open protocols and public APIs Search Engine Optimization Source of external links to our content Everything works fine as long as links are working  - persistent identifiers might be useful

How objects and metadata from digital repositories can be reused?

http://lib.psnc.pl/Content/246/05-reuse.pdf

How objects and metadata from digital repositories can be reused?

Abstract

Similar works

Full text

Available Versions

PSNC Institutional Repository