PDF of a powerpoint presentation from the 2014 International Internet Preservation Consortium (IIPC) General Assembly, Paris, France, May 21, 2014.  Also available on Slideshare.https://digitalcommons.odu.edu/computerscience_presentations/1011/thumbnail.jp

Alsum, Ahmed

de Sompel, Herbert Van

Nelson, Michael L.

Rosenthal, David

Weigle, Michele C.

English

Old Dominion University

Old Dominion UniversityODU Digital CommonsComputer Science Presentations Computer Science5-21-2014Profiling Web ArchivesMichael L. NelsonOld Dominion University, mnelson@odu.eduAhmed AlsumOld Dominion UniversityMichele C. WeigleOld Dominion University, mweigle@odu.eduHerbert Van de SompelDavid RosenthalFollow this and additional works at: https://digitalcommons.odu.edu/computerscience_presentationsPart of the Archival Science CommonsThis Book is brought to you for free and open access by the Computer Science at ODU Digital Commons. It has been accepted for inclusion inComputer Science Presentations by an authorized administrator of ODU Digital Commons. For more information, please contactdigitalcommons@odu.edu.Recommended CitationNelson, Michael L.; Alsum, Ahmed; Weigle, Michele C.; de Sompel, Herbert Van; and Rosenthal, David, "Profiling Web Archives"(2014). Computer Science Presentations. 9.https://digitalcommons.odu.edu/computerscience_presentations/9Profiling Web ArchivesMichael L. NelsonAhmed AlSum, Michele C. WeigleHerbert Van de Sompel, David RosenthalIIPC General AssemblyParis, France, May 21, 20141Where's that issuewith the Afghan girl?789Prior IIPC Memento Aggregator Project• Ten IIPC archives, led by LANL• Conceived at 2011 IIPC meeting• Results reported at 2012 IIPC meetingo http://netpreserve.org/sites/default/files/resources/Sanderson.pdf• Two highlights:Stop and Rethink…• LANL's processing was informative from a "big data" perspective, but was neither scalable nor sustainableo "send us your CDX" == hard for both partieso there are lots of URIs in the world• Will only get worse with:o more archives… o …doing more archivingLeverage Memento Aggregators• Memento aggregator currently broadcast URI lookups to all known archives• New approach: 1. build profiles based on sampling from URI lookups (optionally supplement with CDX files when available)2. Use archive profiles for informing Memento aggregator "query routing" decisions 3. Share serialized profiles with other IIPC partnershttp://mementoproxy.lanl.gov/aggr/timemap/link/1/http://www.bnf.fr/Profiling Studies• TPDL 2013o 12 archives, March 2013, public web archives used but techniques apply generallyo sampling only, no CDX access• IJDL 2014 (to appear)o 15 archives (+4, -1), October 2013o slightly larger sample URI dataseto results similarURI Lookup = Limited Information16GET /aggr/timegate/http://www.bnf.fr/ HTTP/1.1Host: mementoproxy.lanl.gov Accept-Datetime: Sun, 29 May 2005 02:46:53 GMTAccept-Language: fr; q=1.0, en; q=0.5…1. Original URI2. Memento-Datetime3. Preferred URI213Where to find Mementos for …17http://www.japantimes.co.jp/Where to find Mementos for …18http://www.japantimes.co.jp/Where to find Mementos for …19http://www.bnf.frWhere to find Mementos for …20http://www.bnf.frResearch QuestionProblem• Profile public web archives according to the following dimensions:o Top-level domainso Languageso Growth rateo Archival dateMotivation• Determine who is archiving what• Optimize query routing for a Memento Aggregator21Web Archives in this ExperimentFull text URI-lookupInternet Archive √Library of Congress     √Icelandic Web Archive √Library and Archives Canada √ √British Library √ √UK National Library   √ √Portuguese Web Archive √ √Web Archive of Catalonia √ √Croatian Web Archive  √ √Archive of the Czech Web √ √National Taiwan University √ √Archive It √ √22Experiment Set Up• Sample URIs from seven different sources• Retrieve the TimeMap for each URI from all archiveso A TimeMap lists all Mementos for a given URIo A Memento is an archived version of a resource• Analyze who has holdings for which URIs23Sampling URIs - DMOZ1. DMOZ:Randomo 10,000 URIs randomly sampled from DMOZ directory (~5M URIs).2. DMOZ:TLD - 2% for each TLD from DMOZ  or 100 URIs whichever is greatero 52 TLDs (com 23,470) (de 6,332), (org 4,025), (uk 3,309), (net2,073), (it 1,775), (jp 1379), (ru 1244), (fr 1154), (pl 1062), (au764), (ca 642), (at 438), (edu 390), (cz 385), (tr 334), (info 319), (cn 278), (us 266), (nz 265), (es 238), (ar 213), (no 150), (br 149), (tw 141), (za 118), (fi 113), ( 100 URIs for [ae, cat, cl, cu, eg, gov, id, in, ir, is, ke, kr, ma, mt, mx, my, na, pe, pk, pt, sa, to, uy, zw])3. DMOZ:Languages - 100 URIs for each languageo 24 languages: Icelandic, Portuguese, Catalan, Afrikaans, Arabic, Indonesian, Chinese (Simplified), Chinese (Traditional), Dutch, Spanish, French, Greek, Hindi, Italian, Japanese, Korean, Norwegian, Persian, Polish , Russian, Turkish, Ukrainian 24• Query the fulltext search interface of select web archives with two sets of query terms.4. Top 1-Gram from Bingo Most are English5. Top 1000 query terms from Yahoo in 9 languageso Excluding general keywords such as: Obama, Facebook.25Sampling URIs – Web Archives Full TextChineseEnglishFrenchGermanItalianJapaneseKoreanPortugueseSpanishYahoo BingArchive with FullText searchAIT 26 2066 3512 3837 3321 119 2 2434 2141 12617 3953BL 163 2354 2350 2240 2068 225 131 1940 2056 6430 3187CAN 49 800 804 646 601 77 113 580 514 1351 1107CR 54 706 697 703 701 74 19 599 600 1599 1201CZ 363 1782 1578 1695 1519 577 114 1310 1278 6081 3360CAT 28 2775 2496 2448 2280 209 129 2164 2429 8996 4241PO 91 2460 3603 3081 3113 53 69 3267 3177 14126 5004TW 357 178 176 165 157 106 7 198 119 1004 354UK 0 2698 2009 2049 2046 0 0 1903 1871 8261 343126Sampling URIs – Web Archives Full TextChineseEnglishFrenchGermanItalianJapaneseKoreanPortugueseSpanishYahoo BingArchive with FullText searchAIT 26 2066 3512 3837 3321 119 2 2434 2141 12617 3953BL 163 2354 2350 2240 2068 225 131 1940 2056 6430 3187CAN 49 800 804 646 601 77 113 580 514 1351 1107CR 54 706 697 703 701 74 19 599 600 1599 1201CZ 363 1782 1578 1695 1519 577 114 1310 1278 6081 3360CAT 28 2775 2496 2448 2280 209 129 2164 2429 8996 4241PO 91 2460 3603 3081 3113 53 69 3267 3177 14126 5004TW 357 178 176 165 157 106 7 198 119 1004 354UK 0 2698 2009 2049 2046 0 0 1903 1871 8261 343127Sampling URIs – Web Archives Full TextSampling URIs – User Requests• Sampling from user requests for archived web resources6. Sample from IA Wayback Machine Log fileso 1,000 URIs randomly sampled from Feb 22, 2012 to Feb 26, 2012.7. Sample from Memento Aggregator log fileso 100 URIs randomly sampled from LANL Memento Aggregator between 2011 to 2013.28Archive Coverage per Sample29100%35%Entire SampleTLD Coverage across Archives (1)30Entire SampleTLD Coverage across Archives (2)31Entire SampleTLD Distribution per Archive32DMOZ:TLD SampleTLD Distribution per Archive33Web Archives Full Text SampleLanguage Coverage per Archive34DMOZ SampleArchive Growth Rate35Entire SampleQuery Routing Evaluation36Study Results• Introduced sampling to profile web archives using available infrastructure, no privileged access• Coverage:o Internet Archive provides broad coverageo National archives have good coverage for their domainso Surprising coverage by certain archives• Query Routing:o In 84% of the cases, all existing Mementos for a TLD can be found by using IA and two additional top archives for a TLDo In 55% of the cases, all existing Mementos for a TLD can be found by using the top 3 archives for a TLD, excluding IA37Next Steps With the IIPC38• Finding the right granularityo too fine: http://www.bnf.fr/fr/evenements_et_culture/a.passe_bnf.htmlo too coarse: .fro just right?: bnf.fr, www.bnf.fr, gallica.bnf.fr, www.bnf.fr/fr/• Generating profileso what are desirable / representative sample sets: domains, languages, regions, etc. -- what's missing?o local CDX analysis tools (can help with cold start problem)• Profile formato community input (yet another metadata format)o github (or other tools) for exchange & integration{"Profile":{"Name":"Taiwan Web Archive","URI":"http://webarchive.lib.ntu.edu.tw","TimeGate":    "http://mementoproxy.cs.odu.edu/tw/timegate/","Code":"TW","Age":"Tue, 15 Jul 1997 00:00:00 GMT","TLD":[{"tw":0.6},{"cn":0.08},{"hk":0.04},{"eg":0.04},{"gov":0.04},{"my":0.04},{"jp":0.04},{"kr":0.02}],"Language":[{"zh-TW":0.5},{"zh-CN":0.25},{"id":0.08},{"ar":0.08}],"GrowthRate":[{"199707":[4,4]},{"200202":[1,1]},{"200607":[30,62]},{"200608":[20,80]},{"200609":[5,9]},{"200612":[77,129]},... // other values truncated{"201308":[7,94]},{"201309":[2,94]}]}}A Possible Serialization{Light, Dim, Dark} Archives42• Work to date has assumed light archives because our focus has been on sampling archives we don't control• Applicable to a continuum of archives:o download/fork and run "dark-sample.py"o it accesses sample URIs from IIPC githubo issues URI lookups to local archiveo write/update your archive profile in IIPC github with machine readable IP restrictions o all profiles -- light/dim/dark -- now available to Memento aggregators and other IIPC analysis toolsProfiles = Easy Discovery, Sharinghttp://netpreserve.org/aggr/timemap/link/1/http://www.bnf.fr/

Profiling Web Archives

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1011&context=computerscience_presentations

Profiling Web Archives

Abstract

Similar works

Full text

Available Versions

Old Dominion University