PDF of a powerpoint presentation from a Mathematics & Computer Science Seminar at Emory University, Atlanta, Georgia, October 2, 2009.  Also available on Slideshare.https://digitalcommons.odu.edu/computerscience_presentations/1025/thumbnail.jp

Klein, Martin

Nelson, Michael L.

English

Old Dominion University

Old Dominion UniversityODU Digital CommonsComputer Science Presentations Computer Science10-2-2009(Re-) Discovering Lost Web PagesMartin KleinOld Dominion UniversityMichael L. NelsonOld Dominion University, mnelson@odu.eduFollow this and additional works at: https://digitalcommons.odu.edu/computerscience_presentationsPart of the Archival Science Commons, and the Computer Sciences CommonsThis Book is brought to you for free and open access by the Computer Science at ODU Digital Commons. It has been accepted for inclusion inComputer Science Presentations by an authorized administrator of ODU Digital Commons. For more information, please contactdigitalcommons@odu.edu.Recommended CitationKlein, Martin and Nelson, Michael L., "(Re-) Discovering Lost Web Pages" (2009). Computer Science Presentations. 23.https://digitalcommons.odu.edu/computerscience_presentations/23(Re-) Discovering Lost Web PagesMathematics & Computer Science SeminarEmory UniversityOctober 2, 2009Martin Klein & Michael L. NelsonDepartment of Computer ScienceOld Dominion UniversityNorfolk VA www.cs.odu.edu/~{mklein,mln}• Web links “break”• 404 http status code -- “not found”• “soft 404” -- http server returns “200 OK”, but the resource isn’t really there• Is the content really gone? • Did it just move somewhere else in the web?• Is there a copy in search engine caches or web archives?• To find new or different copies, we need to augment digital preservation with information retrieval techniquesThe ProblemThe ActorsPut a human -- lots of humans -- in the loop for preservation purposesWeb Infrastructure (WI) [McCown07]• Web search engines (Google, Yahoo,    MSN Live) and their caches• Research Projects (CiteSeer, NSDL)• Web archives (Internet Archive,Web Base)The Environmentftp://techreports.larc.nasa.gov/pub/techreports/larc/93/tm109025.ps.Zhttp://techreports.larc.nasa.gov/ltrs/PDF/tm109025.pdfWeb Infrastructure: Refreshing & MigratingLapsed Website1same URI maps to sameor very similar content at a later time2same URI maps to differentcontent at a later time3different URI maps to sameor very similar content at the same or at a later time4the content can not be found at any URI8U1C1U1C1timeA BU1C2U1C1timeA BU2C1U1C1U1404timeA BU1??U1C1timeA BURI Content Mapping Problem9JCDL 2008http://www.jcdl2008.org/July 2008http://www.jcdl2008.org/TodayScenario 1: Same URI, Same Content10Hypertext 2006http://www.ht06.org/August 2006http://www.ht06.org/TodayScenario 2: Same URI, Different Content11PSP 2003http://www.pspcentral.org/events/annual_meeting_2003.htmlAugust 2003http://www.pspcentral.org/events/archive/annual_meeting_2003.htmlTodayScenario 3a: Same Content, Different URI12ECDL 1999http://www-rocq.inria.fr/EuroDL99/October 1999http://www.informatik.uni-trier.de/~ley/db/conf/ercimdl/ercimdl99.htmlTodayScenario 3b: Similar Content, Different URI13Greynet 1999http://www.konbib.nl/infolev/greynet/2.5.htm1999Today? ?Scenario 4: Content Not Findable At Any URI Otto: You eat a lot of acid, Miller, back in the hippie days? Miller: A lot o' people don't realize what's really going on. They view life as a bunch o' unconnected incidents 'n things. They don't realize that there's this, like, lattice o' coincidence that lays on top o' everything. Give you an example; show you what I mean: suppose you're thinkin' about a plate o' shrimp. Suddenly someone'll say, like, plate, or shrimp, or plate o' shrimp out of the blue, no explanation. No point in lookin' for one, either. It's all part of a cosmic unconsciousness.15• Experience of causally unrelated events occurring together in a meaningful manner• Events reveal underlying pattern, framework bigger than any of the synchronous systems• Carl Gustav Jung (1875-1961)• “meaningful coincidence”picture from http://www.crystalinks.com/jung.htmlSynchronicity16The Bigger PictureSynchronicity Architecture• Firefox extension catches 404 error (or initiated by user if a “soft” 404 is suspected)• Discovers copy of missing page in WI (1) and provides to user (2)• Generates a search engine query based on what the missing page is “about” (3)• Finds old content at new URI or provides a “good enough” alternative page (4,5,6)• If an “old” copy can be found:– Lexical Signatures– <title>…</title>• If no archived/cached copy:– Tags– Link Neighborhoods; LSs, anchor tagsWhat Was That Web Page About??ABCGET https://user:pass@api.del.icio.us/v1/posts/suggest?url=http://yahoo.com/<?xml version="1.0" encoding="UTF-8"?><suggest><popular>web</popular><popular>tools</popular><popular>searchengines</popular><recommended>yahoo!</recommended><recommended>yahoo</recommended><recommended>web</recommended><recommended>tools</recommended><recommended>search</recommended><recommended>reference</recommended><recommended>portal</recommended><recommended>news</recommended></suggest>What is a Signature?(aka “message digest”, examples include “md5” and “sha-1”)image from Eddie Kohler http://www.cs.ucla.edu/~kohler/What is a Lexical Signature?• First introduced by Phelps and Wilensky [Phelps00]• Small set of terms capturing the “aboutness” of a document• Phelps and Wilensky assumed 5• “lightweight metadata”“Removal Policies in Network Caches for World-Wide Web Documents”QueryGoogleResourceAbstract REMOVALHITRATEPROXYCACHELS• “Robust Hyperlink Cost Five Words Each”• Append LS to URL:21LSs as Proposed by Phelps and Wilensky• Limitations:1. Applications (browsers) need to be modified to exploit LSs2. LSs need to be computed a priori3. Works well with most URLs but not with all of themhttp://www.cs.berkeley.edu/~wilensky/NLP.htmlbecomes:http://www.cs.berkeley.edu/~wilensky/NLP.html?lexical-signature=texttiling+wilensky+disambiguation+subtopic+iagoRank/Results URL LS1/1 http://www.cs.berkeley.edu/˜wilensky/NLP.htmltexttiling wilensky disambiguation subtopic iagohttp://www.google.com/search?q=texttiling+wilensky+disambiguation+subtopic+iago1/221,000(1/174,000 in 01/2008)http://www.loc.gov library collections congress thomas americanhttp://www.google.com/search?q=library+collections+congress+thomas+american1/51(2/77 in 01/2008)http://www.jcdl2008.org libraries jcdl digital conference psthttp://www.google.com/search?q=libraries+jcdl+digital+conference+pst0/10 http://www.dli2.nsf.gov nsdl multiagency imls testbeds extramuralhttp://www.google.com/search?q=nsdl+multiagency+imls+testbeds+extramural22Lexical Signatures -- ExamplesA “Googlewhack” (http://en.wikipedia.org/wiki/Googlewhack) can be thought of as a two-term LS that produces a 1/1 ranking.Generating LSs• Term Frequency (TF)• “How often does this term occur in this document?”• Inverse Document Frequency (IDF)• “In how many documents does this term appear?”Generating LSs• Park et al. [Park03] investigated performance of various LS generation algorithms• Evaluated “tunability” of TF and IDF• Weight on TF increases recall (completeness, ex. “photography, blog”)• Weight on IDF improves precision (exactness, ex. “nicnichols, penitentiary”)• Computed IDF on closed system (not live web)• Also assumed “5” to be a good number• Compared results after 6 months, but did not do an in-depth analysis of LSs over timeTheoretical Underpinnings of Synchronicity• Estimating IDF values for the Web (WIDM 2008, ECIR 2009)• Investigated how lexical signatures change over time (ECDL 2008)• Compared retrieval performance of lexical signatures with titles, tags and lexical signatures generated from link neighborhoods (submitted)• Investigated how titles change over time (InDP 2009, in preparation)Hacks for Estimating IDF1. everyone knows this value is flaky2.    get N from: http://www.worldwidewebsize.com/URL: http://www.perfect10wines.comYear: 2007Union: 12 unique termsFor LS purposes, it doesn’t matter much…Comparing LSsTop 5, 10 and 15 termsLC – local universe SC – screen scraping NG – N-Grams~4 of 5 LS terms are the sameHow Does Google N-grams TC Relate to DF?• Google N-grams has only Term Count (TC), not Document Frequency– where TC >= DF– http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html• Idea: compare TC & DF in a known collection, then compare that collection’s TC to the Google N-grams TC– we used ukWaC, from WaCKy: http://wacky.sslmit.unibo.it/TC Ranks vs DF Ranks Within ukWaCRank Correlation Within ukWaCsemi-log scaleTC Frequencies in ukWaC and N-GramsN-Grams have a threshold of 200LS Evolution Over TimeCopies of web pages from the IA (1996-2007)300 Random URLs, winnowed to 98, 10493 observations over 12 yearsEvolution Over Time -- Example10-term LSs generated forhttp://www.perfect10wines.comTwo Methods for Measuring EvolutionIdea• Generate LSs from copies of URLs• Conduct overlap analysisRooted SlidingEvolution Over Time - Rooted• Little overlap between the early years and more recent ones• Highest overlap in the first 1-2 years after creation of the LSs• Rarely peaks after that – once terms are gone they do not returnEvolution Over Time - Sliding• Overlap increases over time• Seem to reach steady state around 2003Performance of LSsIdea• Measure performance in respect to age of LS and number of terms it contains• Query Google search API with LSs• Identify URL in result set:1. Top ranked2. Ranked between 2-103. Ranked between 11-1004. Ranked beyond 100 (considered undiscovered)Performance – Number of Terms• 2-, 3- and 4-term LSs perform poorly• 5-, 6- and 7-term LSs seem best• Top mean rank (MR) value with 5 terms• Most top ranked with 7 terms• Binary pattern: either top 10 or undiscovered• 8+ terms -- decreased performanceFair Optimistic40The Bigger PictureP rformance – AgScore of LSs consisting of 2, 5, 7 and 10 terms•Example, scores for the position of an URL in a list of 10:• fair: 10/10, 9/10, 8/10 ... 1/10, 0• optimistic: 1/1, 1/2, 1/3 ... 1/10, 0Titles (TI), 5- & 7-term Lexical Signatures (LS5, LS7), Tags (TA)500 random URLs from dmoz.org winnowed to 309 (only 47 of 309 had tags in delicious.com).Due to query restrictions, link neighborhood only run on Yahoo -- results were similar to tags.Frequency of Changeordered in increasing order by:1) observations2) changesNumber of Title Changes and Observations in the IA• generally low number of changes• max changes: 25• number of observations does not impact the number of changes6000 random URLs from dmoz.org, winnowed to 1090 URLs and 100k+ observationsTimes of ChangeMean Time Delta Between ChangesTime Span Between First and Last Observation in the IAordered in increasing order by:1) observations2) changes• time span between observations decreases with increasing number of observations• overall time span just slightly increases• URLs with many observations are being crawled frequently in a short period of timeDegree of ChangeMean Levenshtein Scores of all Titles - Sliding• 5 URLs with score = 0• 85% of URLs with score >=0.8• titles rarely change drasticallyDegree of ChangeMean Levenshtein Scores of all Titles - Rooted• 9 URLs with score = 0• 56% of URLs with score >=0.8• titles more likely to change compared to their first observationhttp://www.sun.com/solutionsmean Levenshtein score sliding: 0.84 rooted: 0.291998-01-27Sun Software Products Selector Guides -Solutions Tree1999-02-20Sun Software Solutions2002-02-01Sun Microsystems Products2002-06-01Sun Microsystems - Business & Industry Solutions2003-08-01Sun Microsystems - Industry & Infrastructure Solutions2004-02-02Sun Microsystems - Solutions2004-06-10Gateway Page - Sun Solutions2006-01-09Sun Microsystems Solutions & Services2007-01-03Services & Solutions 2007-02-07Sun Services & Solutions2008-01-19Sun Solutionshttp://www.datacity.com/mainf.htmlmean Levenshtein score sliding: 0.68 rooted: 0.152000-06-19DataCity of Manassas Park Main Page2000-10-12 DataCity of Manassas Park sells CustomBuilt Computers & Removable Hard Drives2001-08-21 DataCity a computer company in ManassasPark sells Custom Built Computers & RemovableHard Drives  2002-10-16computer company in Manassas Virginia sellsCustom Built Computers with Removable HardDrives Kits and Iomega 2GB Jaz Drives(jazz drives) October 2002 DataCity800-326-5051 toll free2006-03-14 Est 1989 Computer company in StaffordVirginia sells Custom Built SecureComputers with DoD 5200.1-R ApprovedRemovable Hard Drives, Hard Drive Kitsand Iomega 2GB Jaz Drives (jazz drives),introduces the IllumiNite&reg; lightedkeyboard DataCity 800-326-5051 ServiceDisabled Veteran Owned Business SDVOBContent Change vs. Title ChangeConclusions & Future Work• LSs decay over time, Titles decay less• Rooted: quickly after generation• Sliding: seem to stabilize• Titles give comparable performance to LSs• Titles + LSs give better performance• Future work:• can we know in advance if a title is “good”?  (i.e., not “welcome to my home page”)• can we use tags to augment titles / LS?• how big should a link neighborhood be?• Contact us to get a beta version of the Firefox extension (real soon now!)Necronomicon

(Re-) Discovering Lost Web Pages

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1025&context=computerscience_presentations

(Re-) Discovering Lost Web Pages

Abstract

Similar works

Full text

Available Versions

Old Dominion University