PDF of a powerpoint presentation from the Archive-It Partners Meeting in Montgomery, Alabama, November 18, 2014.  Also available on Slideshare.https://digitalcommons.odu.edu/computerscience_presentations/1032/thumbnail.jp

AlNoamany, Yasmin

Alsum, Ahmed

Brunelle, Justin

Kelly, Mat

Nelson, Michael L.

SalahEldeen, Hany

Weigle, Michele C.

English

Old Dominion University

Old Dominion UniversityODU Digital CommonsComputer Science Presentations Computer Science11-18-2014Tools for Managing the Past WebMichele C. WeigleOld Dominion University, mweigle@odu.eduMichael L. NelsonOld Dominion University, mnelson@odu.eduYasmin AlNoamanyOld Dominion UniversityAhmed AlsumOld Dominion UniversityJustin BrunelleOld Dominion UniversitySee next page for additional authorsFollow this and additional works at: https://digitalcommons.odu.edu/computerscience_presentationsPart of the Archival Science Commons, and the Computer Sciences CommonsThis Book is brought to you for free and open access by the Computer Science at ODU Digital Commons. It has been accepted for inclusion inComputer Science Presentations by an authorized administrator of ODU Digital Commons. For more information, please contactdigitalcommons@odu.edu.Recommended CitationWeigle, Michele C.; Nelson, Michael L.; AlNoamany, Yasmin; Alsum, Ahmed; Brunelle, Justin; Kelly, Mat; and SalahEldeen, Hany,"Tools for Managing the Past Web" (2014). Computer Science Presentations. 30.https://digitalcommons.odu.edu/computerscience_presentations/30AuthorsMichele C. Weigle, Michael L. Nelson, Yasmin AlNoamany, Ahmed Alsum, Justin Brunelle, Mat Kelly, andHany SalahEldeenThis book is available at ODU Digital Commons: https://digitalcommons.odu.edu/computerscience_presentations/30Tools for Managing the Past WebMichele C. WeigleWeb Sciences and Digital Libraries (WS-DL) GroupDepartment of Computer ScienceOld Dominion UniversityNorfolk, VAIncludes joint work with Michael L. Nelson and our PhD students, Yasmin AlNoamany, Ahmed AlSum (PhD 2014), Justin Brunelle, Mat Kelly, Hany SalahEldeenArchive-It Partners MeetingNovember 18, 2014Start-Up and Implementation Grants– WARCreate– WAIL– Mink– Assessing Memento DamageWeb Archiving Incentive– Thumbnail Summarization– Detecting Off-Topic MementosNovember 18, 2014 Archive-It Partners Meeting 2OutlineWARCreate WAIL Minkhttps://ws-dl.cs.odu.edu/SoftwareArchive What I See Now• Standard web archiving tools are difficult for non IT experts.• "Save Page As" is not suitable for archiving purposes.• Pages are behind authentication.• Pages change quickly, but current state needs archiving.Archive-It Partners MeetingNEH Digital Humanities Implementation Grant, 2014-2017, http://bit.ly/odu-dhig-20143November 18, 2014How we're addressing the problem4Kelly and Weigle, "WARCreate - Create Wayback-Consumable WARC Files from Any Webpage", JCDL 2012Kelly, Weigle, and Nelson. "WARCreate - Create Wayback-Consumable WARC Files from Any Webpage," Digital Preservation 2012, Tools Demo SessionNovember 18, 2014 Archive-It Partners MeetingGoogle Chrome extensionArchive the current state of the page in standard Web Archive (WARC) formatCompatible with WaybackWARCreateWARCreate - Work in Progress• New modes of operation– record mode• while activated, add capture of each page visited to the WARC– countdown mode• every interval, refresh and add new capture of page – event mode• add new capture of page every time it dynamically reloads or refreshesNovember 18, 2014 Archive-It Partners Meeting 5• Uploading created WARCs to Archive-It or other archives– consideration of data integrity– merging local WARCs with crawled WARCs • how do we account for your www.facebook.com vs.       my www.facebook.com?– privacyWARCreate - Work in Progress6November 18, 2014 Archive-It Partners MeetingWhat to do with created WARCs?November 18, 2014 Archive-It Partners Meeting 7Kelly, Weigle, and Nelson. "Making Enterprise-Level Archive Tools Accessible for Personal Web Archiving," Personal Digital Archiving 2013, Poster SessionKelly, Nelson, and Weigle. "WARCreate and WAIL: WARC, Wayback and Heritrix Made Easy," Digital Preservation 2013WAILLoad created WARCs into a Wayback instance on your local computerSingle-click install of Wayback(and other archiving tools)Includes IIPC's OpenWayback 2.0 and Heritrix 3.2Available for Windows, OS X (Linux coming soon!)WAIL - Work in Progress• More tools– integration with Ilya Kreymer's pywb• User interface enhancements– ease of installation– intuitive GUI– configuration of Wayback display and HeritrixcrawlsNovember 18, 2014 Archive-It Partners Meeting 8Bridging the gap between the past web and the live webNovember 18, 2014 Archive-It Partners MeetingGoogle Chrome extensionFor each page you visit, displays the number of archived versions availableProvides access by dateAllows for submission to public archiving servicesMink9Kelly, Nelson and Weigle, "Mink: Integrating the Live and Archived Web Viewing Experience Using Web Browsers and Memento," poster, ACM/IEEE Digital Libraries (DL), September 2014.Mink - Work in Progress• Pick public archives (Memento Aggregator) or private archive (local computer)November 18, 2014 Archive-It Partners Meeting 10ToolsArchive-It Partners MeetingWARCreateMinkWAIL11https://ws-dl.cs.odu.edu/SoftwareNovember 18, 2014Start-Up and Implementation Grants– WARCreate– WAIL– Mink– Assessing Memento DamageWeb Archiving Incentive– Thumbnail Summarization– Detecting Off-Topic MementosNovember 18, 2014 Archive-It Partners Meeting 12OutlineWAIL Minkhttps://ws-dl.cs.odu.edu/SoftwareWARCreateHow damaged are these mementos?Archive-It Partners MeetingM = 0.17D = 0.09(live web)M = 0.24D = 0.41(missing main) M = 0.29D = 0.36(missing logo + navigation)Brunelle, Kelly, SalahEldeen, Weigle, and Nelson, "Not All Mementos Are Created Equal: Measuring the Impact of Missing Resources", IEEE/ACM Digital Libraries (DL) 2014, Best Student Paper13November 18, 2014M = percentage missingD = our damage metricArchive-It Partners MeetingGood News: Although M is steady/increasing, D is decreasingNovember 18, 2014 14M = percentage missingD = our damage metricSampled 45,000 URI-Ms- one URI-M each year of ~1850 URI-Rs- URI-Rs from Bitly URIs shared over Twitter and Archive-It collectionsStart-Up and Implementation Grants– WARCreate– WAIL– Mink– Assessing Memento DamageWeb Archiving Incentive– Thumbnail Summarization– Detecting Off-Topic MementosNovember 18, 2014 Archive-It Partners Meeting 15OutlineWAIL Minkhttps://ws-dl.cs.odu.edu/SoftwareWARCreateBrowsing TimeMapsNovember 18, 2014 Archive-It Partners Meeting 16How were these 4 thumbnails chosen?Which tells you more about the past of www.apple.com?Archive-It Partners MeetingNovember 18, 2014700 thumbnails(not even all of them!)32 sampled thumbnails17AlSum and Nelson, "Thumbnail Summarization Techniques for Web Archives", ECIR 2014Thumbnail Summarization• Process– compare HTML of consecutive mementos• more efficient than image diff– when diff threshold passed, generate thumbnail– return data + thumbnail as JSON• Considerations– diff threshold too low -> near duplicate images– diff threshold too high -> miss important changes• Work in Progress– wayback plugin– embeddable versionNovember 18, 2014 Archive-It Partners Meeting 18Thumbnail Summary ScreencastArchive-It Partners MeetingNovember 18, 2014 19Start-Up and Implementation Grants– WARCreate– WAIL– Mink– Assessing Memento DamageWeb Archiving Incentive– Thumbnail Summarization– Detecting Off-Topic MementosNovember 18, 2014 Archive-It Partners Meeting 20OutlineWAIL Minkhttps://ws-dl.cs.odu.edu/SoftwareWARCreateHave you ever had this problem?November 18, 2014 Archive-It Partners Meeting 21May 21, 2012nothing but spamMay 16, 2013Detecting Off-Topic Mementos• Goal: Build a tool to alert curators of potential off-topic mementos in a collection• Compare text of mementos– Intersection of top terms (TF)– Cosine similarity– Jaccard similarity coefficient– Clustering with topic modelingNovember 18, 2014 Archive-It Partners Meeting 22Test CollectionsNovember 18, 2014 Archive-It Partners Meeting 23Turns out to be rather difficult• Egyptian Revolution– lots of non-English pages• Occupy Movement– lots of Facebook and social media pages– template extractors have trouble with these• Boston Marathon BombingNovember 18, 2014 Archive-It Partners Meeting 24but we're making progress(stay tuned!)Storytelling For ArchivesArchived collectionsStorytelling servicesArchived enriched storiesNovember 18, 2014 Archive-It Partners Meeting 25AlNoamany, "Using Web Archives to Enrich the Live Web Experience Through Storytelling", TCDL Bulletin, December 2013.Story TypesFixed Page – Fixed Time: differences in GeoIP, mobile, etc.Fixed Page – Sliding Time: evolution of a single page (or domain) through timeSliding Page – Fixed Time: different perspectives on a point in timeSliding Page – Sliding Time: broadest possible coverage of a collectionsameTimedifferentURIsamedifferentIssues: topic modeling, eliminating duplicates, maximizing novelty, structural & content qualityNovember 18, 2014 Archive-It Partners Meeting 26Tools for Storytelling• Tools for Curators– create stories from your collections• candidate mementos automatically selected– use existing stories to augment your collections• Tools for Users– use existing tools like Storify to view the stories of a collectionNovember 18, 2014 Archive-It Partners Meeting 27Start-Up and Implementation Grants– WARCreate– WAIL– Mink– Assessing Memento DamageWeb Archiving Incentive– Thumbnail Summarization– Detecting Off-Topic MementosNovember 18, 2014 Archive-It Partners Meeting 28Tools for Managing the Past WebWAIL Minkhttps://ws-dl.cs.odu.edu/SoftwareMichele C. Weiglemweigle@cs.odu.edu@weiglemchttp://www.cs.odu.edu/~mweigle/Web Science and Digital Libraries (WS-DL) Group@WebSciDLhttp://ws-dl.cs.odu.edu/http://ws-dl.blogspot.com/WARCreate

Tools for Managing the Past Web

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1032&amp;context=computerscience_presentations

Tools for Managing the Past Web

Abstract

Similar works

Full text

Available Versions

Old Dominion University