The Internet Archive\u27s Wayback Machine is the most common way that typical users interact with web archives. The Internet Archive uses the Heritrix web crawler to transform pages on the publicly available web into Web ARChive (WARC) files, which can then be accessed using the Wayback Machine. Because Heritrix can only access the publicly available web, many personal pages (e.g. password-protected pages, social media pages) cannot be easily archived into the standard WARC format. We have created a Google Chrome extension, WARCreate, that allows a user to create a WARC file from any webpage. Using this tool, content that might have been otherwise lost in time can be archived in a standard format by any user. This tool provides a way for casual users to easily create archives of personal online content. This is one of the first steps in resolving issues of  long term storage, maintenance, and access of personal digital assets that have emotional, intellectual, and historical value to individuals 

Kelly, Mat

Weigle, Michele C.

English

Old Dominion University

Old Dominion University ODU Digital Commons Computer Science Faculty Publications Computer Science 2012 WARCreate: Create Wayback-Consumable WARC Files From Any Webpage Mat Kelly Old Dominion University Michele C. Weigle Old Dominion University Follow this and additional works at: https://digitalcommons.odu.edu/computerscience_fac_pubs  Part of the Computer Sciences Commons Original Publication Citation Kelly, M., & Weigle, M. C. (2012). WARCreate: Create wayback-consumable WARC files from any webpage. Paper presented at the 12th ACM/IEEE-CS Joint Conference on Digital Libraries, Washington, DC. This Conference Paper is brought to you for free and open access by the Computer Science at ODU Digital Commons. It has been accepted for inclusion in Computer Science Faculty Publications by an authorized administrator of ODU Digital Commons. For more information, please contact digitalcommons@odu.edu. WARCreate - Create Wayback-Consumable WARC Filesfrom Any WebpageMat KellyDepartment of Computer ScienceOld Dominion UniversityNorfolk, Virginiamkelly@cs.odu.eduMichele C. WeigleDepartment of Computer ScienceOld Dominion UniversityNorfolk, Virginiamweigle@cs.odu.eduABSTRACTThe Internet Archive’s Wayback Machine is the most com-mon way that typical users interact with web archives. TheInternet Archive uses the Heritrix web crawler to trans-form pages on the publicly available web into Web ARChive(WARC) files, which can then be accessed using the Way-back Machine. Because Heritrix can only access the pub-licly available web, many personal pages (e.g., password-protected pages, social media pages) cannot be easily archivedinto the standard WARC format. We have created a GoogleChrome extension, WARCreate, that allows a user to createa WARC file from any webpage. Using this tool, contentthat might have been otherwise lost in time can be archivedin a standard format by any user. This tool provides a wayfor casual users to easily create archives of personal onlinecontent. This is one of the first steps in resolving issues of“long term storage, maintenance, and access of personal dig-ital assets that have emotional, intellectual, and historicalvalue to individuals” [3].Categories and Subject DescriptorsH.3.4 [Information Storage and Retrieval]: Systemsand Software; H.3.7 [Digital Libraries]: Personal WebArchivingGeneral TermsDesignKeywordsPersonal Web Archiving, WARC, Browser, Wayback Ma-chine, Internet Archive1. INTRODUCTIONThe Internet Archive, along with web archives at other li-braries and institutions, has done a remarkable job at archiv-ing the public web. But in recent years, the web has becomea home for a significant amount of original user-generatedcontent, such as that posted on social media sites. Usersare becoming increasingly aware of the need for personalweb archiving [4, 5]. Unfortunately, this content is largelyunavailable to standard web archives because it lives behindthe“walled garden”of authentication and is part of the“deepCopyright is held by the author/owner(s).JCDL’12, June 10–14, 2012, Washington, DC, USA.ACM 978-1-4503-1154-0/12/06.web” [1]. Our goal is to allow users, once past authentica-tion, to generate their own archives that can be browse-ablein a user-friendly manner.The Internet Archive’s Wayback Machine is the most well-known interface for accessing web archives. The archivedpages are stored in the standard Web ARChive (WARC) for-mat [2] and are generated by the Heritrix1 crawler. Unfortu-nately, Heritrix is limited to crawling only publicly accessi-ble pages, so many personal pages (e.g., password-protectedpages, social media pages) cannot be easily archived. In ad-dition, for pages that are location or user-agent aware, theversion archived at Internet Archive is the one that the Her-itrix crawler (run from San Francisco) sees. For example, themost recently available version2 of http://www.craigslist.orgredirects to http://sfbay.craigslist.org.In an effort to facilitate the use of the standard WARCformat for personal web archives, we have developed a toolto allow a user to archive any page, edit its metadata, andsubmit it to an instance of the Wayback Machine (from hereon referred to as Wayback).2. WARCREATEWARCreate3 is an extension for the Google Chrome webbrowser that allows a user to generate a WARC file from thecurrent webpage. In addition to creating a valid WARC thatcan be viewed in Wayback, the extension provides optionsthat address privacy concerns (e.g., a user might want thedata encrypted in some way), potential bleed over (e.g., twodifferent users see different content at http://facebook.com),and other issues that may not be relevant to conventionalweb archiving as performed by Heritrix and the InternetArchive.To create a WARC file from the current webpage, the userclicks on the browser extension’s icon in the address bar andthen presses the Generate WARC button (see Figure 1). Thebrowser extension gathers the resources (including externalscripts, CSS and images) and HTTP headers normally usedby the web browser to generate a webpage and adds meta-data (the warcinfo records) to generate a WARC file thatconforms to the standard’s specification (Figure 2). Adher-ence to the specification allows the WARC to be read byWayback.When the compilation of the WARC file is complete, thefile is downloaded to the local file system. The browser ex-1https://webarchive.jira.com/wiki/display/Heritrix/Heritrix2Archived on July 25, 20113http://matkelly.com/warcreate437Figure 1: Creating a WARC is as simple as select-ing the extension’s icon and pressing the GenerateWARC button. Other options can be selected forfurther customization of the generated file.tension also allows a user to open an existing WARC filegenerated by the tool to make modifications. Any changesto the WARC file must be accounted for in the content-lengthWARC fields to ensure compatibility with the WARC stan-dard. This overcomes the potential corruption of the WARCrecords that might occur if the file is edited with a simpletext editor. Further functionality is being developed to allowthe extension to read in, interpret and allow modification ofany arbitrary WARC file. This would require comprehensiveimplementation of the WARC specification.Unlike a web crawler, this tool archives only the singlewebpage that the user is visiting. In the future, we intendto add more of a crawling-like functionality. For now, thecurrent extension can append the archive of the current pageto an existing WARC file. This allows links to content fromseparate archiving sessions to be resolvable by Wayback.Figure 2: A generated WARC file is prefixed witha warcinfo record that conforms to a liberally at-tributed template while still maintaining minimumrequirements.3. OTHER CONSIDERATIONSThere are many facets of this type of personal web archiv-ing that we leave for future exploration. The main issues wewill investigate are ensuring privacy, maintaining archive in-tegrity, preventing bleed over, and developing a service toconsume the potentially private WARCs generated by theextension.3.1 Privacy ConcernsMuch of the data in personal web archives is of a sensitiveor personal nature, thus the data placed in a WARC file mustbe protected. One way to achieve this is to encrypt the dataso that only the recipient can interpret the archive. Thereare several issues to consider, such as ease of decryption,decentralization, increased space and time complexity, andloss of access when a central authority disappears. In thecurrent Chrome extension, we have implemented a simplekey-based encryption/decryption scheme for users that wishto test the reliability of this preliminary technique.3.2 Archive IntegrityHeritrix subtly manipulates content in an archive for easeof navigation and to allow a webpage to be viewed in themodified context of the Wayback Machine user interface.Because data is manipulated, the integrity of the archiveis compromised, as the page is not retained exactly in itsoriginal form. In future work, we hope to expand on theWARC format (and demonstrate this with an implementa-tion) to provide WARC records containing the differentialof how the webpage was manipulated. By utilizing this dif-ferential, the original content can be re-created much in thesame way that software versioning uses differentials to storeand revert to older versions of code.3.3 Bleed OverAn issue that is not as common on the surface web as onthe personalized deep web is that of a single URI referringto potentially different information. This can be seen whentwo users access a social media website while each is loggedin (e.g., http://www.facebook.com). The website uses au-thentication to provide customized information to each user.If two users were to create an archive of the same page, theresulting WARCs should be different and each user shouldonly be able to access the WARC they created. This bleedover issue will be addressed in future work.3.4 Service to Consume WARCsMaking WARC files accessible to the end user is just thefirst step. Unless a user has a personally deployed instance ofWayback, little can be done with the resulting file. We willinvestigate the feasibility of rapid deployment of a personalinstance of Wayback or of finding an alternative for quickreplay of small archives that were created with the Chromeextension. Initial approaches to accomplish this will attemptto decouple the code used to interpret WARC files from theopen source Wayback and minimize the need for reinventionof these procedures while maintaining consistency of imple-mentation with future versions of Wayback.4. REFERENCES[1] M. K. Bergman. The deep web: Surfacing the hiddenvalue. Journal of Electronic Publishing, 2000.[2] ISO. Information and documentation - WARC fileformat, 2009.[3] C. C. Marshall. Rethinking Personal Digital Archiving,Part 1: Four Challenges from the Field. DLibMagazine, 14(3/4):2, 2008.[4] C. C. Marshall, S. A. Bly, and F. Brun-Cottan. Thelong term fate of our digital belongings: Toward aservice model for personal archives. In Proceedings ofIS&T Archiving, pages 25–30, 2006.[5] C. C. Marshall, F. McCown, and M. L. Nelson.Evaluating personal archiving strategies forinternet-based information. In Proceedings of IS&TArchiving, pages 151–156, 2007.4381 WARC/1. 0 [ Generate WA~ Other options 2 WARC-Type: warcinfo 3 WARC-Date: 2012 01-18T22:12:49.445Z all. 9 Software: WARCreate/ 0. 4 .1 http://matkelly.com/warcreate 10 128.82.5.133 11 cs. odu. edu 12 format: WARC File Format 1. 0 13 conforms To: http:/ /bibnum. bnf. fr/ WARC/ WARC _ ISO _ 28500 _ versionl _ latestdraft. pdf 

WARCreate: Create Wayback-Consumable WARC Files From Any Webpage

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1154&context=computerscience_fac_pubs

WARCreate: Create Wayback-Consumable WARC Files From Any Webpage

Abstract

Similar works

Full text

Available Versions

Old Dominion University