[First paragraph] Preservation of the Web ensures that future generations have a picture of how the web was. Web archives like Internet Archive\u27s Wayback Machine, WebCite, and archive.is allow individuals to submit URIs to be archived, but the captures they preserve then reside at the archives. Traversing these captures in time as preserved by multiple archive sources (using Memento [8]) provides a more comprehensive picture of the past Web than relying on a single archive. Some content on the Web, such as content behind authentication, may be unsuitable or inaccessible for preservation by these organizations. Furthermore, this content may be inappropriate for the organizations to preserve due to reasons of privacy or exposure of personally identifiable information [4]. However, preserving this content would ensure an even-more comprehensive picture of the web and may be useful for future historians who wish to analyze content beyond the capability or suitability of archives created to preserve the public Web

Alam, Sawood

Kelly, Mat

Nelson, Michael L.

Weigle, Michele C.

English

Old Dominion University

Old Dominion University ODU Digital Commons Computer Science Faculty Publications Computer Science 2018 Client-Assisted Memento Aggregation Using The Prefer Header Mat Kelly Old Dominion University Sawood Alam Old Dominion University Michael L. Nelson Old Dominion University Michele C. Weigle Old Dominion University Follow this and additional works at: https://digitalcommons.odu.edu/computerscience_fac_pubs  Part of the Archival Science Commons, and the Databases and Information Systems Commons Original Publication Citation Kelly, M., Alam, S., Nelson, M. L., & Weigle, M. C. (2018) Client-assisted memento aggregation using the prefer header [Conference Paper]. Web Archiving & Digital Libraries Workshop, Fort Worth, Texas. http://hdl.handle.net/10919/97988 This Conference Paper is brought to you for free and open access by the Computer Science at ODU Digital Commons. It has been accepted for inclusion in Computer Science Faculty Publications by an authorized administrator of ODU Digital Commons. For more information, please contact digitalcommons@odu.edu. Client-Assisted Memento Aggregation Using the Prefer HeaderMat Kelly, Sawood Alam, Michael L. Nelson, and Michele C. WeigleOld Dominion UniversityDepartment of Computer ScienceNorfolk, Virginia, USA{mkelly,salam,mln,mweigle}@cs.odu.edu1 INTRODUCTIONPreservation of the Web ensures that future generations have apicture of how the Web was. Web archives like Internet Archive’sWayback Machine1, WebCite2, and archive.is3 allow individuals tosubmit URIs to be archived, but the captures they preserve then re-side at the archives. Traversing these captures in time as preservedby multiple archive sources (using Memento [8]) provides a morecomprehensive picture of the past Web than relying on a singlearchive. Some content on the Web, such as content behind authenti-cation, may be unsuitable or inaccessible for preservation by theseorganizations. Furthermore, this content may be inappropriate forthe organizations to preserve due to reasons of privacy or exposureof personally identifiable information [4]. However, preserving thiscontent would ensure an even-more comprehensive picture of theWeb and may be useful for future historians who wish to analyzecontent beyond the capability or suitability of archives created topreserve the public Web.State-of-the-art Memento aggregators relay requests to a “static”set of archives. Thus, a client requesting an aggregated TimeMaphas no say in which Web archives are used as the sources ({A0}). Byleveraging our previous work [4] of supplementing the capabilityof Memento aggregators (e.g., adding query precedence, aggrega-tion short-circuiting, and multi-dimensional content negotiationof TimeMaps), we reuse this functionality for a more standards-based approach. This approach provides the novel contribution ofinvolving the client’s request in the Memento aggregation processbeyond the specification of a URI-R and datetime.More sophisticated aggregationmay require filtering on amemento-level (e.g., only source mementos from archives with a certain qual-ity of capture) or on a TimeMap-level. For instance, a user maywish to provide a previous unaggregated public archive (e.g., the“Freedonia Web Archive” in Figure 1b) or a private/personal Webarchive as an additional source for aggregation. A conventionalMemento aggregator may be required to provide additional param-eters or communication flows to obtain mementos for a URI-R fromprivate Web archives (as we discuss more in-depth in our precedingwork [4]). In the current operation, a Memento aggregator assumesthat all archives in a set are willing to provide a TimeMap in all in-stances. This may not be the case for a client’s personal archive or apublic Web archive that is not currently included in the aggregatedset.1https://web.archive.org/2http://www.webcitation.org/3http://archive.is/WADL’18, June 2018, Fort Worth, Texas USA2018. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00https://doi.org/10.1145/nnnnnnn.nnnnnnnThis submission represents a preliminary investigation in allow-ing the clients of Memento aggregators to be involved in determin-ing the set of archives aggregated. In this work, we leverage theHTTP Prefer header [7]. Previous discussions have revolved aroundusing Prefer for memento-level negotiation [5, 9]. This work consid-ers using Prefer for TimeMap-level aggregation, particularly for theset of archives via archive specification instead of the representationof an individual memento.2 BACKGROUND AND RELATEDWORKMemGator [2], the open-source Memento aggregator, provides con-ventional Memento aggregation with extended features includingadditional support for TimeMap formats beyond Link [6] and cus-tomization of the set of archives on startup of the aggregator soft-ware. CDXJ [1] is one such TimeMap format that is leveraged bythe TimeMap endpoints in MemGator. Originally created as a re-placement for CDX4 files that act as an index to WARC [3] files,the CDXJ format allows for additional attributes about memen-tos to be specified within a JSON block. This capability allows forCDXJ-formatted TimeMaps to be much richer than Link-formattedTimeMaps due to the extensible semantics.(a) Client requests archives list from aggregator(b) Client supplies own list, potentially with custom attributesFigure 1: A client first requests a list of aggregated archivesfrom a Memento aggregator then modifies the response, en-codes it, and supplies the encoded archive specification us-ing the Prefer header for the aggregator to process.Previously [4], we introduced the “Memento Meta-Aggregator”(MMA) concept to supplement functionality to conventional Me-mento aggregators using a hierarchical approach. There, we alsointroduced a rudimentary approach for a client to specify additional4https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2015/WADL’18, June 2018, Fort Worth, Texas USA Mat Kelly, Sawood Alam, Michael L. Nelson, and Michele C. Weigle> GET /timemap/link/http ://fox.cs.vt.edu/wadl2017.html HTTP /1.1> Host: mma.cs.odu.edu> Prefer: archives ="data:application/json;charset=utf -8;base64 ,Ww0KICB7 ... NCn0="< HTTP /1.1 200< content -type: application/link -format< vary: prefer< preference -applied: return=representation; archives ="data:application/json;charset=utf -8;base64 ,Ww0KICB7 ... NCn0="< content -location: /timemap/link/5bd...8e9/http ://fox.cs.vt.edu/wadl2017.htmlFigure 2: Client-side specification of a set of archives via encoded JSON using HTTP Prefer. TheMemento aggregator respondswith a location of a TimeMap for the URI-R at a URI-T representative of the set.archives to an MMA using an ad hoc X-Archives HTTP requestheader. We also explored utilizing the Prefer [7] HTTP header toaccomplish negotiation of mementos in dimensions beyond time,as may be facilitated with the usage of CDXJ TimeMaps.Van de Sompel et al. [9] described using the Prefer header to dis-tinguish mementos that have been rewritten when replaying Webarchives to those with an untouched response body. By using Pre-fer header values like original-content and original-headers,a client may request that the representation return not be trans-formed by the Web archive.Various presentations exist for an aggregator to use as the defin-ing a set of archives to be aggregated, inclusive of definitions byMementoWeb.org5, Webrecorder.io, and MemGator6.3 ARCHIVE SET SPECIFICATIONWITHPREFERAn objective of this work is to allow a client of a Memento aggre-gator to be able to specify a custom set of archives ({Af }) to beaggregated using standard syntax and semantics. We anticipatea 3-step process for a client to specify the archive set: (1) Clientrequests the set of archives to be aggregated by default from aPrefer-aware Memento aggregator (Figure 1a). (2) The aggrega-tor returns the set of archives, e.g., as a JSON (per MemGator)or an XML (per mementoweb.org) file (Figure 1a), represented as{A0}. (3) Once a response is received from the aggregator (e.g.,https://git.io/archives), a client may manipulate the contents to beeither an identical set ({Af } = {A0}), subset ({Af } ⊂ {A0}), supple-mentary set ({Af } ⊃ {A0}), or disjoint set ({Af }⋃̇{A0}) (Figure 1b)and submit back to the aggregator for subsequent queries (Figure 2).A client may also manipulate an existing archive’s specificationin the response received. For instance, a profiling probability (avalue already defined in the MemGator specification) may be ma-nipulated or a value of query precedence or short-circuiting maybe modified, both of which we discussed in previous work [4].Given that no Memento aggregator yet supports the client-sidearchive specification, we extend this idea with the assumption thata JSON response is received (like MemGator and Webrecorder’saggregator). A client may perform step 3 using the HTTP Preferrequest header. After potentially manipulating the JSON response,a client would encode the JSON as a base64-encoded data URI (orsupply some other URI for specification-by-reference) and submita request with the Prefer header and a URI-R (Figure 2).Archive supplementation may be accomplished using a hier-archical MMA approach (Figure 3), as we described in previouswork [4]. This approach is necessary to adapt the capability of5http://labs.mementoweb.org/aggregator_config/archivelist.xml6https://git.io/archivesFigure 3: Using a hierarchical MMA approach, a previouslyunaggregated public Web archive may be aggregated withthe results for a URI-R from a conventional Memento aggre-gator.conventional Memento aggregators while still allowing them tobe functionally cohesive. However, this hierarchical approach isinsufficient if the a client would rather that subsequent queries tothe aggregator after step 3 not be sent to certain archives supportedby the base Memento aggregator. With the disclosure of the ag-gregated archives from a conventional aggregator (which is notconventionally exposed), an MMA could configure the default setfrom the conventional aggregator as the default to be queried andsubsume the functions of the conventional aggregator.4 FUTUREWORKIn future work we will explore this approach’s interoperability withusing Prefer on mementos, which is ongoing research. We will alsolook to additional approaches like Cookies, and usage of CoRE’swell-known syntax for archive specification.REFERENCES[1] SawoodAlam. 2015. CDXJ: AnObject Resource Stream Serialization Format. http://ws-dl.blogspot.com/2015/09/2015-09-10-cdxj-object-resource-stream.html. (Sep-tember 2015).[2] Sawood Alam and Michael L. Nelson. 2016. MemGator - A Portable ConcurrentMemento Aggregator: Cross-PlatformCLI and Server Binaries in Go. In Proceedingsof JCDL. 243–244.[3] ISO 28500. 2009. WARC (Web ARChive) file format. http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml. (August 2009).[4] Mat Kelly, Michael L. Nelson, and Michele C. Weigle. 2018. A Framework forAggregating Private and Public Web Archives. In Proceedings of JCDL. Acceptedfor Publication.[5] Ilya Kreymer. 2018. Feedback on new implementation of Prefer header in pywb.https://github.com/mementoweb/rfc-extensions/issues/7. (March 2018).[6] M. Nottingham. 2017. Web Linking. IETF RFC 8288. (October 2017).[7] J. Snell. 2014. Prefer Header for HTTP. IETF RFC 7240. (June 2014).[8] Herbert Van de Sompel, Michael Nelson, and Robert Sanderson. 2013. HTTPFramework for Time-Based Access to Resource States – Memento. IETF RFC 7089.(December 2013).[9] Herbert Van de Sompel, Michael L. Nelson, Lyudmila Balakireva, Martin Klein,ShawnM. Jones, and Harihar Shankar. 2016. Mementos In the Raw, Take Two. http://ws-dl.blogspot.com/2016/08/2016-08-15-mementos-in-raw-take-two.html. (Au-gust 2016).

Client-Assisted Memento Aggregation Using The Prefer Header

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1298&context=computerscience_fac_pubs

Client-Assisted Memento Aggregation Using The Prefer Header

Abstract

Similar works

Full text

Available Versions

Old Dominion University