Peyssard, Jean-Christophe

English

New York University Faculty Digital Archive

Archiving Web Content with ConiferJean-Christophe Peyssard (CNRS - MMSH) WIDH@NYCDH 2021 Wednesday, February 10, 2021 1.Short introduction to Web Archiving2.Conifer demo through a journey to Mars● “The first-ever website (info.cern.ch) was published on August 6, 1991 by British physicist Tim Berners-Lee while at CERN, in Switzerland”● In the January 2021 survey we received responses from 1,197,982,359 sites across 262,949,225 unique domains and 10,649,817 web-facing computers. This reflects a gain of 95,900 computers, but a loss of 30.13 million sites and 465,000 domains. [from previous 22nd December, 2020]https://news.netcraft.com/archives/2021/01/28/january-2021-web-server-survey.html● “The Indexed Web contains at least 5.46 billion pages” (Tuesday, 9 February, 2021, https://www.worldwidewebsize.com/)The World Wide Web by numbers The structure of URLs on the World Wide Web (www):protocol://subdomain.domain.top-level domain/path/page/Ex: https://wp.nyu.edu/widh/widhnycdh-2021/URL : Uniform Resource Locatorhttps://en.wikipedia.org/wiki/URLhttp://dac.au.dk/forskning/forskningsprogrammer/ p. 51● “The average life span of a Web page is only 44 days, and 44 percent of the Web sites found in 1998 could not be found in 1999. […] As ubiquitous as the Web seems to be, it is also ephemeral, and much of today's Web will have disappeared by tomorrow.” (Lyman, 2002 p. 38)● “40% of the material on the Internet disappears within a year, while another 40% has been changed, which is why today we can only expect to find 20% of the material that was on the Internet one year ago.” (Brügger, 2005 p. 15)● “We now know that Web pages only last about 100 days on average before they change or disappear.” (Kahle, 2015)● In 2013, the average life span of a URL is 9.3 years (Musiani at al., 2019, https://books.openedition.org/oep/8743)● In 2019, according to the Wayback Machine team the average life span of a Web page is 92 daysLife and death of  Web pages A problem for research and scholarly communicationWhen the Web is broken● Error 404, Broken links, Link rot, Reference rot, Infosuicide, digital ruins, content drift, zombie media,..● Shut down & take down, mergers and acquisitions:On March 18, 2019, it was revealed that MySpace lost all of their user content from 2016 and earlier in "a server migration gone wrong". It was widely reported that over 50 million songs and 12 years worth of content was permanently lost, and there was no backup (https://en.wikipedia.org/wiki/Myspace)● History : • Yougoslavia (.yu) breakup (now Serbia and Montenegro, .rs and .me) • Czechoslovakia (.cs) dissolution (now Czech Republic and Slovakia, .cz and .sk)● Reference rot, a combination of: • Content decay: The content of the linked resource may change over time and, as a result, the degree to which that content remains representative of the content that was intended to be linked to may decrease over time.● Link rot: The linked resource may disappear altogether. (Thoughts on Referencing, Linking, Reference Rot http://mementoweb.org/missing-link/)● The integrity of research is at risk ! (James G. Neal, http://library.ifla.org/id/eprint/907)● To maintain our digital cultural heritage● To stabilize and preserve web materials as a research object● To be able to document and illustrate claims based on analyses of web materials (whether the web itself is the research object or a source of knowledge about other research objects).Why Web Archiving?Nielsen, J. (2016). Using web archives in research: an introduction. Retrieved from http://www.worldcat.org/oclc/960018046 p. 7● Web archiving is the process of collecting portions of the World Wide Web, preserving the collections in an archival format, and then serving the archives for access and use. (IIPC Web site, http://netpreserve.org/web-archiving/)● “Web archiving is the process of gathering up data that has been recorded on the World Wide Web, storing it, ensuring the data is preserved in an archive, and making the collected data available for future research.” (Niu, J. (2012). An Overview of Web Archiving. D-Lib Magazine, 18 (3/4). https://doi.org/10.1045/march2012-niu1)A definition of Web archiving1537 Legal deposit in France (1619 Spain, 1710 United Kingdom)1989 The World Wide Web was invented by Tim Berners-Lee1996 Internet Archive is founded by Brewster Kahle1996 Kulturarw3 in Sweden for archiving the .se top level domain name1998 Google is launched2001 The Wayback Machine (Internet Archive)2003 UNESCO Charter on the Preservation of Digital Heritage2003 the International Internet Preservation Consortium is formally chartered at the National Library of France with 12 participating institution2005 Youtube is launched2006 [The] Facebook and Twitter are launched2006 National Library of France is in charge of collecting and preserving the “French Internet” (new French Copyright Law) alongside with the National Audiovisual Institute (Ina)2013 EU Web ArchiveA short chronology of Web archives● Macro archiving● Micro archiving● Thematic or selective archiving● Bulk or snapshot harvesting, or broad crawls● “Exhaustivity” of a National domain name archiving (.se in Sweden)● Event and ‘real-time’ institutional archiving (after 2015 terrorist attacks and Notre Dame fire in 2019 France)● Shared archiving among institutions (in France btwn BnF and Ina)● …Different strategies & methods for Web archiving● Finnish Web Archive (since 2006) https://www.kansalliskirjasto.fi/en/collections-and-content-online#finnish-web-archive The contents of the archive can be only accessed from special legal deposit workstations that are available in selected libraries within Finland (including The National Library of Finland).● Portugal (since 2008) accessible in Open Access at Arquivo.pt● The Wayback Machine (since 2006) accessible in Open Access at https://archive.org/web/Different ways of accessing Web archivesAs much as for other kind of archives, one must know the history of a Web archive and how it was constructed to better understand it and use it in research work. What you see is a reconstruction, not a copy of the site● “What is harvested is both a point in time (the time of harvesting) and a period of time (the period up to the time of harvesting).” (Brügger, 2008 p. 158)● “On the one hand the archive does not look like the internet as it actually was in the past (we have lost something), but on the other hand the archive might look like the internet as it never was in the past (we get something different).” (Brügger, 2001 p. 6) Web archiving projects often needs to gather diverse and multiples expertises and skills : archivists and librarians, researchers, legal officers, IT and computer specialists and… users and stakeholdersThere is a lot of different types of Web archivesAs for any other kind of archives one must act lawfully and ethically when archiving and using Web archives:● The materials in the web archives are protected by copyright law as they were on the live web● There is “tensions around the archival principles of preserving the public record vs the individual’s expectation of the right to be forgotten” (http://netpreserve.org/ga2018/programme/abstracts/#paper21)● The processing of personal data is submitted to laws and even more to the research project ethics● New laws to take into account ex. General Data Protection Regulation (GDPR, https://gdpr-info.eu/)Legal and ethical issuesIIPC: International Internet Preservation Consortiumhttp://netpreserve.org/http://resaw.eu/https://web90.hypotheses.org/https://archivesunleashed.org/New research fields and projects● Robots.txt● Captcha (ie Completely Automated Public Turing-test to tell Computers and Humans Apart)● User interaction needed● Password protected content● Technologies and dynamic content : Flash, java scripts,…● Distant content● Temporal inconsistencies● Bot traps● …Challenges and setbacks in Web archiving● Define a strategy● Use a log and write throughout the life of the project● You may need to use additional methods and tools● Screen capture and screen recording● Link crawling● On demand archiving● ...Method and tools for Web archiving● WARC file format = Web ARChive archive format● ARC was accepted as an international standard in 2009 (ISO 28500:2009)● WARC is now recognised by most national library systems as the standard to follow for web archivalA standard format for Web archiveshttps://en.wikipedia.org/wiki/Web_ARChivehttps://wiki.archivematica.org/Significant_characteristics_of_websiteshttps://wiki.archivematica.org/File:WARCdiagram.pnghttps://archive.org/Internet Archive’s Wayback Machine• Launched in 2001• 357 billion archived Web pages so far• Archived content going back to 1996https://conifer.rhizome.org/https://rhizome.org/https://webrecorder.net/https://archiveweb.page/Bibliographyhttps://www.zotero.org/peyssard/collections/CNGX47Z3Thank you / Merci / ًشكرا jean-christophe.peyssard@univ-amu.frWIDH@NYCDH 2021

Archiving Web Content with Conifer

http://archive.nyu.edu/bitstream/2451/62208/2/peyssard-archiving-web-content-conifer%20-%20Jean-Christophe%20Peyssard.pdf

Archiving Web Content with Conifer

Abstract

Similar works

Full text

Available Versions

New York University Faculty Digital Archive