Brügger, Niels

Laursen, Ditte

Nielsen, Janne

English

Ministry of Culture Research Portal

Looking Back, Looking Forward: New Strategies for Coverage of  a National Web Sphere By  Eld Zierau The Royal Library of Denmark  Ditte Laursen The State and University Library of Denmark   2 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland                         011001        010010101110110    101100110011100 0110101010100010  1011110110110 11010010100110 100100010010 110100               010 0010                     00  010                                                      101                                          0110101  001001     01000010                                    1011011    0100101000101110                                 100001011111111001111000111                                                                                                     100001011111111001111000111                            1001010101100110010010011100111                         1010100010101001100100100      1001                       0100010101001100100    100         0010                    101101101101001010                001                  0110100001001010000             10111                100010110010101011001         10010010              1010100010101001100100100110011110            11000101010011001001001100100111100           10101000101010011001001001100101010001          10101000101010011001001001100101010001         101010011001001001101010001010100110        01001101010001010100110010101  011       0010101001100101010001010100      001010100010101001101100101     00101010001010101000101010     001010100010101101010101       001010100010101     0101         0010101000                 00         00 10101000101         1               001010               001010   1011                       11001010                           10100010                                        110                                         1100111010                                          0  0010011101101                                             10011001000110101                                          01010001010111001011                                       1011011011010010100010                                        01101000010010100001011111                                         100010110010101011001100100100                                         1010100010101001100100100110011                                         101010001010100110010010011001                                          10101000101010011001001001100                                                10101000101010011001001001                                                       10101000101010011001001                                                           101010001010100110010                                                           1010100010101001100                                                             1010100010101001100                                                                                                                                                     101010001010100                                                                         0101000101                                                                                     0101                                             01                                            11001                                             010010                                               10111011                                              1011001100                                              111000110101                                            01010001010111                                      101101101101001010                                      011010   010001001011                             0         01  00     001001011111                       01 01           1          0100010110010                      11  101     101011001100100100101                          1100  01000101010011001001001                           1100100010101001100100100111                             110010001010100110 010   00111                      1100100  010 10100110                 01001                      11001          0     0010101 001100100100                        11                    001   110 1000100100001                       0010011101                                10100101                     100110010001101                    101001010                   0101000101011100101111100111100111              101101101101001010001011100111001111           0110100001001010000101111111100111100        10001011001010101100110010010011100111          1010100010101001100100100110011111001       1010100010101001100100100110011111001         1010001010100110010010011001001111001        1010100010101001100100100110010101000         101010011001001001101010001010100110            0100110101000101010011001010101110                0010101001100101010001010100101                                         0010101000101010011                                              0010101000101010                                             001010100010101101                                            00101010001010101                                           0010101000101001                                           001010100010101                                                 00101010001010                                           00101010001                                            00101010                                            001010                                            0010                                            0  The Web Sphere Danish web sphere Rest of WWW .dk-domænet Outside .dk-domænet increasing amount of national webpages moves to generic Top Level Domains like .com or .org  3 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Contents  A bit of history       The implementation 4 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland .dk and non .dk 38.000 5 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland .dk and non .dk 6 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Top 10 biggest domains 2006   tv2.dk  dbu.dk miniclip.com microsoft.com inforce.dk radioorbit.dk omegn.dk karstenskj.dk um.dk pornaccess.com 2015   snatch.dk cloudfront.net pinterest.com twitter.com google.com vimeocdn.com googleusercontent.com blogspot.com amazonaws.com dmi.dk 7 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland .dk and non .dk 38.000 44.000 91.751 (Zierau 2015) 8 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland WebDanica project Tested Different Methods Internet Archive method NetArchive Link method NL-data Outlinks  from Danish broad crawl 2012 0 Find Danish  webpages IA-data World wide collection 2012 Wide0005 Very few common results NL  results IA  results General implementation  covering more methods Host: 1. part of URL http://abc.xx/def/ghi/... Only in NL 46.552 Only in IA 43.185   Both in IA and NAL 2.014 9 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Implemention – what to do How to implement?  Automate whenever possible  Support web curators work  Support several methods ◦ Similar to NL (based on seed) ◦ Similar to IA (based on existing extracts) ◦ Known Danish outside .dk seeds      10 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Present setup for  harvest and preservation Implementation – the seed washer            Netarkivet   Seed list for  special harvest  (Sub)domains for bulk harvests              Find outside .dk    Seeds to be examined Extracts to be examined Known DK seeds to be included Runs independently of present operation 11 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Seed washer – Seeds method ”Clean” seed list for known or banned seeds List of seeds to be procecessed  Known or banned   seeds /domæne e.g. from • NA outlinks • researchers • .nu .tv … • … Seeds  to be examined         Find outside .dk Seeds list 12 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Seed washer – Seeds method ”Clean” seed list for known or banned seeds Harvest and generate harvest extracts List of seeds to be procecessed  Known or banned   seeds /domæne Harvest extracts Seeds  to be examined         Find outside .dk Seeds list Monitoring: Special harvests  - not under legal legislation  - Not bit preserved 13 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Seed washer – Seeds method ”Clean” seed list for known or banned seeds Harvest and generate harvest extracts List of seeds to be procecessed  Known or banned   seeds /domæne Harvest extracts Seeds  to be examined         Find outside .dk Seeds list e.g. Parsed text  from IA Extracts  to be examined Generated in form of inputting  14 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Seed washer – Seeds method ”Clean” seed list for known or banned seeds List of seeds to be procecessed  Calculate basis for nationa-lity determination of seeds Seeds with nationality related information Harvest extracts Seeds  to be examined         Find outside .dk Seeds list Known or banned   seeds /domæne Extracts  to be examined Harvest and generate harvest extracts Some are only likely Danish  Some are NOT Danish 15 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Seed washer – Seeds method ”Clean” seed list for known or banned seeds Find Danish seeds List of seeds to be procecessed  Found Danish seeds Calculate basis for nationa-lity determination of seeds Seeds with nationality related information Harvest extracts Seeds  to be examined         Find outside .dk Seeds list Known or banned   seeds /domæne Extracts  to be examined Harvest and generate harvest extracts 16 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Seed washer – Seeds method ”Clean” seed list for known or banned seeds Find Danish seeds List of seeds to be procecessed  Found Danish seeds Calculate basis for nationa-lity determination of seeds Seeds with nationality related information Harvest extracts Seeds  to be examined         Find outside .dk Seeds list Known or banned   seeds /domæne Extracts  to be examined Harvest and generate harvest extracts    Known DK  seeds to be   included    e.g. from researchers, manual search, Facebook API, … Generated in form of inputting  17 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Find (sub)domain candidates for bulk harvest  Seed washer – Seeds method ”Clean” seed list for known or banned seeds Find Danish (sub)domains   Seed list for  special harvests  (Sub)domains for bulk harvests List of seeds to be procecessed  Create seed list for special harvest Found Danish seeds Calculate basis for nationa-lity determination of seeds Seeds with nationality related information Harvest extracts Seeds  to be examined         Find outside .dk Seeds list List of bulk harvest (sub)domains candidates Known or banned   seeds /domæne Extracts  to be examined    Known DK  seeds to be   included    Harvest and generate harvest extracts Find Danish seeds Manual task: 18 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Find (sub)domain candidates for bulk harvest  Seed washer – Seeds method ”Clean” seed list for known or banned seeds Find Danish (sub)domains   Seed list for  special harvests  (Sub)domains for bulk harvests Harvest and generate harvest extracts Find Danish seeds List of seeds to be procecessed  Create seed list for special harvest Found Danish seeds Calculate basis for nationa-lity determination of seeds Seeds with nationality related information Harvest extracts Seeds  to be examined         Find outside .dk Seeds list List of bulk harvest (sub)domains candidates Known or banned   seeds /domæne Extracts  to be examined    Known DK  seeds to be   included    Minimize time gap between harvests 19 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland Assumptions  It is possible to minimize the harvest gap sufficiently  There are no legal issues in harvesting outside the .dk top level domain  It is acceptable  ◦ That we lose material The only included seeds, are the ones where there are about 90% probability of being relevant for Danish heritage ◦ That we include some noise: About 10% of the included seeds are not relevant for Danish heritage    20 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland The way forward …  Implementations in 2016 – just started        Expects to evaluate and adjust after 2-3 years Netarkivet 21 Looking Back, Looking Forward: New Strategies for Coverage of a National Web Sphere IIPC 2016, Reykjavik, Iceland    Questions Images of this style from digitalbevaring.dk 

A nation’s web over time: .dk registry – DK Archive – Internet Archive

https://pure.kb.dk/ws/files/9063729/WAC05_KATLAII_Eld_Zierau_Ditte_Laursen.pdf

A nation’s web over time: .dk registry – DK Archive – Internet Archive

Abstract

Similar works

Full text

Available Versions

Ministry of Culture Research Portal