Vermeulen, Brecht

English

Ghent University Academic Bibliography

     Study of bandwidth needs and possible connectivity for a digital archive in Flanders    Studie van bandbreedtenoden en mogelijke connectiviteit binnen Vlaanderen voor een digitaal archief          BOM-VLAANDEREN WP6: Demonstrator – ontwikkeling van een gemeenschappelijk innovatieplatform Deliverable 2 (D.6.2)       March 31th, 2008    BOM-VL D.6.2.  2   1 Table of contents 1 TABLE OF CONTENTS.............................................................................................................. 2 2 INTRODUCTION......................................................................................................................... 3 3 DEMONSTRATOR NETWORK................................................................................................ 4 4 NEEDED BANDWIDTH ............................................................................................................. 5 4.1 HIGH RESOLUTION FILES ........................................................................................................ 5 4.2 BROWSE FORMAT AND KEYFRAMES ....................................................................................... 6 5 PRACTICAL FEEDBACK.......................................................................................................... 7 6 CONCLUSION ............................................................................................................................. 8     BOM-VL D.6.2.  3   2 Introduction  This is the second deliverable of work package 6 (Demonstator – common innovation platform). This deliverable will describe the bandwidth needs and network layout of the demonstrator setup.      BOM-VL D.6.2.  4   3 Demonstrator network As already slightly described in D.6.1, a network was setup when building the demonstrator to link the ingest platforms at VRT (Brussels) and Videohouse (Vilvoorde) to the archive storage at IBBT (Gent). See also Figure 1.  Figure 1: Demonstrator setup     BOM-VL D.6.2.  5   VRTVideohouseVMMaBoekentorenUGentIBBTRegional broadcastersMTN network(20Mb/s)videoserversvideoserversMgmt serverMgmt serverVPN over Belnet (1Gb/s)Ugent net (100Mb/s)BELNET MTN connection 20Mb/s(under construction)FirewallFirewall1Gb/s1Gb/sWDM(1Gb/s)1Gb/sDark fiberothersTemp storageTranscodingDisk storageBrowse/keyframesdatabase Figure 2: Detailed network configuration  Figure 2 illustrates the current BOM-VL demonstrator in more detail. The regional broadcasters have set up in the past (for daily usage) the MTN network amongst them with a bandwidth of 20Mb/s (symmetric). IBBT has now requested to the provider of the network to have also a connection in Gent, so that the link between the central archive at IBBT and the regional broadcasters will be at least 20Mb/s. Connections with the other partners in BOM-VL are more difficult as they are spread around flanders and have lack of high bandwidth connections mostly (because of the high price for commercial bandwidth). Research bandwidth (Belnet) is much cheaper, but the problem is that an organization has to make the physical connection (fiber) to the local Point of Presence (POP) of Belnet. In the following part, some numbers will be shown about needed bandwidth.  4 Needed bandwidth How much bandwidth is needed ? The more the better of course, but we’ll try to show some numbers based on the experience of the demonstrator by now.  4.1 High resolution files For the video files of broadcasters, 3 common formats were agreed upon in WP6. D10 – IMX 50 MXF: 50Mb/s video, about 30GB per hour IMX 30 MXF: 30Mb/s video, about 15 GB per hour DV 25 MXF: 25 Mb/s video, about 15 GB per hour     BOM-VL D.6.2.  6   The table below shows some typical transfer times over different networks, for IMX50, IMX30/DV25, uncompressed video and an uncompressed TIFF (about 30MB/s for the TIFF scans of Boekentoren). Note that his handles only about standard definition video !  D10-IMX50 IMX30/DV25 uncompressed uncompress TIFFGB/hour 30 15 100 30MB1Gb/s 5 3 17 0100Mb/s 45 23 152 320Mb/s 250 125 833 154Mb/s 1000 500 3333 60min min min sec  From this it may be clear, that for transferring high res video in bulk, a 1Gb/s has to be used to have reasonable transfer speeds. 30MB uncompress TIFFs may seem small, however at a number of 37.000 (Boekentoren ingest in BOM-VL), this becomes also huge. Note however that these speeds have been reached on real networks, but local networks. If delay and multiple hops are involved, a lot of tuning on both sides is needed, to reach this throughput. And it is best to use also multiple parallel transfers which can better use the total bandwidth. For transferring large bulks of digital files without 1Gb/s network, other ways have to be found (external harddisks, LTO tapes, , …).  In the future, if high definition or more will be envisioned, even 1Gb/s may be too slow…  As such, the links between Videohouse, VRT and IBBT are 1Gb/s to let the ingest run at maximum speed. The 1Gb/s link between VRT/Videohouse and IBBT is saturated when running the ingest at maximum capacity at both sites. An upgrade to 10Gb/s does involve a lot of money and is also technically a challenge to let it run at optimal speed ! It will be needed however for archiving high definition material. 4.2 Browse format and keyframes However, for searching and browsing the archive, a normal internet connection (4Mb/s or better) is enough. The video browse format used in BOM-VL is MPEG2, 1Mb/s with 256kb/s audio. For watching real-time 4Mb/s is enough, if however one wants to browse fast through the video (e.g. for searching a particular part of the video) a higher bandwidth will give a better user experience.  The keyframes are about 5-10kB per picture (320x240 pixels, JPG), so a sheet of 100 frames is only about 1MB in size. For the large uncompressed TIFFs of 30MB, also a thumbnail is created.  The search engine itself transfers largely text (as shown below) with one thumbnail per item, so this also is not bandwidth hungry.     BOM-VL D.6.2.  7     5 Practical feedback Gigabit networks are very common today (servers, desktop PCs and even laptops have gigabit network cards). Gigabit switches are affordable. However, using those gigabit networks at full speed is not straight forward. Bottlenecks can be e.g. chipsets (e.g. cheap desktop PCs or laptops can not handle 1Gb/s), harddisks (a single harddisk can only handle max 500-600Mb/s), distances between locations causing extra delay and/or packet loss and lowering the used bandwidth (related to the default TCP window size in operating systems), cheap switches, multiple flows hampering each other, firewalls…  In the BOM-VL demonstrator setup, following things were seen on the gigabit links: • Transfers from VRT to IBBT: only 1Gb/s is used when using up to 10 parallel transfers (can be configured in vivesta MediaFlow). A single flow only uses about 400-500 Mb/s. Probably caused by the VRT firewall. • Local transfers at VRT: if both videoservers run, quality control is done, transfers are done to IBBT and MD5 checksumming is calculated, the local storage for BOM-VL is not fast enough to handle everything • Transfers from Videohouse to IBBT: this works at about 800-900Mb/s, even with a single flow • Transfers from IBBT to Videohouse or VMMa (windows desktops): only about 400 Mb/s can be used after full optimization of the windows desktop (causes are delay 4ms RTT and packet loss). Tests are ongoing to increase this bandwidth, e.g. by using Linux operating system • Locally in IBBT: the firewall used at the start of the project had to be substituted, because it hanged after about 2 weeks continuous traffic • Belnet network: at the start of the project, a Cisco switch of the Belnet network lost its MAC table now and then, making the VPN going offline. They have never seen this before.     BOM-VL D.6.2.  8   It may be clear from this, that transferring Terabytes of data over gigabit networks does ask everything from all the hardware and software (servers, network cards, operating system, switches, disks, …). These transfers go on for days and days and this is a big difference with typical IT use. 10Gb/s hardware may be still too immature to use in these high demanding workflows.  6 Conclusion  From this practical feedback from the demonstrator, it may be clear that for transferring the digitized high resolution files, 1Gb/s or faster connections should be used. If not, harddisks or LTO tapes may be more easy. Keep in mind however, that a single harddisk can only handle about 60MB/s or 500Mb/s, so even copying these files from disk to disk is slow (and slower than a good configured network).  However, for browsing and searching the archive, the MediaDRAIN software is optimized to make this as feasible as possible on typical home broadband connections (about 4Mb/s), by using browse format video of lower quality and thumbnails.  Depending on the application (e.g. using high res files to edit in schools, exchange between broadcasters, …) faster networks (1Gb/s) have to be used.   

BOM_vl WP6.2 Demonstrator: Study of bandwidth needs and possible connectivity for a digital archive in Flanders

https://biblio.ugent.be/publication/764259/file/764260

BOM_vl WP6.2 Demonstrator: Study of bandwidth needs and possible connectivity for a digital archive in Flanders

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography