Presentation on the use of the Squid caching proxy server as a tool for bandwidth management. This formed part of the International Network for the Availability of Scientific Publications' Bandwidth Management and Optimisation Open Source Tools and Solutions project, being a series of workshops conducted throughout the developing world

Halse, G A

Halse, Guy A

South East Academic Libraries System (SEALS)

Bandwidth Management with the Squid Caching Proxy ServerGuy Antony Halse<G.Halse@ru.ac.za>Quick Overview of Squid• Squid is a caching proxy server.• It’s the open-source equivalent of products like Novell’s BorderManager, Microsoft’s ISAS, and Cisco’s ACNS.• You can download it for free at http://www.squid-cache.org/How Squid Fits Into Your NetworkBorder FirewallCaching Proxy ServerClient ComputersThe InternetBandwidth Control Features in Squid• Access Control Lists• Redirectors• Authenticators• Delay PoolsWe’ll talk about each of these in turn, and then look at some examples that show how they can all fit togetherAccess Control Lists• Traditionally used to define who can access what …acl all src 0.0.0.0/0.0.0.0acl RHODESIP src 146.231.0.0/16acl RHODESDNS srcdomain .ru.ac.zaacl ACZADEST  dstdomain .ac.zahttp_access deny all !RHODESIP !RHODESDNShttp_access allow ACZADESThttp_access deny allAccess Control Lists• But in reality have a lot more flexibilityacl ACLNAME keyword where keyword is one of:arpsrcdomain, dstdomain, src, dsttimeurl_regex, url_path, urllogin, port, proto, methodbrower, referer_regexproxy_auth, proxy_auth_regex, ident, ident_regexsrc_as, dst_asreq_mime_type, req_header, rep_mime_type, rep_headermyip, myportexternalAccess Control Lists• Which allows us to write rich rule sets to match our needs• e.g. All student public labs are only allowed to access academic sites during working hours.acl PUBLICLAB src 146.231.104.0/21acl WORKHOURS time MTWTF 08:00-17:00acl ACADEMIC dstdomain .ac.za .edu .ac.ukhttp_access deny PUBLICLAB WORKHOURS !ACADEMIChttp_access allow PUBLICLABRedirectors• Redirectors allow us to re-write URLs before we fetch them• For instance, we could rewrite a popular site to a local mirror:http://www.php.net/ → http://za2.php.net/Redirectors• Redirectors are simple programs and can be easily customised:• And added to Squid:#!/usr/bin/perl –wwhile (<STDIN>) {s{^http://www.php.net}{http://za2.php.net};print;}redirect_program /usr/local/bin/myredirectorredirector_access allow allRedirectors• Redirectors are commonly used to block adverts on web pages.• There are lots of open-source packages that do this, for example AdZapper (http://adzapper.sourceforge.net/)• Advert blocking saves bandwidth but is controversial because many sites rely on advertising for revenue.Redirectors• Redirectors offer a lot of control over contentBUT• You can only have one redirector, so you have to think carefully what you want to do with itAuthenticators• Authenticators are external programs that define how the proxy_auth ACL works.• This lets you force your clients to supply a username and password before granting them access – good for public access computers.• Like redirectors, you can only have one authenticator.  That’s not usually a problem though.Authenticators• Authenticators are also simple programs that read from STDIN and write to STDOUT.guy@walrus:~% ./sampleauthenticatorguy notmypasswordERRghalse mypasswordERRguy mypasswordOKAuthenticators• Squid comes with a lot of authenticators out-the-box: smb, ldap, pam, unix, ntlm, yp/nis, etc.• Perhaps the most useful of these is pam_auth which uses the pluggable authentication module architecture.• PAM allows you to chain authenticators and use multiple authentication sources.Delay Pools• Delay pools are Squid’s answer to bandwidth management.• They allow you to control the amount of bandwidth a particular computer, subnet or proxy server may use.Delay Pools• Delay Pools work like a bucket and a tap.• You can empty the bucket as fast as you like, but it can only fill as fast as the tap will let it.• So important variables are the size of the bucket and the rate at which it refills.Pic: http://www.wildlife-art.co.nz/Delay Pools• Squid defines three types of buckets– aggregate an aggregate bucket applies to the whole proxy server– networka network bucket applies to the user’s class C network (/24, i.e. third octet of IP address).– individualan individual bucket applies to the user’s PC (i.e. the fourth octet of an IP address)Delay Pools• These buckets combine into three classes of delay pool:– Class 1has only an aggregate bucket– Class 2has an aggregate bucket and an individual bucket– Class 3has aggregate, network and individual bucketsDelay Pools• For each bucket we define a restore rate (B/s) and a maximum size (B).• We use -1 to signify “unlimited”• The maximum size is important as it specifies the burst bandwidth available –this can be used to penalize only certain types of download.delay_pools 1delay_class 1 2delay_parameters 1 -1/-1 8000/8000 600/8000Delay Pools• We use ACLs to define who gets put into which delay pools.• e.g. All residence machines are subject to bandwidth controlsacl RESIDENCES src 146.231.136.0/20acl all src 0.0.0.0/0delay_pools 1delay_class 1 2delay_parameters 1 -1/-1 8000/8000 600/8000delay_access 1 allow RESIDENCESdelay_access 1 deny allExamples and Case StudiesAdvert Blocking• Advert blocking saves bandwidth, but• School of Journalism needs adverts to teach new media• Caches bind two IP addresses and only block adverts on one.• Clients choose whether they want adverts or not.http_port cache.ru.ac.za:3128http_port adcache.ru.ac.za:3128redirect_program /usr/local/bin/adzapperacl ADCACHE myip adcache.ru.ac.zaredirector_access deny ADCACHEredirector_access allow allAdvert Blockingcache.ru.ac.za adcache.ru.ac.zaQuota System @ RhodesQuota System @ Rhodesacl IDLOW       proxy_auth_regex -i "/idquotalow.acl"acl IDHIGH      proxy_auth_regex -i "/idquotahigh.acl"acl IDBLOCK     proxy_auth_regex –I "/squid/idquotablock.acl"deny_info       ERR_IDBLOCK IDBLOCK# SLOW = 0.5% * total PVC, rounded to nearest 0.5kBps# V. SLOW = 0.5 * SLOWdelay_parameters 3 -1/-1 -1/-1 2560/20480delay_parameters 4 -1/-1 -1/-1 1280/10240# This MUST be the first rule that requests a usernamehttp_access allow LOGIN !IDBLOCK !NOAUTH# proxy auth based delay poolsdelay_access    3 allow IDLOWdelay_access    3 deny alldelay_access    4 allow IDHIGHdelay_access    4 deny allQuota System @ RhodesDynamic Delay Pools @ UKZN• All TENET sites have bandwidth that’s categorized into national/international traffic, and this is displayed on graphs at http://www.tenet.ac.za/• The problem is how to make most efficient use of this bandwidth – how to allow users to download as fast as possible without impacting usability This is the work of Richard Stubs <stubbs@ukzn.ac.za> at the University of Kwazulu Natal.Dynamic Delay Pools @ UKZNUnauthenticated Requests• One of the biggest bandwidth users these days is software updates.  In particular, software that polls for updates every time it detects a network connection• One way to keep some degree of control over this is to enable proxy authentication – to require a username and password to use the web.Unauthenticated Requests• The down side of this is that some software goes mad when it gets an HTTP 407 response.  Software developers don’t implement incremental back-off algorithms.• How do we let users know what’s going on?Unauthenticated Requests• Our solution forms part of the quota system you just heard about.  It’s just another quota.• Users lose access when they’ve exceeded 1440 TCP_DENIED/407 messages a day, and regain it automatically when they drop below this.• Just another file-based ACL list.

Bandwidth management with the Squid caching proxy server

Rhodes Repository (SEALS)

http://vital.seals.ac.za:8080/vital/access/services/Download/vital:6975/SOURCEPDF

Bandwidth management with the Squid caching proxy server

Abstract

Similar works

Full text

Available Versions

South East Academic Libraries System (SEALS)

Rhodes Repository (SEALS)