[Web4lib] Optimizing services for the low-bandwidth communities

[David Dorman]

At 05:58 AM 10/24/2006, K.G. Schneider wrote:
>On the heels of giving a talk in South Africa (and having a week of serious
>bandwidth withdrawal) I'm writing a Techsource post about optimizing
>web-based services for low-bandwidth communities. I'd appreciate any war
>stories, tips, and lessons-learned that you folks would like to share.

Karen,

Here are some guidelines for managing Internet Bandwidth I wrote two 
years ago for university libraries in East and West Africa, and 
recently revised based on feedback from Chris Wilson, the Chief 
Engineer of Aidworld.  INASP, in conjunction with ICTP, has published 
a book on bandwidth management that is available online at 
http://www.bwmo.net/.

David

INTERNET BANDWIDTH MANAGEMENT GUIDELINES FOR UNIVERSITIES IN EAST AND 
WEST AFRICA

A report written in December 2003, entitled "VSAT Case 
Studies:  Nigeria and 
Algeria" 
(http://www.researchictafrica.net/images/upload/vsatngdz.pdf), 
indicated that Nigerian ISPs paid between $3.40 and $4.70 per Kbps of 
bandwidth per month for VSAT connections.   Some Nigerian 
universities are currently pay much higher prices.  This report 
stated that in Lagos, almost 20% of cyber cafe users were 
scholars.  This figure is an indication of the current failure of 
Nigerian universities to provide adequate Internet access to its 
faculty. Through negotiations with Satellite ISPs, the Partnership 
for Higher Education in Africa 
(http://www.foundation-partnership.org/linchpin/index.php) has been 
able to reduce those rates through consortial purchase of bandwidth, 
to slightly less than $3.00 per Kbps.  This is, however, still 
hundreds of times more expensive than the cost of bandwidth for 
Universities in North America.  Because Internet connectivity is so 
costly in most of sub-Saharan Africa, and is likely to remain so for 
some time, careful and continuous bandwidth management is essential 
for effective access to Internet-based resources.

It is this writer's estimation, based on the expected utilization 
level of Internet-connected workstations on University campuses 
(minimum 50%) and the file size of many typical Internet-based 
library resources (in excess of 2MB), that a 4 Mbps to 6 Mbps 
downlink capacity is the minimum realistic baseline for Internet 
connectivity for large universities.

Even with a 4 to 6 Mbps bandwidth capacity, faculty and students will 
not have effective access to educational resources on the Internet 
unless the university's bandwidth is managed effectively.  As Chris 
Wilson, the Chief Engineer of Aidworld (www.aidworld.org) has noted: 
"upgrading the bandwidth will not provide any improvement in real 
performance unless the network is already well managed, and viruses 
and music downloads are eliminated."

The following suggestions are meant to be an overview of the 
technical and organizational components that an effective bandwidth 
management program might contain.  They are not meant as a detailed blueprint.

         EMAIL BANDWIDTH MANAGEMENT STRATEGIES

1.Only allow the use of Web email services that are bandwidth 
efficient, such as Fastmail (http://www.fastMail.fm) and  Gmail 
(http://mail.google.com/mail/help/about.html).  One of the criteria 
in evaluating which such services to allow should be the service's 
ability to effectively filter spam.  In addition to providing the 
university community with stable and reliable university-based email 
accounts, the university should also consider offering "an internal 
webmail server that uses IMAP to communicate with these services 
(e.g. Fastmail or Gmail), so that students have a permanent address 
that is accessible from outside the university and does not cease to 
function when they leave the university." (Chris Wilson).

2.Implement a spam filter to support efficient email use (e.g. 
SpamAssassin (http://spamassassin.apache.org/)).  Although spam 
filtering by the university's email server will not increase 
effective bandwidth, it will enhance the experience of those using 
the university's email service.

3.Try to find some way to get spam filtering services provided 
outside of the campus network, so that email can  be filtered for 
spam before it gets downloaded from the Internet.  This will save 
considerable bandwidth usage.  Possible strategies are to arrange for 
the University's ISP provider to filter spam, or making an 
arrangement from a European or North American university that 
would  be willing to intercept all incoming email and filter it 
before passing it on to the University's mail server.

4.Establish a policy of holding all outgoing email messages that 
exceed a certain size (e.g. 10KB), sending them out late at night 
when bandwidth usage is low.  This will encourage responsible email 
behavior without adversely affecting most communication.

         WEB BANDWIDTH MANAGEMENT STRATEGIES

5.Obtain and use Internet Filtering Software to limit unrestricted 
Internet surfing during times of peak use (e.g. DansGuardian 
(http://dansguardian.org/?page=introduction), SquidGuard 
(http://www.squidguard.org/intro))

6.Cache frequently accessed sites using a web proxy cache.  This is 
middleware that acts as an intermediary between clients and remote 
web servers, caching frequently requested pages to avoid contacting 
the server repeatedly for the same page.  One example is Squid Web 
Proxy Cache (see http://www.squid-cache.org/).

7.Limit the amount of bandwidth an individual user has available for 
downloading, either by establishing a fixed individual limit or by 
dynamically sharing the available bandwidth "fairly" among active 
users.  The word "fairly" is used, rather than "equally," because 
sharing bandwidth equally among users can have an adverse effect on 
obtaining library resources.  Files are measured in bytes, whereas 
bandwidth is measured in bits.  This means that it takes eight 
seconds to download a 64 KB file fully utilizing a 64 Kbps 
connection.  Excessively slowing download speeds of library resources 
will frustrate researches and effectively curtain effective access to 
those resources.  It is not uncommon for a JSTOR* .pdf-formatted 
document to be 2MB in size.  If downloading such a document were 
limited to a rate of 9.6Kbps, for example, it would take over 25 minutes.
(*A service providing African universities free access to high 
quality academic journal articles.)

8.Make arrangement with universities in Europe or North American to 
host mirror sites of the library's web site and locally stored 
resources that the library makes available over the Internet.  This 
will prevent access from remote users from using the library's scarce 
bandwidth resources.

         MISCELLANEOUS SERVER-SIDE SUGGESTIONS

9.Use a traffic shaper to prioritize Internet use by application and 
by individual computer or group of computers.  Packeteer 
(www.packeteer.com) is an example of a company that offers hardware 
and software for traffic shaping.  Instruction for building one's own 
traffic shaper can be found in the BMO book (http://www.bwmo.net/)

10.Monitor Internet traffic to identify network bottlenecks, overload 
situations and chronically heavy bandwidth users.  The Multi Router 
Traffic Grapher (MRTG), distributed under an Open Source license, is 
one such traffic monitoring tool 
(http://people.ee.ethz.ch/~oetiker/webtools/mrtg/).

11.Locate DNS servers on the campus intranet to avoid Internet 
bandwidth being taken up by DNS lookups.

12.Supplementing Bandwidth with "Storewidth."  This means storing all 
library content leased or otherwise obtained from publishers, full 
text aggregaters and other content distributors on the universities 
servers whenever possible.

13.Carefully tune the TCP/IP stack on the proxy server.  This can 
make Internet connections over satellite (such as VSAT) significantly faster.

         PC ADMINISTRATION AFFECTING INTERNET BANDWIDTH PERFORMANCE

14.Keep all network-connected PCs installed with up-to-date antivirus 
and anti-spyware software.

15. See that all network-connected PCs are using a proxy cache for Windows.

16.Lock down all network-connected computers to prevent the 
installation of such unwanted software as file sharing (peer-to-peer) 
applications.

17.Configure all network-connected computers so that when hard disks 
are corrupted by such things as power outages or the installation of 
bad software, they can be easily restored to a working state.  The 
use of Ghost images (e.g. Norton's Ghost utility) is one way to 
accomplish this.  Another way is to use a network-based client 
configuration manager to maintain and restore individual PC configurations.

         TRAINING FOR NETWORK ADMINISTRATORS

No strategy for bandwidth management will succeed without 
well-trained network administrators who are able to diagnose and fix 
network and bandwidth problems.  High quality and continuous training 
of network managers is the single most important ingredient in 
effective Internet bandwidth management.




>K.G. Schneider
>kgs at bluehighways.com
>AIM/skype freerangelib
>
>
>_______________________________________________
>Web4lib mailing list
>Web4lib at webjunction.orghttp://lists.webjunction.org/web4lib/

David Dorman
US Marketing Manager, Index Data
52 Whitman Ave.
West Hartford, Connecticut  06107
dorman at indexdata.com
860-389-1568 or toll free 866-489-1568
fax: 860-561-5613

INDEX DATA Means Business
for Open Source and Open Standards
- - - - - - - - - - - - - - -
www.indexdata.com