[WEB4LIB] Re: MSIE's "Web Archive" feature

[Thomas Dowling]

> ----- Original Message -----
> From: Roy Tennant <rtennant at library.berkeley.edu>
>
>
> > I'm surprised that no one has yet mentioned a cool feature of MSIE 5.0
for
> > the Mac. MSIE now has the same capability of "Web Whacker" and other
> > downloading tools that allow you to download a web site for offline
> > browsing. I've already used it to grab sites for talks when I don't
want
> > to bet the farm on a net connection or use static screen shots.
> >


----- Original Message -----
From: "Nancy Sosna Bohm" <plum at ulink.net>



> Yes, I've been using it too and wondered why people were still talking
about
> Web Whacker....
> Hope the Web Whacker folks don't feel like they've been Netscaped by MS.
>

[I suppose the moral of that story would be not to bet the family farm--or
the venture capital--on something that's so easy to duplicate.]

For several versions now, IE's Add To Favorites menu has included a
serious crawler option.  If you tell IE to make a Favorite available
offline, you can go on to establish a schedule for crawling that site and
specify how many levels of links to follow.  Our experience here is that
not all versions of IE have correctly read robots.txt or honored the
Standard for Robot Exclusion.  Because of that, and because we have some
content whose license forbids crawling, we've had to ban hits from
MSIECrawler (along with WebZIP and Go!Zilla) on one of our sites.
MSIECrawler also seems to pull files off a server as fast as it can,
rather than politely pausing between to prevent tying up the server.


Thomas Dowling
OhioLINK - Ohio Library and Information Network
tdowling at ohiolink.edu