Public workstation home pages survey -- results (fwd)

[Oliver Heyer]

> 
> If I recall, robots.html is a file you include on your server if you do not
> want your server to be hit by a robot.  The robot looks for such a file; if it
> exists, the robot disregards your server for harvesting.  There is an
> explanation of how robots work on the net somewhere, and this is explained. 
> Sorry, I don't have the URL on hand; perhaps someone else does?
> 
> Mia Massicotte, Systems Librarian
> Concordia University Library, Montreal, Quebec CANADA
> miamass at vax2.concordia.ca
> 


There is a pretty detailed explanation of Web robots at the following site:

http://info.webcrawler.com/mak/projects/robots/robots.html

Oliver Heyer
Art History/Classics Graduate Service
U.C. Berkeley