Public workstation home pages survey -- results (fwd)

Oliver Heyer oheyer at
Fri Sep 22 20:19:23 EDT 1995

> If I recall, robots.html is a file you include on your server if you do not
> want your server to be hit by a robot.  The robot looks for such a file; if it
> exists, the robot disregards your server for harvesting.  There is an
> explanation of how robots work on the net somewhere, and this is explained. 
> Sorry, I don't have the URL on hand; perhaps someone else does?
> Mia Massicotte, Systems Librarian
> Concordia University Library, Montreal, Quebec CANADA
> miamass at

There is a pretty detailed explanation of Web robots at the following site:

Oliver Heyer
Art History/Classics Graduate Service
U.C. Berkeley                              

More information about the Web4lib mailing list