Public workstation home pages survey -- results (fwd)

[Mia Massicotte]

Walter W. Giesbrecht  (walterg at yorku.ca) asked:

>On another note: my access log notes about a dozen attempts to
>retrieve a file called robots.txt from the root directory of the
>server. Such a file has never existed here, and my colleagues who
>manage other servers on campus have noticed the same thing. Several
>of them have come from query2.lycos.cs.cmu.edu, which might be a
>Lycos search & index attempt; others are not obvious. Any ideas?

If I recall, robots.html is a file you include on your server if you do not
want your server to be hit by a robot.  The robot looks for such a file; if it
exists, the robot disregards your server for harvesting.  There is an
explanation of how robots work on the net somewhere, and this is explained. 
Sorry, I don't have the URL on hand; perhaps someone else does?

Mia Massicotte, Systems Librarian
Concordia University Library, Montreal, Quebec CANADA
miamass at vax2.concordia.ca