protecting web servers from robots

[Charles A. Summerhill]

chris at sparkie.osl.state.or.us wrote:
> Can anyone provide information about how to protect web servers from the
> robots that roam the internet, such as Altavista, Lycos, etc. I have several
> Unix servers and  a Mac server. I know that I have heard of a way to do
> this, but can't remember what it was.
> 
> Christopher Adams


Check out the following URL: http://info.webcrawler.com/mak/projects/robots/norobots.html

This document is entitled "A Standard for Robot Exclusion" and discusses the creation of a 
"robots.txt" file which limits which robots and which directories said robots have access 
to.

Hope this helps...

--Charles A. Summerhill
  LibraryLRC Network Manager
  University of Arkansas for Medical Sciences
  http://www.librarylrc.uams.edu/