protecting web servers from robots

Jian Liu jiliu at nickel.ucs.indiana.edu
Thu Jun 13 19:05:22 EDT 1996


Check the following:

http://info.webcrawler.com/mak/projects/robots/norobots.html

Jian
Indiana University Libraries

> 
> Can anyone provide information about how to protect web servers from the
> robots that roam the internet, such as Altavista, Lycos, etc. I have several
> Unix servers and  a Mac server. I know that I have heard of a way to do
> this, but can't remember what it was.
> 
> Christopher Adams
> Oregon State Library
> chris at sparkie.osl.state.or.us
> 
> 



More information about the Web4lib mailing list