protecting web servers from robots

Mary-Ellen Mort memort at netcom.com
Fri Jun 14 01:05:23 EDT 1996


I just finished reading Dr. Website's column in the June 3 edition of Web 
Week. He says you can create a file called robots.txt and place it at the 
root level of your server. He gives an example of how you can limit 
specific robots, or all robots, from certain sections of your server.

Unfortunately I don't see this particular question in Dr. Website's 
column ont he Web Week site:
http://www.webweek.com/drwebsite/index.html

Dr. Website recommends WWW Robots FAQ at:
http://info.webcrawler.com/mak/projects/robots/faq.html

HTH, 
Mary-Ellen Mort, JobSmart Project Director
http://jobsmart.org



More information about the Web4lib mailing list