[WEB4LIB] no robots

[John Hubbard]

Place a robots.txt file in your root Web server folder containing something
like:

User-agent: *
Disallow: /secret/folder/ # This will forbid robots from spidering this
folder

More at:
http://info.webcrawler.com/mak/projects/robots/norobots.html

You can also mark individual pages with a meta robots tag, i.e.:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
in the document header.

- John


-----Original Message-----
From: web4lib at webjunction.org
[mailto:web4lib at webjunction.org]On Behalf Of Stéphane Dudart
Sent: Friday, January 19, 2001 9:14 AM
To: Multiple recipients of list
Subject: [WEB4LIB] no robots


Could you tell me how to abort robots indexation on html files centralized
in a same folder?

Thank you

Stéphane