Spiders & META Tags

[Thomas Dowling]

http://calafia.com/webmasters/meta.htm includes: 

"One other meta tag worth mentioning is the robots tag. This lets you
specific that a particular page should not be indexed by a search engine.
The format is like this:
<META NAME="ROBOTS" CONTENT="NOINDEX">
Not all search engines support this tag. As an alternative, all the major
engines support the robots.txt convention of blocking indexing..."

[N.B. if you have access to your server or can get a favor from your
webmaster, I'd recommend you use the robots.txt file as described in the
Standard for Robot Exclusion.]

I'm not aware of any search engine the penalizes you simply for using META
tags in general.

Thomas Dowling
OhioLINK - Ohio Library and Information Network
tdowling at ohiolink.edu

 ----
From: Pat Anderson <pfa at nwu.edu>
To: Multiple recipients of list <web4lib at library.berkeley.edu>
Date: Wednesday, May 21, 1997 12:29 PM
Subject: Spiders & META Tags

>I am trying to verify something which I thought I had
>read previously on this list. Yes, I searched the archive,
>and didn't find it.
>
>My question is:
>
>Are there any major web search engines, robots, spiders,
>etc. (such as InfoSeek, Altavista, Yahoo, Excite, etc.)
>which have decided *not* to index sites which use META tags?
>
>I remember this being discussed in the context of using
>a META tag to tell robots and spiders to "go away."
>
>Our reason is that we want to use META tags to assign
>subject terms for our web site for our own search engine,
>but do not want to scare off any possible indexing by other
>search engines. We want as many people as possible to find
>the site. If META tags have been banned by a major web search
>service due to abuse by poor web designers, then we will
>find some other way to assign subject terms.
>
>Thanks in advance for all your help!
>
>Pat Anderson
>