Spiders & META Tags

[Diane L Morey]

>Are there any major web search engines, robots, spiders,
>etc. (such as InfoSeek, Altavista, Yahoo, Excite, etc.)
>which have decided *not* to index sites which use META tags?
>
>I remember this being discussed in the context of using
>a META tag to tell robots and spiders to "go away."
>
>Our reason is that we want to use META tags to assign
>subject terms for our web site for our own search engine,
>but do not want to scare off any possible indexing by other
>search engines. We want as many people as possible to find
>the site. If META tags have been banned by a major web search
>service due to abuse by poor web designers, then we will
>find some other way to assign subject terms.
>
>Thanks in advance for all your help!
>
>Pat Anderson
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
>P. F. (Pat) Anderson               "Learning, study, reading,
>Barnes Learning Resources Center    and the preservation of
>Galter Library / Northwestern U.    books are all integral
>303 E. Chicago Ave.                 to spiritual practice.
>Chicago IL 60611-3008               We get into trouble when
>312/503-8238, voice                 we give up any of these ..."
>312/503-8028, fax
>Internet: pfa at nwu.edu               Thomas Moore, Meditations,
>WWW: http://www.galter.nwu.edu/        pg. 29
>WWW: http://pubweb.acns.nwu.edu/~pfa/pfahome.html
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Pat,

Hmm, I think you've confused two 'bot-related issues here.

A robot-exclusion file (which is not part of any META tag information
that I am aware of, but is a separate specific file on a server) will
exclude robots from searching a site altogether.  Site administrators
may choose this option for privacy reasons, or simply to reduce server
load from robots.  But the default option is that sites are searchable
by robots.

A current issue with META tags is whether search engines will use
those particular tags or not while indexing your site.  Of the search
engines you've listed, I believe only Excite will currently NOT use
META tags in indexing; Excite _will_ index your site using only the
visible text on the page.  I know some other sites do reserve the
right to exclude your site for excessive or inappropriate use of META
tags, as there have been cases where META tags were misleading, or
specifically designed to increase a relevancy score for a document.

But no, AFAIK no search engine will currently refuse to index your
site simply because you include META tags.  If anyone else is aware of
examples contrary to what I just stated, please share them...  


Lee Morey	dmorey at gpo.gov  (202)512-1698
Electronic Transition Staff, U.S. G.P.O.