[WEB4LIB] Alexa project

[Dan Lester]

At 07:02 AM 3/16/99 -0800, Grace Garbe wrote:
>There are certainly pages that I don't want archived and even some that I 
>probably don't want robots indexing.  Our website just recently

Well, not at all sure they're willing or able to "un-index" it, but at 
least you can cover this for the future.

>grew from 3 pages to 13 and I had not put any meta tags in the documents 
>to prohibit indexing, nor had I created a robots.txt file.  I'm doing that 
>today.  But my question to those of you with more

Sounds like the correct thing to do.

>experience is do you permit Alexa (or other projects, if there are any) to 
>archive your library's website?  Should I be concerned about this 
>archival?  Do you permit robots to index all of your pages?
We restrict our intranet content to our ip range so that it is only 
available on campus.  I tell page preparers how to handle having pages not 
indexed, but don't know that any of them do so (there are about twenty of 
them).  As a public university, we don't have any need or desire to 
restrict things other than those that are under development and those that 
are for internal use only (such as statistical reports, budget reports, 
minutes of departmental meetings, etc.)

cheers

dan

--
Dan Lester, 3577 East Pecan, Boise, ID 83716-7115 USA 208-383-0165
dan at 84.com   http://www.84.com/  http://www.idaholibraries.org/
http://library.boisestate.edu/   http://cyclops.boisestate.edu/
http://www.lili.org/  http://www.postcard.org/  http://www.homesight.org/