[Web4lib] Sitemap.xml

[VanderHart, Robert]

A speaker on SEO at the IA Summit earlier this week stated that it's
very important to have a sitemap.xml file for your website to indicate
to spiders how often to visit your site.  I know from reviewing our
server access logs that spiders should request a robots.txt file before
indexing a site, and when I grep the logs I see plenty of requests for
that file.  But when I grep "sitemap.xml", I don't see a single request.


So the question is, if a sitemap.xml file is so important, why aren't
any spiders looking for the file?  I didn't raise the question to the
speaker because I couldn't view our log files while I was at the Summit,
so I wasn't certain whether we were getting any requests for sitemap.xml
or not.

Robert Vander Hart
Electronic Resources Librarian
Lamar Soutter Library
University of Massachusetts Medical School
Worcester  MA  01655

Voice: 508-856-3290
Email: Robert.VanderHart at umassmed.edu
Web: http://library.umassmed.edu