[Web4lib] How completely are you crawled?

[Robert Sullivan]

> This made me wonder what others have been experiencing regarding crawling
> coverage of their database-driven sites even when providing a Google
> Sitemap. Can anyone else report their crawling statistics? If you're
> registered your site map at Google Webmaster Tools[2], you can find the
> appropriate statistic by selecting "Sitemap" from the menu on the left, then
> clicking on the "Details" link beside the appropriate sitemap. Thanks,
> Roy

My pages are static and were over 90% crawled, but I created the
sitemap anyway to improve coverage - after a couple of months Google
has indexed 1943 out of 2116 URLs in the sitemap.  On the other hand,
doing a "site:url" search tells me Google has indexed about 2100 URLs.
 Haven't figured that out yet.

-- 
Bob Sullivan
Schenectady Digital History Archive
<http://www.schenectadyhistory.org/>
Schenectady County (NY) Public Library