Controlling Bloglines crawler?

Thomas Dowling tdowling at ohiolink.edu
Tue Nov 9 16:28:49 EST 2004


Web4Libbers--

Has anyone encountered a way to throttle (in either sense of the word!) 
the Bloglines RSS crawler?  We have a server with a large number of 
feeds.  While Bloglines doesn't fetch any individual feed more than once 
an hour, they tend to fire off requests for all of our feeds in rapid 
succession, creating bursts of heavy traffic that can do bad things if 
they come in during an already high load on the server.  Any chance 
they'd understand a 304 Not Modified HTTP status?


-- 
Thomas Dowling
tdowling at ohiolink.edu



More information about the Web4lib mailing list