Controlling Bloglines crawler?
Thomas Dowling
tdowling at ohiolink.edu
Tue Nov 9 16:28:49 EST 2004
Web4Libbers--
Has anyone encountered a way to throttle (in either sense of the word!)
the Bloglines RSS crawler? We have a server with a large number of
feeds. While Bloglines doesn't fetch any individual feed more than once
an hour, they tend to fire off requests for all of our feeds in rapid
succession, creating bursts of heavy traffic that can do bad things if
they come in during an already high load on the server. Any chance
they'd understand a 304 Not Modified HTTP status?
--
Thomas Dowling
tdowling at ohiolink.edu
More information about the Web4lib
mailing list