[Web4lib] Spidering speed

[Ryan Eby]

You'll probably find them mixed. I would hope that they could handle
the slightly extra load in case patrons actually decide to use it.
However I guess it depends on whether your threading multiple requests
to the same opac. I've done various experiments with ours and never
noticed a decrease in performance. There is also z39 and xml running
on the same server which doesn't seem to hinder the performance. Ours
could probably take more then a single hit but I know there are
other's that would probably crash and burn under any load.

The library can always opt-out of the widgets at a later time if they
notice adverse affects, right?

Ryan Eby

On 3/31/07, Tim Spalding <tim at librarything.com> wrote:
> Related to the sitemap issue...
>
> Does anyone know what sort of tolerances OPACs have for spidering? Can
> most handle the industry-standard 1 hit/second? (It may be industry
> standard, but insofar as most OPACs are not spiderable with
> conventional means, they may not be tested for it.) Can they handle
> faster?
>
> We're going to be introducing a bunch of library widgets soon,
> providing LibraryThing functionality to conventional OPACs. Some of
> the widgets require some data to be fetched off the OPAC, basically
> through screen-scraping. We need to know how hard we can hit them.
>
> Tim