[Web4lib] Cheap DIY web crawling?

Cab Vinton bibliwho at gmail.com
Wed Sep 30 09:22:09 EDT 2009


1. Crawl up to 2 billion web pages/day just by filling out a web form.

2. Design and run your own crawls in minutes -- API for full customization.

3. $2 per million pages crawled and $0.03 per CPU-hr used.

Wow, sounds like something libraries might be interested in, if the
service works as advertised:

http://80legs.com/

[found via Mashable blog: http://mashable.com/2009/09/30/80legs/]

Cheers,

Cab Vinton, Director
Sanbornton Public Library
Sanbornton, NH




More information about the Web4lib mailing list