WiseCrawler? Or dumb as a stone?
Roy Tennant
roy.tennant at ucop.edu
Mon Nov 19 10:42:53 EST 2001
I was checking out my "snapshot" statistics this morning, thanks to a
script Thomas Dowling shared with me, and was surprised to find out
that I was being hit hard for my web server statistics. Thinking that
I had overlooked putting a block in my robots.txt file for my
statistics directory, I ran to edit it. Low and behold, I had already
blocked it.
In looking at my raw log file, I discovered that I was indeed being
crawled by something that calls itself "WiseCrawler". Since it
completely blew past my block, and was crawling my site at 2-4
requests a second, it doesn't seem very wise to me. At the very
least, it is not abiding by common rules of decency for web crawlers.
I searched Google for any information on it, and came up blank. Does
anyone know anything about it, and is it hitting you hard, and in
places where it shouldn't? Thanks,
Roy
More information about the Web4lib
mailing list