WiseCrawler? Or dumb as a stone?

Roy Tennant roy.tennant at ucop.edu
Mon Nov 19 10:42:53 EST 2001


I was checking out my "snapshot" statistics this morning, thanks to a 
script Thomas Dowling shared with me, and was surprised to find out 
that I was being hit hard for my web server statistics. Thinking that 
I had overlooked putting a block in my robots.txt file for my 
statistics directory, I ran to edit it. Low and behold, I had already 
blocked it.

In looking at my raw log file, I discovered that I was indeed being 
crawled by something that calls itself "WiseCrawler". Since it 
completely blew past my block, and was crawling my site at 2-4 
requests a second, it doesn't seem very wise to me. At the very 
least, it is not abiding by common rules of decency for web crawlers.

I searched Google for any information on it, and came up blank. Does 
anyone know anything about it, and is it hitting you hard, and in 
places where it shouldn't? Thanks,
Roy


More information about the Web4lib mailing list