Webcrawler no longer reviewing sites.

Danny Sullivan danny at calafia.com
Thu Dec 11 20:05:38 EST 1997

> I think we are seeing more results of the corporate skakedown in
> search engines and indexes for the Internet.  This type of thing
> is changing my thoguhts on librarians cataloging the Internet.  I
> am starting to believe we should.

I wouldn't criticize WebCrawler as a defunct search tool because it 
no longer does web site reviews (something they've haven't done in a 
comprehensive fashion for some time, actually).

Many librarians are no turning to search engines for reviews. There 
are plenty of sites where they can find sites reviewed. Often, 
however, they seem to turn to search engines in order to find sites 
that match obscure topics.

In these cases, what you want is a search engine with a large index 
of web pages. This is where you should really fault WebCrawler. It 
continues to list only 2 million web pages, while others are hitting 
up to 100 million.

More is not always better, but for professional researchers, it will 
be. It provides them a huge sample of the web, which they are more 
than capable of filtering.

Don't expect the major search engines to return to wide-spread 
reviewing, either. That requires people, and people cost money. That 
money is being spent in either places.

Instead, look toward some up and coming places for reviews:


Encyclopedia Britannica Internet Guide

>  Webcrawler was acquired by Excite Corp. not too long ago.

It's been over a year. How time flies.... 


Danny Sullivan, Search Engine Watch

More information about the Web4lib mailing list