[Web4lib] Google limit of 1,000 results

[Jennifer A. Heise]

> Sure. Given that Google's ranking algorithm tends to weight pages  
> that important sites point to more heavily than others, there are  
> times when I want to see what someone says about a topic that 1)  
> isn't being pointed to by anyone -- perhaps because it is too new,  
> for example, or 2) isn't linked to by an "important" web site. The  
> point here is that those seeking information have a plethora of  
> purposes -- not all information needs can be adequately served by  
> one, rather specific, model.

Well, the web is full of sites that suggest the best search engines for 
different purposes.  If I was looking for websites that were not linked 
to by 'important' sites, I would go look at, say, Altavista, and maybe 
use a NOT.

What search engines would other people suggest for such a task?

Of course, I'm not sure how ANY web search engine would find a site that 
'isn't being pointed to by anyone'. I guess you would need to find a 
search engine that crawls directory structures instead of links-- are 
there any large search engines out there?

If you were an information scientist and wanted to see what wasn't 
popular, I would think that you could take a basic list of the most 
popular linking sites on your topic and use a standard retrieval web 
search engine to pull up sites on that topic that aren't linked to by 
those popular sites. Or download a google set and a set from another 
search engine, and eliminate all the sites listed in the google search 
from your other search engine listing.

If we are really interested in these kinds of questions (that is, how to 
get this information), we need to be discussing alternatives that are 
solutions to those problems. Nobody on this list is unaware that 
different types of rankings and different types of indexing have their 
strengths and weaknesses.

In other words, instead of wasting all our time ranting about how a 
screwdriver is not a paint can opener, why don't we talk about where to 
get a paint can opener?

-- Jenne Heise