Web Search Engines "Made Simple" -Reply

[j. y. c h e n]

>Along similar lines, has anyone been able to view all of the results from a
>search (over 500 hits)?  I did a search for medical residency sites which
>yielded over 10,000 hits.  Over a period of several days, I viewed these
>results, 10 or 20 at a time (I think it was Excite), bookmarking the last
>page I visited at the end of the day.  When I got to the 500th result and hit
>the 'next' button, the same 490-500 were displayed.  I could go no
>further.  I tried the search a couple of times, but still could go no further
>than 500.
>
>Is this common with all search engines?  Is there a trick around it?  I
>know that viewing more than a few sites will not usually be necessary,
>but my concerns go deeper than that.  I would like to do some research
>on web sites, and would like to be able to pull up thousands of sites, not
>just the first 500.  I know that research has been done, but I don't know
>how they succeeded if the search engine stops at 500.
>
>Any ideas?

HotBot displays the first 1100 hits of your search results (which is only
11 pages to click through since you have the option to display 100 hits per
page).  I would suggest that you revise your search to pull up a more
manageable number of hits, since the relevancy decreases the further down
the list you go.  personally, in terms of search strategy, I find that
brute numbers do not necessarily mean better hits.   as you've stated,
there is usually no need to view more than the first page of results, and
sometimes that handful of top sites will also be a goldmine of references
and links to other sites, one of the great advantages of the non-linearity
of hypertext.  information seeking behavior is always informed by such
serendipitous finds.



___________________________________

j. y.  chen | hotbot  tutor | WIRED  d i g i t a l
(v) 415. 276 .8464  | (f) 415. 276. 8499
	http://www.hotbot.com

The beatings will continue until morale improves!