Web Search Engines "Made Simple" -Reply

[Karen Harker]

Along similar lines, has anyone been able to view all of the results from a
search (over 500 hits)?  I did a search for medical residency sites which
yielded over 10,000 hits.  Over a period of several days, I viewed these
results, 10 or 20 at a time (I think it was Excite), bookmarking the last
page I visited at the end of the day.  When I got to the 500th result and hit
the 'next' button, the same 490-500 were displayed.  I could go no
further.  I tried the search a couple of times, but still could go no further
than 500.  

Is this common with all search engines?  Is there a trick around it?  I
know that viewing more than a few sites will not usually be necessary,
but my concerns go deeper than that.  I would like to do some research
on web sites, and would like to be able to pull up thousands of sites, not
just the first 500.  I know that research has been done, but I don't know
how they succeeded if the search engine stops at 500.

Any ideas?

Karen R. Harker, 
Information Resources Center Web Developer
UT Southwestern Medical Library
5323 Harry Hines Blvd.
Dallas, TX  75235
214-648-2001, fax: 214-648-3007
http://www.swmed.edu/home_pages/library/