Web Search Engines "Made Simple"

[Byron C. Mayes]

Sorry about that too short earlier message.

What *I'm* seeing in this thread as a central issue is not so much that we
need better controlled-vocabulary search tools for the web (which may be
true), nor that we as information professionals need to take a "you get 
what you pay for" mentality towards those tools which are available and 
just be glad they're not charging us for it. What seems to be a real need 
is more information about how the tools out that are available work.

Yes, it's true that there are a lot of "un-savvy" searchers out there, but
there are a lot of us out there who are "search-savvy" (not just
librarians) and an added group who have some understanding of basic set
logic (which a lot of us learned back in elementary school). The
expectation is that when one submits a search (a direct order phrase, for
example) that should logically produce a smaller set than another set
(e.g., a string of words that can appear anywhere), one logically expects
fewer hits. It doesn't matter that one is not going to look at all 24,000+
records found: one did the more restrictive search presumably to avoid
24,000+ hits in the first place. What matters is taht one retrieved an 
illogical search result. If the more restrictive search retrieves 24 
times the results of a more broad search, the savvy searcher will 
logically conclude that either s/he did something wrong or that the 
engine has a bug, but not that it's a "feature". 

Controlled-vocab, thorough indexing, and advanced search techniques 
aren't the only added value we get from those databases and CDs we pay 
for, and they don't in themselves preclude illogical behavior on their 
parts. Dialog, SPIRS, and theothers don't always work as they should 
either. We do get two resources to help us resolve any problems 
encountered, though: documentation and tech support.

Now I wouldn't expect the free search engine providers to provide
technical support even at the level that the commercial vendors do (which
can often leave a bit to be desired anyway), but they *can* offer a bit
more information about how things work (i.e., documentation). What we 
have now is akin to those quick ref cards we leave out by the search 
stations. Fine when one does a simple search, but they generally fall 
short when more advanced techniques are required.

There is definitely room for improvement here. If an engine is going to
provide advanced tools, it makes sense to assume a slightly more advanced
user. The manuals that come with my high-end page layout software assume
that I know the basics about my computer and page layout and rightfully
offers very little assistance with either (and it's not a function of
price, as most shareware/freeware I download assumes even more knowledge
on my part). Why not here?

Okay, so full disclosure leaves the tool open to spammers, and the stop
list is fluid, changing regularly to accomodate new information. Fine.
Don't tells us the algorithm in use, but at least tell us that a dynamic
stop list exists. Does that need to show up on the first page for every
searcher doing the most simple search? Perhaps not, but if one does a
search on "date rape" and retrieves a Sagan of hits (billions and
billions), why not tell the user at this point, "Hey, you got a lot of
hits here because you used a stop word <date> that was ignored. Wanna try
again using a similar term, or plow through the list you retrieved?" That
way the user is once agin in control and can either start browsing or 
thinking, "Maybe I'll try 'acquaintance rape' as it's sometimes called."

In short, why not give the searcher some information about what's going
on, especially if s/he's using the more advanced capabilities offered?
High-end features require high-end feedback. Free or not, the more we know
about the tools we're using, the better we can be at evaluating their
usefulness and making the right decisions at the right time. 

 Prof. Byron C. Mayes
 Systems Librarian/Assistant Professor
 Hunter College of the City University of New York
 695 Park Avenue * New York, New York 10021
 bcmayes at shiva.hunter.cuny.edu  * 212-772-4168 * Fax: 212-772-5113