Performance of Infoseek

[Ian Winship, Univ Northumbria Info Services]

----------------------------Original message----------------------------
Pardon the cross posting on PACS-L, GOVDOC-L, and IR-L


Response by Steve Kirsch, President, InfoSeek Corporation to
"Performance of Four World Wide Web (WWW) Index Services:
 Infoseek, Lycos, Webcrawler and WWWWorm" by H Vernon Leighton

Vernon,

I received a copy of your study. Very interesting.

There are a couple of things which you may wish to add before
posting on the net again (and append as notes to your study):

1. InfoSeek has extensively tested and benchmarked its precision recall
   performance against other system and has consistently emerged better
   than or equal to the top rated systems in TREC. This is based upon
   hundreds of queries and professional relevance judgements.

2. It's dangerous to reach conclusions based on 8 queries.

3. In Query #1, Lycos was told to get all three words, while InfoSeek
   was not given the same instructions. That's not fair!

4. Except in one case, you failed to take advantage of the incredible power that
   InfoSeek offers. The biggest precision booster in queries is to use
   discriminating phrases. Lycos does *not* support this. InfoSeek does!
   There is absolutely no doubt in my mind that for any given query,
   you can formulate it more precisely and get more precise results
   using InfoSeek than with Lycos and I'd challenge you to come up
   with a just a *single* counter example of this!

   For example,
        +chess tournament       should be       chess-tournament
        law school              should be       law-school
        precision recall        should be       precision-recall
        civil war               should be       Civil War

5. I recently needed to get Jane Anderson's email address. I typed Jane Anderson
   in InfoSeek and virtaully all hits were relevant. The top hit had
   exactly what I needed. In Lycos, I got one hit in the top 10 that was
relevant.
   Clearly there are lots of practical, real life queries, where Lycos doesn't
come
   close to matching InfoSeek's precision.

If you want to be fair, you may wish append this response to your study or
incorporate it in the conclusions.

Steve Kirsch
President
InfoSeek

 Reply by H. Vernon Leighton to Steve Kirsch

I would like to address the points raised by Mr. Kirsch.

1.  First, I reread my announcement, and I am afraid I did not make
    clear in it that the one conclusion that I was confident of was
    that both Lycos and Infoseek are better than either Webcrawler
    and WWWWorm. In my study, I did not conclude that either Lycos
    or Infoseek was better than the other, in part because I felt
    that my sample was too small to infer any statistical difference.
    So your claim that Infoseek *is* in fact statistically better does
    not contradict my basic finding.

2.  I totally agree that eight queries a definitive study does not make.
    My main reason for not doing more was the tremendous amount of time
    required for following the links of each result's set to establish
    even an admittedly imperfect number of relevant hits. In that case,
    of course, then I could be charged justifiably with advertising my
    study as more than it actually is. For that reason you are
    understandably concerned.

3.  On the point that I failed to use the syntax of Infoseek, well,
    that *may* be true. I tried to read the online help in each
    service to find service specific optimizations, but if as you say,
    I really missed the boat, then I apologize. On the other hand, if
    I missed the optimizations after having attempted to read the help
    pages, then many other users probably have missed them, too.

    I tried in general to come up with an expression in each service that
    customized it. I often tried two or three candidates and took the best.
    For instance, I often had to shave words off of my query in WWWWorm to
    get any hits. I had to work the query about ATM in Lycos to keep from
    generating many false hits. With the query that you questioned, I
    believe that I tried a few different expressions in Infoseek, especially
    because I had bad results in what was otherwise a very good service. But
    even without that question, my statistical results were about the same.
    I could be unconsciously biased, but I felt that I made a good faith
    effort to try to get some relevant results in each index, if possible.

4.  Thank you for the syntax tips.

I was impressed by the extent to which you test and evaluate your
own service. It shows in the quality of the results. I have recommended
to our colleciton development librarian that we subscribe to the fee-based
Infoseek service for our Reference Desk at the library. We have not done
so yet, unfortunately, but I remain hopeful. I congratulate you on offering
a free service, and wish you best of luck in generating sales, you clearly
deserve it.

I would like to explain again why I published my study the way I did. I did not
send it to a peer-review journal, not because I was afraid of responsible
critisisms such as Mr. Kirsch's (and on Internet, one can publish irresponsibly)
but because the results, with their admittedly limited validity, become
increasingly inaccurate with each passing day. I felt that announcing on
Listservs and publishing on the Web was the most timely way to deliver the
finding while it had any usefulness at all. Had my other duties allowed,
I would have advertised it at the beginning of September.

Vernon

H. Vernon Leighton
Government Documents Librarian
Winona State University
Winona MN 55987
(507) 457-5148
Leighton at vax2.winona.msus.edu