New search engine precision study

[LEIGHTON at VAX2.WINONA.MSUS.EDU]

pardon the cross posting on Govdoc-L, IASSIST-L, IR-L,
and WEB4LIB-L   
pardon also the self-promotion.

Announcing a new study of relevance among Web Search Engines,
located at: 

http://www.winona.msus.edu/is-f/library-f/webind2/webind2.htm

Here is a condensed version:

-----------------------------------------

Precision among World Wide Web Search Services (Search Engines): 
Alta Vista, Excite, Hotbot, Infoseek, Lycos

                                    By 
                H. Vernon Leighton, MS in Computer Science, 
          Government Documents Librarian, Winona State University,

                                    and 
                            Dr. Jaideep Srivastava, 
          Associate Professor of Computer Science, University of Minnesota. 

[FROM THE TEXT OF THE STUDY]

[FROM THE FORWARD]

  This study was conducted in part to correct the problems present in 
Leighton's early study (Leighton, 1995). In that study, the test suite 
was inadequate, the statistical model was inappropriate, and the methods 
were subject to possible bias. ...

[FROM THE ABSTRACT]

  Five search engines, Alta Vista, Excite, Hotbot, Infoseek, and Lycos, are 
compared for precision on the first twenty results returned for fifteen 
queries. All searching was done from January 31 to March 12, 1997. Steps 
have been taken to ensure that bias has not unduly influence the evaluation.
Friedmann's randomized block design is used to perform multiple comparisons 
for significance. ...

[FROM THE INTRODUCTION]

...
  We began this study because many previous studies have arrived at 
conflicting conclusions as to which services are better at delivering 
superior precision and because most of those studies have either had small 
test suites or have not reported how the study was conducted. This study 
compares all search services that have been recommended recently for their 
performance at delivering relevant results for the user, and uses a 
carefully controlled and documented experimental design to ensure valid 
results. 

  The results of the various experiments indicate that there are three 
services, alphabetically Alta Vista, Excite and Infoseek, that are superior 
in most respects to the other two studied, Hotbot and Lycos, in terms of 
first twenty precision. How these top three rank compared to each other
depends on how one defines "relevant." The chief problem with Hotbot is 
the frequent occurrence of duplicate links; the chief problems with Lycos's 
performance are irrelevant and inactive links. Lycos's irrelevant links are 
often caused by the lack of a "required" operator (often denoted by a + 
prefix). Both Hotbot and Lycos do not seem to implement the fuzzy match 
for unstructured queries, in which lower case search terms match both lower 
and upper case terms, but upper case only match upper case terms. 
...

[FROM THE RESULTS]

The different experiments show how stark the contrast is in scores depending 
on how one defines "relevant." Experiment one called a link good if it at 
least technically satisfied the search expression. Here, the overall median 
was a healthy 0.81 with the best service scoring a 0.93. If ones definition
of relevant is stricter [EXPERIMENT 2], dealing with only potentially useful 
pages, the overall median drops to 0.39, with the top scorer only making 
an estimated median of 0.51. If ones criterion is a page that is very likely 
to be useful [EXPERIMENT 3], the median disappears down to 0.06, with the top 
scorer only rising to 0.10. ... [FOR FURTHER EXPERIMENTS, SEE FULL VERSION]



Table 3: The rankings and significant differences among the services 

                               Rankings 
Experiment      Lowest                           Highest         
--------------------------------------------------------
    1            [Lyc   Hot]     [Inf]     [Alt   Exc]
                  ---------       ---       ---------

    2            [Lyc   Hot]       [Alt   Exc   Inf]
                  ---------         ---------------

    3            [Lyc   Hot]     [Alt   Exc]     [Inf]
                  ---------       ---------       ---
    .
    .
    .

Table 3: Each service is ranked by experiment. Underlining and brackets 
indicate no significant difference. 
Alt = Alta Vista, Exc = Excite, Hot = Hotbot, Inf = Infoseek, Lyc = Lycos. 

6/24/97


H. Vernon Leighton
Government Documents Librarian
Winona State University
Winona MN 55987
(507) 457-5148
Leighton at vax2.winona.msus.edu