Organizing Web Information

[marc at ckm.ucsf.edu]

Karen Wrote:
|In my head, I keep returning to old Z39.50.  It hasn't caught on the way
|it would need to, but the idea of searching distributed databases through a
|common interface needs to be explored and revisited.  

But we have to break down the process to resource discovery and retrival.

Resource discovery can in turn be broken down to source discovery, and from
that source, discovering which resource meets your criteria.

The WAIS [1] (similar to Z39.50-1988) model used a directory of sources that 
you could search to determine which source would be the best match for your 
query.  You would then fire off a WAIS query to that source, get a set of hits
and retrieve the thing itself in the WAIS protocol using the docid as a key.
I'm not sure if the directory of services was heirarchical or not, but if
this model is pursued it should be.  It'd be much easier to propigate up,
collect and index subj headings from distributed source servers for searching
than to do free-text on all web docs.

Protocols used for the tasks of source discovery, resource discovery and 
resource retrieval need not be the same, as these processes look quite 
different over the wire.  Source discovery might be handled by a distributed 
indexing/search protocol (Common Indexing Protocol) similar to work going on 
in the internet directory services working groups of the IETF (asid, find and
ids)[2].  Resource discovery is probably best handled by a query protocol such 
as Z39.50-199x or a Z39.50-lite (zeee-lite) as proposed at the W3C indexing
conference [3].  Resource retrieval might be best handled by a protocol with 
content-negotiation features such as HTTP.

If I had a hammer, I'd try to make everything look like a nail.  Fortunately,
we have multiple power tools in our protocol belts, so we can use the
proper tool for the proper job.

-marc
[1] - ftp://ftp.think.com/wais 
[2] - ftp://ftp.ietf.cnri.reston.va.us/ietf/asid/asid-charter.txt
      ftp://ftp.ietf.cnri.reston.va.us/ietf/ids/ids-charter.txt
      ftp://ftp.ietf.cnri.reston.va.us/ietf/find/find-charter.txt
[3] - http://www.w3.org/pub/WWW/Search/9605-Indexing-Workshop/