[WEB4LIB] "spiderability" problem

[Nick Arnett]

At 09:35 AM 5/6/99 -0700, Erwin Wodarczak wrote:
>I chair a committee which oversees the Web site for our professional
>organization.  When we tried to register our URL with Lycos, we received
>this response:
>
> >We were unable to spider your page.
> >url: http://aabc.bc.ca/
> >error: HTTP Client: missing header

That's strange, since that page appears to have a valid header.  I don't 
see any explanation on Lycos for this, either, so I'll toss it to the 
robots list and see what folks there have to say.

Nick